From cdad09b2bb9092cdb3d9a8f39d9e0fe5a3af94d1 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Thu, 13 Aug 2026 12:45:41 -0700 Subject: [PATCH 01/24] delta-harness: OpenHouse Iceberg behavioral test harness + guide Adds a self-contained Scala behavioral test harness that characterizes OpenHouse + Apache Iceberg table behavior end-to-end. The harness crosses a large matrix of table layouts (partitioning, MoR/CoW, ordered writes, nested types) with DDL, DML, maintenance, branching/WAP, streaming, and negative-path operations, asserting deltas against observed pre-state so each case holds under any layout. It runs locally against a real embedded OpenHouse catalog (harness/openhouse/Env.scala boots OpenHouseLocalServer + the OpenHouse Spark catalog; see run-openhouse.sh and HARNESS-GUIDE.md). The scenario and framework sources are also structured as a publishable Gradle library module (openhouse-spark-delta-harness_2.12) that excludes the embedded-only Env so downstream environments can supply their own adapter. Genuine product or upstream bugs are tagged in Plan.knownBugs with a prose explanation and skipped rather than silently passed, so the suite stays green while documenting the defect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../spark/delta-harness/HARNESS-GUIDE.md | 360 ++++++++ integrations/spark/delta-harness/build.gradle | 48 ++ .../spark/delta-harness/run-openhouse.sh | 101 +++ .../scripts/print-cp.init.gradle | 36 + .../openhouse/BranchWapScenarios.scala | 348 ++++++++ .../harness/openhouse/DmlScenarios.scala | 779 ++++++++++++++++++ .../main/scala/harness/openhouse/Env.scala | 217 +++++ .../harness/openhouse/ForkScenarios.scala | 511 ++++++++++++ .../scala/harness/openhouse/Framework.scala | 332 ++++++++ .../HazardReaderWriterScenarios.scala | 340 ++++++++ .../openhouse/InteractionScenarios.scala | 472 +++++++++++ .../openhouse/MaintControlScenarios.scala | 214 +++++ .../harness/openhouse/MorMaintScenarios.scala | 241 ++++++ .../openhouse/NegativeDdlScenarios.scala | 428 ++++++++++ .../openhouse/NestedTypesScenarios.scala | 217 +++++ .../harness/openhouse/OpenHouseMatrix.scala | 23 + .../main/scala/harness/openhouse/Plan.scala | 282 +++++++ .../scala/harness/openhouse/ScenarioKit.scala | 275 +++++++ .../harness/openhouse/SurfaceScenarios.scala | 566 +++++++++++++ settings.gradle | 2 + 20 files changed, 5792 insertions(+) create mode 100644 integrations/spark/delta-harness/HARNESS-GUIDE.md create mode 100644 integrations/spark/delta-harness/build.gradle create mode 100755 integrations/spark/delta-harness/run-openhouse.sh create mode 100644 integrations/spark/delta-harness/scripts/print-cp.init.gradle create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala diff --git a/integrations/spark/delta-harness/HARNESS-GUIDE.md b/integrations/spark/delta-harness/HARNESS-GUIDE.md new file mode 100644 index 000000000..e90fe3068 --- /dev/null +++ b/integrations/spark/delta-harness/HARNESS-GUIDE.md @@ -0,0 +1,360 @@ +# delta-harness — a guide to grokking the tests + +This is the single document to read to understand what this harness is, how it is built, why it is built +that way, and what it found. It is written for a person picking the harness up cold. If you read only one +file, read this one; `run-openhouse.sh` and the `*.scala` sources are the ground truth beneath it. + +--- + +## 1. What it is, in brief + +`delta-harness` is a self-contained Scala test rig that drives real, customer-facing Spark SQL against a +real embedded OpenHouse catalog and asserts what actually happened to the table. It is not a unit test of +OpenHouse internals. It is a behavioral matrix over the surface a data engineer actually touches: +`DELETE`, `UPDATE`, `MERGE`, `INSERT`, and `OVERWRITE`; copy-on-write versus merge-on-read; DDL; +branching and Write-Audit-Publish; time travel; restore; maintenance procedures; streaming and CDC +readers; the drop-then-undrop lifecycle; and the behaviors specific to LinkedIn's `com.linkedin.iceberg` +1.5.2 fork. + +A few facts set expectations before you read further. + +- The suite runs a few thousand cases in each mode (the in-memory-stub mode and the real-HTS mode), and + every case passes with no divergence between the ORC and Parquet encodings. The guide deliberately does + not quote an exact case count, because that number changes every time a case is added; the exact figure + is whatever the final line of a full run prints. +- A test is written as a typed pipeline (`TableTest[S <: Schema]`). A preparation prefix (create and + seed, or RTAS, or drop and undrop) is composed with an operation suffix (the thing under test), and + every step asserts a delta against the observed pre-state rather than an absolute row set. +- The suite scales by crossing one authored operation against many substrates (file format, partitioning, + copy-on-write versus merge-on-read, replace-lineage, branch, and restored-from-undrop). File format is + a per-case parameter, so most blocks run on both Parquet and ORC automatically. +- The purpose is to find broken feature interactions, not to accumulate green cases. The findings — the + `G`-series product-behavior notes, the `WAP1` note, the fork behaviors, and an error-message + readability audit — are the real output, and the green count only tells you that the tripwires are + still where they were left. + +--- + +## 2. How to run it + +The harness requires JDK 17, because the repository pins Lombok 1.18.20, which does not compile on JDK 21 +or newer. Point the script at a 17 through `JAVA17_HOME`; it also accepts `JAVA_HOME` when that already +points at a 17. + +```bash +export JAVA17_HOME=/usr/lib/jvm/java-17-openjdk-amd64 # or wherever your 17 lives + +./run-openhouse.sh # the full matrix; the last printed line is the case count +./run-openhouse.sh delete parquet # a fast slice (~25s): delete tests, Parquet only +./run-openhouse.sh merge parquet # merge tests on Parquet +./run-openhouse.sh delete.byPredicate # one operation across its layouts +``` + +Each positional argument is an AND-substring filter on the case id, so a case runs only if its id +contains all of the arguments. The match is a substring rather than an exact token, which means +`partitioned` also matches `unpartitioned`. A narrow slice takes roughly 25 seconds end to end, because +the embedded-server and Spark startup dominate while the assertions themselves take milliseconds. You +should iterate on a slice and run the whole matrix only as a final gate. + +Two environment variables change what is exercised. + +| Variable | What it does | +|---|---| +| `HARNESS_REAL_HTS=1` | This boots the real House Table Service as a second in-JVM Spring context and runs the drop-then-undrop blocks (`undrop:*`, `undropAdmin.*`, and the `undropInteract` three-way compositions) against it. When the variable is unset, the harness uses an in-memory stub and skips those blocks, which is why the real-HTS run has more cases than the default run. | +| `ICEBERG_RUNTIME_JAR=` | This is branch-testing mode. It swaps the shaded Iceberg runtime jar on the classpath for a locally built fork-branch-HEAD jar, so the whole suite runs against un-released fork bytecode. The swap is reversible, and it hard-fails when the jar it is asked to replace is not found, so a typo cannot silently leave the release jar in place. | + +`HARNESS_PARALLELISM=N` overrides the worker count, which otherwise defaults to the CPU count; a value of +one or less runs sequentially. + +### What the script does + +`run-openhouse.sh` performs three steps. First, it resolves the OpenHouse classpath through a system +Gradle — the Gradle wrapper cannot download behind the proxy, as noted in the pitfalls below — and caches +the result. Second, it compiles every `.scala` file under `src/main/scala/harness/openhouse/` with +`scalac`. Third, it runs `harness.Main` on JDK 17 with the `--add-opens` flags that Spark 3.5 needs. +Gradle is used only to produce the classpath and OpenHouse's own jars; it does not build the harness. + +--- + +## 3. The mental model, and why a test looks the way it does + +A test is a typed pipeline, `TableTest[S <: Schema]`. The type parameter `S` names the table +implementation the test depends on, and every step references that schema's columns through typed handles +such as `row.get(CoreTable.long0): Long`. The compiler therefore forbids mixing schemas or naming a column +the schema does not declare, so a whole class of "the test drifted from the table shape" bug is impossible +by construction. + +Four ideas do all of the work. + +1. **A schema is columns only.** `CoreTable` has one column per common type plus a `datepartition` string + in the form `YYYY-MM-DD-HH`, while `NestedTable` and `TypesTable` cover struct and complex types and + type-edge coverage respectively. Each `Column[T]` carries its Scala type and a deterministic + `literalAt(rowIndex)` generator, so seeding is reproducible and schema-checked. + +2. **A preparation prefix and an operation suffix compose with `andThen`.** An operation — the thing under + test, such as a `DELETE`, a `MERGE`, or an `ADD COLUMN` — is authored headless, meaning it assumes a + seeded table and does not create one. The run composes a preparation before it. Because the preparation + and the operation are the same kind of object, you can swap the preparation without touching the + operation, and that is the entire trick that lets the whole DML catalog be re-run on an RTAS'd table, a + branch-routed table, or a table that has been through a real drop-then-undrop round trip. The operation + set is authored once, and the substrate set multiplies it. + +3. **The layout axis is file format crossed with partitioning.** A `Layout` is expressed as a literal + `CREATE` statement. There are six base layouts — the two partitionings crossed with Parquet, ORC, and + Avro — plus merge-on-read variants, plus dedicated single-data-file layouts used as a physical + copy-on-write versus merge-on-read discriminator. On such a layout, a strict-subset delete on one data + file must produce a position-delete file under merge-on-read and must not produce one under + copy-on-write, and the harness asserts exactly that against the `.delete_files` metadata table. + +4. **Assertions are deltas, never absolutes.** Each step's validation thunk receives a `StepView` that + carries `before` and `after` row snapshots along with `snapshotsBefore` and `snapshotsAfter` commit + counts. Every operation asserts a change — two rows fewer, one new snapshot, this key now excluded — so + the identical assertion holds under any layout, any seed size, and any substrate. This is what makes a + single authored operation valid across the whole substrate cross. + +The parallel runner, `harness.Main`, runs cases on a worker pool, and each worker gets its own +`spark.newSession()` with a separate `SQLConf`, so the session-global state that some tests mutate — such +as `spark.wap.branch`, `spark.wap.id`, and changelog temp views — never leaks between cases. Results are +collected and printed in the original case order, so the output is identical to a sequential run. Each +case owns its own table through an atomic counter, so the cases are independent. + +Known product bugs are tagged rather than skipped into silence. `Plan.knownBugs` maps a case-id substring +to a reason, and a matching case is reported as `SKIP (bug: …)`. This is how a genuine defect is deferred +without either failing the suite or silently pretending it passed. + +--- + +## 4. Where things live + +The harness is split by concern, and every file declares `package harness`, so the directory name does +not affect the package. Open the file whose concern matches what you are after. + +| File | What it holds | +|---|---| +| `Framework.scala` | This file holds the DSL and the plumbing: `Ctx`, the REST and `HtsAdmin` clients, `Outcome` and `Check`, the `Column`/`Schema`/`Rows` vocabulary, the three tables, `RowGenerator`, `StepView` and `Step`, and `TableTest` itself. Read it first to learn the vocabulary. | +| `ScenarioKit.scala` | This is the shared kit that every test group builds on. It holds `Layout` and the layout lists, all of the `createAndSeed*` preparations, the format-multiplex hooks (`seedFmt` and `withSeedFmt`), and the cross-cutting helpers. Every `*Scenarios` trait extends it, and any helper used by more than one trait belongs here. | +| `DmlScenarios.scala` | This is the core DML surface. It holds the read, delete, update, merge, and insert/append/overwrite operation catalog; the `operations`, `partitionedOperations`, and `mutationOperations` lists; the DDL-by-consumer battery; the ADD COLUMN family; and the physical copy-on-write versus merge-on-read discriminator. | +| `NestedTypesScenarios.scala` | This holds nested and complex-type coverage, type-edge coverage, and partition transforms together with partition-evolution rejections. | +| `MorMaintScenarios.scala` | This holds merge-on-read delete-file coexistence (operations on a table that already carries a live position delete), merge-on-read maintenance folds, merge-on-read modality hazards, and merge-on-read crossed with branch merge. | +| `MaintControlScenarios.scala` | This holds time travel, restore and rollback, the maintenance procedures such as `expire_snapshots` and `rewrite_data_files`, the REST control-plane operations for lock and unlock, and the undrop admin lifecycle. | +| `ForkScenarios.scala` | This holds the `com.linkedin.iceberg` fork-behavior pins; the fork commits themselves are tabulated in section 8. | +| `BranchWapScenarios.scala` | This holds branching and Write-Audit-Publish: the undrop three-way compositions, the direct-branch operations, and the branch and WAP battery, which covers staged-write publish visibility and the systematic branch-DDL leak. | +| `NegativeDdlScenarios.scala` | This holds the typed negatives and contract pins together with the DDL phases: properties, sort order, rename, namespace, policy, CTAS and RTAS, column tags and ACL, and encryption. | +| `InteractionScenarios.scala` | This holds the three-way compositions where the interesting behavior lives: DDL crossed with history, RTAS crossed with history, lineage, and property-merge, branch crossed with history and maintenance, and the composite branch-expiration-merge defect. | +| `SurfaceScenarios.scala` | This holds surface completion: the error-message readability guard, branch leaks, WAP negatives, streaming and CDC, procedures, metadata tables, concurrency invariants, schema-evolution edges, write-path configs, and expected-unsupported pins. | +| `HazardReaderWriterScenarios.scala` | This holds the hazard and modality interactions (expired checkpoints, RTAS wiping tags, rename breaking consumers) and the reader-by-writer-class battery (changelog, incremental, and streaming over both copy-on-write and merge-on-read). | +| `Plan.scala` | This is the assembly. `object Plan` is where substrates crossed with operations become the actual `Case` list, where `crossFmt` doubles a block across Parquet and ORC, and where `knownBugs` lives. If you want to know what actually runs, read `Plan.cases`. | +| `OpenHouseMatrix.scala` | This mixes the domain traits into `object Scenarios`. The `extends` clause here is the authoritative order in which the traits' `val`s initialize, as explained in section 6. | +| `Env.scala` | This handles boot and run: the embedded OpenHouse server wiring in `OpenHouseEnv`, the embedded real HTS in `HtsEnv` and `HtsBootApp`, the retrying `Runner`, and `Main`. | + +--- + +## 5. The axes, and why the honest target is well below the naive product + +You can think of the suite as substrates crossed with operations crossed with consumers. + +- The operations are the DML catalog (authored once as explicit literals in `DmlScenarios.operations`), + together with the DDL operations and the procedures. +- The substrates are the preparations — plain create-and-seed, RTAS'd (replace-lineage), branch-routed + through `spark.wap.branch`, restored-from-drop on the real HTS, schema-evolved, sort-ordered, and + merge-on-read — and each of them multiplies the operation catalog. +- The consumers answer a question: after a state-changing DDL, does each reader — plain scan, time + travel, changelog, incremental, and streaming — still work? +- File format is a per-case parameter, described below, so blocks double across Parquet and ORC for free. + +The naive product is much larger than what actually runs, because a large fraction of the cells would be +vacuous, and the harness refuses to inflate its count with them. Three arguments carry most of that +reduction. First, a read or insert on a delete-free merge-on-read table is byte-identical to +copy-on-write, because there are no delete files to apply and append is mode-independent, so the real +merge-on-read surface is mutation operations crossed with merge-on-read, plus delete-file coexistence, +plus reads with live deletes, rather than the whole operation catalog crossed with merge-on-read. Second, +RTAS and branch commute with file format, because refs and metadata never touch file encoding, so those +legs run on Parquet only rather than across all three formats. Third, a DDL-by-consumer cross over a +rejected or one-shot DDL has no post-state to consume, so only state-changing DDL crossed with real +consumers is non-vacuous. + +When an estimate turns out to be inflated by vacuous cells, the honest move is to correct the estimate in +the open rather than to chase the vacuous number. File format, however, is not a vacuity axis, as the next +section explains. + +### Format multiplex, and why "format-inert" is a hypothesis rather than an assumption + +Every table-creating block reads a per-case thread-local seed format, `seedFmt`, and `Plan.crossFmt` wraps +a block so that it runs once per format in `dataFormats` (Parquet and ORC), setting `seedFmt` around each +case. The mechanism is safe because cases run sequentially per worker. The point is a philosophical one: +you do not bake a file format into a test. Whether a behavior is format-independent is something this +harness verifies rather than assumes, because the fork carries patched ORC paths and the replace-path +findings showed metadata surprises. Only table-less operations, which issue no `CREATE`, have no format +axis. This is why the summary above says there is no divergence between ORC and Parquet: that is a checked +result, not a design assumption. + +--- + +## 6. Design decisions and pitfalls + +The catalog wiring is copied rather than extended. `OpenHouseEnv` composes an embedded +`OpenHouseLocalServer` together with Spark-catalog configuration lifted from `OpenHouseLocalServer` and +`TestSparkSessionUtil` as components, so no OpenHouse test class is subclassed and no existing test is +altered. The harness is a bolt-on observer. + +The undrop leg drives a real HTS through a single backward-compatible production change. A customer `DROP` +hard-codes `purge=true`, so a customer can never populate the soft-deleted store, and the embedded +server's default `HouseTableRepository` is an in-memory stub, so an undrop test against it would test the +stub rather than production. For that reason, `HARNESS_REAL_HTS=1` boots the genuine House Table Service +as a second in-JVM Spring context and points the tables server at it. The only production-code change is +one `@ConditionalOnProperty` on `HouseTablesH2Repository`, with `havingValue="true"` and +`matchIfMissing=true`, so that the stub can be switched off. The change is fully backward compatible, +because an absent property leaves the stub in place exactly as before, and everything else is on the +harness side. + +Assertions are deltas, and rejections are pins. A negative test asserts a rejection-message substring and, +following the readability audit in section 7, also asserts that the message is not a raw stacktrace, an +`[INTERNAL_ERROR]`, or a bare NullPointerException. These rejections are tripwires rather than contracts, +which means that if OpenHouse later supports the operation, the pinned test is meant to flip and be +updated rather than to keep passing silently. The goal is to catch a change in behavior in either +direction. + +The trait layout determines the initialization order. `object Scenarios`, in `OpenHouseMatrix.scala`, is +assembled by mixing the domain traits on top of `ScenarioKit` through an explicit `extends … with …` +clause, and that clause is the authoritative order. `ScenarioKit` linearizes first, so its shared `val`s +initialize before any domain trait references them, and the domain traits then initialize in the order +written. A helper used by more than one trait must live in `ScenarioKit`, because a reference to a sibling +trait's member will not resolve and the compiler will tell you. As long as the `extends` clause and the +member order within each trait stay stable, initialization stays deterministic. + +Several pitfalls are specific to this harness. The first is that only JDK 17 works, because Lombok 1.18.20 +in the repository does not compile on 21 or newer. The second is that the Gradle wrapper cannot download +behind the proxy and returns a 403, so you must use a system Gradle through `GRADLE_BIN`; the script +caches the resolved classpath after the first run. The third is that Avro required a classpath fix, +because a duplicate shaded and unshaded Iceberg on the classpath broke Avro until a dependency exclusion +was added in `scripts/print-cp.init.gradle`. The fourth is that file format is a hypothesis and the format +policy is additive: you should not optimize a block down to Parquet only on the grounds that it should be +format-inert, because that is precisely the assumption the harness exists to check, and every +table-creating block covers at least Parquet and ORC while the three-format blocks keep Avro. Adding +coverage is additive and never removes an existing format. + +--- + +## 7. What the harness found + +The following are product-behavior findings, and each is demonstrated live by named cases. + +The first group is guard gaps, where an operation that can corrupt or mislead is not blocked. + +- **G2 is that RTAS on a locked table succeeds.** The lock rejects an `UPDATE`, and then `CREATE OR + REPLACE` replaces the locked table, taking it from three rows to two, because the replace path never + reaches the lock check. This is a data-loss-class gap with the cleanest one-line fix, and it is + demonstrated by `interact.rtas.onLockedTable`. +- **G8 is that table-global DDL "on a branch" silently mutates main.** With `spark.wap.branch` set, `ADD + COLUMN`, `SET TBLPROPERTIES`, and `WRITE ORDERED BY` change main's schema, properties, and sort order, + because there is no branch dimension anywhere in the metadata commit path. It is demonstrated by + `branch.ddlLeak.*`. +- **G9 and G10 are that the replace path dodges the update-path guards.** RTAS can change the partition + spec and drop columns that `ALTER` rejects (G9), and RTAS silently wipes the `policies` plane, so that + retention, sharing, and PII column tags are gone after a replace while user properties survive (G10). + G10 is the highest-severity member of the replace-path cluster, and both are demonstrated by + `interact.rtas.*` and `hazard.rtas.wipesColumnTags`. +- **G11 is that a routine snapshot expiration destroys merge connectivity between live refs.** Expiration + retention is per-ref and head-anchored, so nothing protects the ancestry between live refs. The + consequences are all demonstrated: a `fast_forward` merge is spuriously rejected with "main is not an + ancestor" even though main never moved; a cherry-pick silently loses the expired intermediate commit, + which is a partial merge that presents as success and is the worst variant; the branch becomes + permanently unmergeable; and staged WAP snapshots are expired before publish. OpenHouse's default + three-day expiration makes all of this automatic, and it is demonstrated by + `interact.branch.expireMerge.*`. +- **G12 is that a lock starves maintenance for its whole lifetime while not stopping RTAS**, which makes + it the mirror of G2. Scheduled expiration and compaction hit the lock gate and fail every cycle, so + snapshots and files accrete unboundedly. It is demonstrated by `hazard.lock.starvesMaintenance`. +- **G3 through G7 are the lower-severity gaps**: replica-path spec divergence, free WAP and replace + toggling, ref preservation, format-version on update, and the all-or-nothing `skipEligibilityCheck` on + the replica path. G1 was investigated and then withdrawn, because the replication snapshot-walk turned + out to be sound. + +The second group is behavior and limitation findings. + +- **G13 is that CDC changelog is unsupported over a merge-on-read table after an UPDATE or MERGE**, which + fails with "Delete files are currently not supported in changelog scans". Merge-on-read delete-only and + all copy-on-write cases work, but merge-on-read update and merge — the shapes a merge-on-read table + exists to optimize — break CDC silently. This is a stock Iceberg 1.5 limitation, and it is demonstrated + by `readerWriter.changelog.{update,merge}.mor`. +- **G14 is that `rewrite_data_files` leaves a dangling position delete on a merge-on-read table.** + Compaction applies the delete, so the row set is correct, but it does not fold out the now-dangling + delete file until `rewrite_position_delete_files` runs. This is stock Iceberg 1.5, which has no + `remove-dangling-deletes` yet. It is classified as a pin rather than a bug, because the recovery path is + verified to work by `maint.mor.rewritePositionDeleteFolds` across the merge-on-read formats. The + operational takeaway is that, on merge-on-read under 1.5, you should pair `rewrite_data_files` with + `rewrite_position_delete_files`. +- **WAP1 is that a staged DELETE (with `spark.wap.id` set) is not honored by WAP and publishes to main + immediately.** In the same block, staged `INSERT`, `OVERWRITE`, `UPDATE`, and `MERGE` all stage + correctly. The consequence is that an operator relying on WAP to stage and review a deletion gets an + immediate, un-reviewed publish. It is demonstrated by `wapStaged.delete.bypassesWap`. + +There is also an error-message readability finding. A separate sweep grades rejection messages as good, +acceptable, or bad for a non-expert SQL user. The systemic result is that the client drags the entire +error body, including a stacktrace, into the message, so that even a good server sentence reaches the user +as `400 , {json + java frames}`; surfacing only `ErrorResponseBody.message` would upgrade nearly every 4xx +path at once. + +Finally, the tagged and deferred defects are the ones that appear in `Plan.knownBugs` and are reported as +`SKIP (bug: …)`. They are a nested-field DELETE optimizer NullPointerException, a RENAME COLUMN that is a +silent no-op (a genuine OpenHouse regression traced to server commit #558), and encryption that writes +plaintext because the KMS plugin is out of the repository. + +> The exhaustive ledgers behind this section — the findings with code citations, the fork-commit audit, +> the tagged-defect ledger, and the dated run log — live alongside the harness in the pull request that +> developed it, and not necessarily in this tree. You do not need them to grok the tests, so reach for +> them only when you want the evidence behind a specific claim made here. + +--- + +## 8. The `com.linkedin.iceberg` fork + +The harness runs against fork bytecode, namely `com.linkedin.iceberg:iceberg-spark-runtime-3.5_2.12`, +rather than against Apache Iceberg. The tested behaviors are listed below. Each one is pinned by a +`fork.*` case, and each is keyed to the fork's own commit number or the upstream-Iceberg issue number. + +| Commit | The behavior the fork changes | Pinned by | +|---|---|---| +| `#249` | The partitioned default write distribution becomes NONE, where Apache uses HASH, which produces more and smaller files. | `fork.partitionDist.default` | +| `#229` | A `write.delete-file-replication` toggle is added for merge-on-read delete files. | `fork.deleteFileReplication` | +| `#219` | A per-output-file replication factor is stamped by the delete-file write path. | `fork.fileReplicationFactor` | +| `#228` | A `spark.sql.iceberg.split-size` read split-size property is added. | `fork.splitSize` | +| `#233` | Compaction bin-pack weight is computed by data-file length and ignores delete size. | `fork.binPackByLength` | +| `#189` | A budgeted rewrite is ordered by file-sequence-number. | `fork.compactionOrder` | +| `#251` | Column-default APIs and `SchemaParser` serialization are added; this exists on the branch HEAD only and is tabled. | `fork.colDefault.*` | + +The `#251` story is worth understanding, because it is a good example of the harness resisting an +overclaim. `#251` backports column defaults to the API and core, but there is no read-application code and +no Spark wiring in the open fork, because `SparkTable` does not implement `SupportsColumnDefaultValue`. As +a result, over OSS Spark, `ADD COLUMN … DEFAULT 5` parses, but the default is not written into the Iceberg +schema, old rows read NULL, and an INSERT that omits the column is rejected. The serialization does round +trip on a branch build. The harness pins exactly that — the observable OSS-Spark DDL behavior and the +serialization — and it explicitly does not claim the feature is broken, because read-application may exist +in LinkedIn's private Spark, which this harness cannot see. A whole-suite branch-versus-release run, +performed through `ICEBERG_RUNTIME_JAR`, showed no correctness deltas. + +--- + +## 9. Adding a test + +Adding a test follows a short recipe. First, pick the schema, which is `CoreTable` unless you need nesting +or type edges. Second, author the operation headless, as a `TableTest` step that assumes a seeded table +and asserts a delta through its `StepView`, using `view.before` and `view.after`, `snapshotsBefore` and +`snapshotsAfter`, and the metadata tables such as `.delete_files` and `.snapshots`. Third, put it in the +trait whose concern matches, as described in section 4, and if it needs a helper used by another trait, +add that helper to `ScenarioKit`. Fourth, wire it into `Plan` by adding it to the relevant list, and use +`crossFmt(...)` if it creates a table, so that it runs on both Parquet and ORC; do not bake a single +format into it. Fifth, if it exercises a real product bug that you are deferring, tag it in +`Plan.knownBugs` with a reason, and never let it pass or skip silently. Finally, run the slice and then the +full gate, and confirm that the count moved by what you expect and that nothing else regressed. + +--- + +## 10. Decisions worth knowing + +File format is a per-case parameter rather than a baked-in constant, because un-baking the format is what +lets a test multiplex and compose; whether a behavior is format-inert is verified rather than assumed. The +dangling merge-on-read delete described in G14 is a pin rather than a bug, because +`rewrite_position_delete_files` is verified to recover it, and merge-on-read under 1.5 simply requires that +extra maintenance step. Encryption and KMS support is deferred, because the plugin is out of the +repository, so the plaintext behavior is pinned and the intended-behavior assertion waits for the plugin. diff --git a/integrations/spark/delta-harness/build.gradle b/integrations/spark/delta-harness/build.gradle new file mode 100644 index 000000000..628d62056 --- /dev/null +++ b/integrations/spark/delta-harness/build.gradle @@ -0,0 +1,48 @@ +plugins { + id 'openhouse.java-minimal-conventions' + id 'openhouse.maven-publish' + id 'scala' +} + +// The delta-harness behavioral matrix, published as a portable Scala library so it can be authored +// and run ONCE and consumed in two homes: +// 1. LOCALLY in this repo against the embedded catalog (see Env.scala + run-openhouse.sh), and +// 2. As an acceptance test in a downstream environment against a real cluster, which depends on this +// published artifact and supplies only its own environment adapter. +// +// Only the portable scenario/framework sources are published. Env.scala boots the embedded OpenHouse +// server + House Table Service and pulls in Spring/housetables/tables-test-fixtures, so it is EXCLUDED +// from the published library and compiled separately for the local run (it depends on this library). + +ext { + icebergVersion = rootProject.ext.iceberg_1_5_version + sparkVersion = '3.5.2' + scalaLibVersion = '2.12.18' +} + +sourceSets { + main { + scala { + srcDirs = ['src/main/scala'] + // Embedded-only boot/run wiring — not part of the portable, publishable library. + exclude 'harness/openhouse/Env.scala' + } + } +} + +dependencies { + implementation "org.scala-lang:scala-library:${scalaLibVersion}" + + // Compile-only: the consumer (Env.scala locally, or a downstream environment adapter) provides the + // actual Spark, Iceberg and OpenHouse client runtime. The library jar carries only the harness classes. + compileOnly("org.apache.spark:spark-sql_2.12:${sparkVersion}") { + exclude group: 'io.netty' + } + compileOnly("com.linkedin.iceberg:iceberg-spark-runtime-3.5_2.12:${icebergVersion}") { + exclude group: 'io.netty' + } + // Provides com.linkedin.openhouse.javaclient.* (WebClientResponseWithMessageException, etc.). + compileOnly(project(path: ':integrations:spark:spark-3.5:openhouse-spark-3.5-runtime_2.12', configuration: 'shadow')) +} + +jar.enabled = true diff --git a/integrations/spark/delta-harness/run-openhouse.sh b/integrations/spark/delta-harness/run-openhouse.sh new file mode 100755 index 000000000..047c9d086 --- /dev/null +++ b/integrations/spark/delta-harness/run-openhouse.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# Build + run the delta-harness DELETE slice against the REAL OpenHouse catalog +# (embedded OpenHouseLocalServer + OpenHouseCatalog). +# +# Requirements: +# - JDK 17 (the OpenHouse build pins Lombok 1.18.20, which is incompatible with JDK 21+). +# Set JAVA17_HOME, or the script uses $JAVA_HOME if it is a 17. +# - A Gradle able to build the repo (system gradle 8.x works; the pinned 7.6.2 wrapper +# may be blocked from downloading in restricted networks). +# - Scala 2.12.18 compiler jars in the local Maven cache (~/.m2), or adjust SCALAC_CP. +# +# Real-HTS mode (HARNESS_REAL_HTS=1): boots the REAL embedded House Table Service as a 2nd Spring +# context and points the tables server at it (replacing the in-memory stub), and enables the undrop +# preparation axis + undropAdmin lifecycle cases (soft-delete/restore/purge). Requires the housetables +# classes on the classpath — run once with FORCE_CP=1 after adding them (print-cp.init.gradle already +# pulls :services:housetables). See HTS-EMBED-PLAN.md / HTS-EMBED-IMPL.md. Default (unset) uses the stub. +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +HERE="$(pwd)" +WORK="${TMPDIR:-/tmp}/delta-harness-oh" +mkdir -p "$WORK" + +JDK17="${JAVA17_HOME:-${JAVA_HOME:?set JAVA17_HOME to a JDK 17}}" +GRADLE="${GRADLE_BIN:-gradle}" +M2="${HOME}/.m2/repository/org/scala-lang" +SCALAC_CP="$M2/scala-compiler/2.12.18/scala-compiler-2.12.18.jar:$M2/scala-reflect/2.12.18/scala-reflect-2.12.18.jar:$M2/scala-library/2.12.18/scala-library-2.12.18.jar" + +# Classpath resolution is the slow part (~82s of gradle). It only changes when OpenHouse deps +# change, so we cache it in $WORK/oh-cp.txt and reuse it for fast inner-loop iteration. Force a +# fresh resolve with FORCE_CP=1 (do this after pulling dep changes or the first run in a session). +if [[ "${FORCE_CP:-0}" != "1" && -s "$WORK/oh-cp.txt" ]]; then + echo ">> reusing cached OpenHouse classpath ($WORK/oh-cp.txt) — set FORCE_CP=1 to re-resolve" +else + echo ">> resolving OpenHouse itest runtime classpath (builds the runtime uber jar + fixtures)" + ( cd "$REPO_ROOT" && "$GRADLE" -Dorg.gradle.java.home="$JDK17" -DcpOut="$WORK/oh-cp.txt" \ + --init-script "$HERE/scripts/print-cp.init.gradle" \ + :integrations:spark:spark-3.5:openhouse-spark-3.5-itest:printHarnessCp --console=plain ) +fi +OHCP="$(cat "$WORK/oh-cp.txt")" + +# ── Test-the-BRANCH override ──────────────────────────────────────────────────────────────────── +# The harness normally resolves the PUBLISHED com.linkedin.iceberg:iceberg-spark-runtime-3.5_2.12 +# (e.g. 1.5.2.15) — a Maven-Central snapshot that can LAG the openhouse-1.5.2 branch HEAD (it predates +# #251 column-defaults, etc.). To test the actual BRANCH, build the shaded runtime jar from branch HEAD +# (`gradle :iceberg-spark:iceberg-spark-runtime-3.5_2.12:shadowJar`) and point this at it: +# ICEBERG_RUNTIME_JAR=/workspace/iceberg/spark/v3.5/spark-runtime/build/libs/ ./run-openhouse.sh +# That single shaded jar carries all of iceberg api+core+spark, so swapping it makes the whole harness +# JVM (Spark side + embedded server) run the branch. Unset → back to the published release. Reversible. +if [[ -n "${ICEBERG_RUNTIME_JAR:-}" ]]; then + [[ -f "$ICEBERG_RUNTIME_JAR" ]] || { echo "!! ICEBERG_RUNTIME_JAR not found: $ICEBERG_RUNTIME_JAR" >&2; exit 1; } + # How many spark-runtime-3.5 entries does the resolved cp actually have? If zero, the pattern no longer + # matches (module/version rename, jar absent) and swapping would SILENTLY leave the published jar in place + # — so fail loudly instead of pretending we tested the branch. + matches="$(printf '%s' "$OHCP" | tr ':' '\n' | grep -cE '/iceberg-spark-runtime-3\.5_2\.12-[^/]*\.jar' || true)" + if [[ "$matches" -eq 0 ]]; then + echo "!! ICEBERG_RUNTIME_JAR set but no iceberg-spark-runtime-3.5_2.12 jar found on the resolved classpath" >&2 + echo "!! (pattern changed, or cp cache is stale — re-run with FORCE_CP=1). Refusing to run the PUBLISHED jar." >&2 + exit 1 + fi + # Replace the resolved spark-runtime-3.5 jar path (any version) with the override. Use a `|` sed delimiter + # and a literal-ized replacement so `&`/`#`/`/` in the path are not interpreted. + repl="$(printf '%s' "$ICEBERG_RUNTIME_JAR" | sed -e 's/[&|\\]/\\&/g')" + OHCP="$(printf '%s' "$OHCP" | tr ':' '\n' \ + | sed -E "s|.*/iceberg-spark-runtime-3\.5_2\.12-[^/]*\.jar|$repl|" \ + | paste -sd ':' -)" + inserted="$(printf '%s' "$OHCP" | tr ':' '\n' | grep -Fc "$ICEBERG_RUNTIME_JAR" || true)" + [[ "$inserted" -ge 1 ]] || { echo "!! branch-mode swap produced 0 override entries — aborting" >&2; exit 1; } + echo ">> [BRANCH MODE] iceberg-spark-runtime swapped ($matches slot(s)) -> $ICEBERG_RUNTIME_JAR" + echo ">> [BRANCH MODE] override entries on cp: $inserted" +fi + +echo ">> compiling harness (scala 2.12) against the OpenHouse classpath" +mkdir -p "$WORK/classes" +# The harness is split across several .scala files (Framework / Scenario traits / Plan / Env), +# all in `package harness`. Compile every source under src/main/scala together so cross-file +# references resolve (order is irrelevant to scalac — it compiles the whole compilation unit set). +mapfile -t SCALA_SRCS < <(find "$HERE/src/main/scala/harness/openhouse" -name '*.scala' | sort) +echo ">> ${#SCALA_SRCS[@]} source files" +"$JDK17/bin/java" -cp "$SCALAC_CP" scala.tools.nsc.Main \ + -classpath "$OHCP" -d "$WORK/classes" \ + "${SCALA_SRCS[@]}" + +echo ">> running on JDK 17 (embedded OpenHouse server + OpenHouse catalog)" +OPENS=( + --add-opens=java.base/java.lang=ALL-UNNAMED + --add-opens=java.base/java.lang.invoke=ALL-UNNAMED + --add-opens=java.base/java.io=ALL-UNNAMED + --add-opens=java.base/java.net=ALL-UNNAMED + --add-opens=java.base/java.nio=ALL-UNNAMED + --add-opens=java.base/java.util=ALL-UNNAMED + --add-opens=java.base/java.util.concurrent=ALL-UNNAMED + --add-opens=java.base/sun.nio.ch=ALL-UNNAMED + --add-opens=java.base/sun.security.action=ALL-UNNAMED + --add-opens=java.base/sun.util.calendar=ALL-UNNAMED +) +SCALA_LIB="$M2/scala-library/2.12.18/scala-library-2.12.18.jar" +# Args are passed through as case-id filters (AND). E.g. `run-openhouse.sh delete parquet` +# runs just the delete tests on parquet — a ~25s inner loop. No args runs the full matrix. +exec "$JDK17/bin/java" "${OPENS[@]}" -Dio.netty.tryReflectionSetAccessible=true \ + -cp "$WORK/classes:$SCALA_LIB:$OHCP" harness.Main "$@" diff --git a/integrations/spark/delta-harness/scripts/print-cp.init.gradle b/integrations/spark/delta-harness/scripts/print-cp.init.gradle new file mode 100644 index 000000000..5b5bee049 --- /dev/null +++ b/integrations/spark/delta-harness/scripts/print-cp.init.gradle @@ -0,0 +1,36 @@ +// Resolves the harness runtime classpath from the OpenHouse spark itest module and writes it to +// the file named by -DcpOut. +// +// The itest classpath legitimately pulls two copies of Iceberg into one JVM: the shaded +// iceberg-spark-runtime fat jar (client side) and the unshaded iceberg-{api,common,core,data} +// jars (embedded server side). On the Avro data path those two Avro namespaces collide +// (ClassCastException, see FINDINGS.md F1). We resolve that the proper way — a dependency +// exclusion so the graph carries a single Iceberg — rather than filtering resolved jars by hand. +// The shaded fat jar provides all org.apache.iceberg.* classes, so excluding the unshaded modules +// is safe. This exclusion applies only to this classpath-extraction invocation. +allprojects { + if (path == ':integrations:spark:spark-3.5:openhouse-spark-3.5-itest') { + afterEvaluate { + configurations.testRuntimeClasspath { + exclude group: 'com.linkedin.iceberg', module: 'iceberg-core' + exclude group: 'com.linkedin.iceberg', module: 'iceberg-api' + exclude group: 'com.linkedin.iceberg', module: 'iceberg-common' + exclude group: 'com.linkedin.iceberg', module: 'iceberg-data' + } + // HTS-embed (Option A): pull the REAL House Table Service classes + // (UserTablesServiceImpl, controllers, JDBC repos, api-spec model) onto the harness + // classpath so the harness can boot a real HTS as a 2nd Spring context and point the + // embedded tables server's HouseTableRepositoryImpl at it. Only needed for the real-HTS + // mode; the default stub path does not use these classes. Same single-Iceberg exclusion + // above covers housetables' transitive unshaded iceberg. + dependencies.add('testImplementation', project(':services:housetables')) + } + tasks.register('printHarnessCp') { + doLast { + def cp = configurations.testRuntimeClasspath.resolve().collect { it.absolutePath } + new File(System.getProperty('cpOut')).text = cp.join(':') + println "WROTE ${cp.size()} classpath entries" + } + } + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala new file mode 100644 index 000000000..86df19c88 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala @@ -0,0 +1,348 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +trait BranchWapScenarios extends ScenarioKit { + import Rows._ + + // ── Undrop 3-way compositions (Block 9, real HTS only) — restore's state-preservation, per feature ── + // The undrop:* battery proves the whole op catalog works post-restore. These are pointed 3-way + // chains that set up a SPECIFIC feature's state (branch / snapshot history / evolved schema), + // destroy via soft-delete→restore, then consume that exact feature — the direct modality check that + // restore's destruction set does not intersect refs / lineage / schema. + + // A pre-existing branch must survive the drop→undrop round-trip. + def interactUndropBranchSurvives(ctx: Ctx): Unit = { + val (table, db, tbl) = undropSeed(ctx, "t_ud_branch") + ctx.spark.sql(s"ALTER TABLE $table CREATE BRANCH b") + ctx.spark.sql(s"INSERT INTO $table.branch_b ${RowGenerator.valuesClause(Core, 2)}") // branch diverges: 3+2=5 + softDeleteRestore(ctx, db, tbl) + assert(ctx.spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "main row set changed across undrop") + assert(ctx.spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'b'").collect()(0).getLong(0) == 5, "branch 'b' did not survive undrop") + ctx.spark.sql(s"DROP TABLE IF EXISTS $table") + } + + // Snapshot history (time travel) must survive restore. + def interactUndropTimeTravelSurvives(ctx: Ctx): Unit = { + val (table, db, tbl) = undropSeed(ctx, "t_ud_tt") + val firstSnap = ctx.spark.sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getLong(0) + ctx.spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 2)}") // 2nd snapshot: 5 rows + softDeleteRestore(ctx, db, tbl) + assert(ctx.spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 5, "current state changed across undrop") + assert(ctx.spark.sql(s"SELECT count(*) FROM $table VERSION AS OF $firstSnap").collect()(0).getLong(0) == 3, + "pre-restore snapshot not time-travellable after undrop (lineage lost)") + ctx.spark.sql(s"DROP TABLE IF EXISTS $table") + } + + // Evolved schema must survive restore, and the restored table must still accept the evolved shape. + def interactUndropSchemaSurvives(ctx: Ctx): Unit = { + val (table, db, tbl) = undropSeed(ctx, "t_ud_schema") + ctx.spark.sql(s"ALTER TABLE $table ADD COLUMN extra int") + ctx.spark.sql(s"INSERT INTO $table VALUES (CAST(9 AS BIGINT), 9, 'row-9', 9.5, false, '2024-01-09-08', 99)") + softDeleteRestore(ctx, db, tbl) + assert(ctx.spark.sql(s"SELECT extra FROM $table WHERE ${Core.long0.columnName} = 9").collect()(0).getInt(0) == 99, + "evolved column value lost across undrop") + ctx.spark.sql(s"INSERT INTO $table VALUES (CAST(10 AS BIGINT), 10, 'row-10', 10.5, true, '2024-01-10-09', 100)") + assert(ctx.spark.sql(s"SELECT count(*) FROM $table WHERE extra IS NOT NULL").collect()(0).getLong(0) == 2, + "restored table did not accept the evolved schema for new writes") + ctx.spark.sql(s"DROP TABLE IF EXISTS $table") + } + + val undropInteractOps: List[(String, Ctx => Unit)] = List( + "interact.undrop.branchSurvives" -> interactUndropBranchSurvives, + "interact.undrop.timeTravelSurvives" -> interactUndropTimeTravelSurvives, + "interact.undrop.schemaSurvives" -> interactUndropSchemaSurvives + ) + + // ── Branching / WAP (format-agnostic → parquet only; behavior-focused, not matrixed) ───────── + // A CoreTable row literal for branch writes (long,int,string,double,boolean,datepartition). + + // B1(a) direct branch ops (no WAP needed): write to t.branch_b, read it via VERSION AS OF 'b'; + // main stays isolated. + val branchDirectIsolation: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("branch.direct.create")(t => s"ALTER TABLE $t CREATE BRANCH b")() + .step("branch.direct.isolation") { (spark, table) => + spark.sql(s"INSERT INTO $table.branch_b VALUES ${coreRow(99, "branch")}") + val onBranch = spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'b'").collect()(0).getLong(0) + val onMain = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) + assert(onBranch == 4, s"branch b should have 4 rows, got $onBranch") + assert(onMain == 3, s"main should be unchanged at 3, got $onMain") // isolation + }() + + // B1(b) spark.wap.branch conf: with write.wap.enabled, the conf routes BOTH reads and writes to the + // branch transparently; unsetting reverts to main. + val branchWapConfRouting: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("branch.wapconf.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .sql("branch.wapconf.create")(t => s"ALTER TABLE $t CREATE BRANCH wapbr")() + .step("branch.wapConf.routing") { (spark, table) => + spark.conf.set("spark.wap.branch", "wapbr") + val onBranch = + try { + spark.sql(s"INSERT INTO $table VALUES ${coreRow(99, "wap")}") // routed to branch + spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) // reads branch + } finally spark.conf.unset("spark.wap.branch") + assert(onBranch == 4, s"on-branch read should see 4, got $onBranch") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "main leaked") + }() + + // B2 WAP stage → publish: a staged write (spark.wap.id) does NOT advance main; cherrypick publishes it. + val wapStagePublish: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("wap.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("wap.stagePublish") { (spark, table) => + spark.conf.set("spark.wap.id", "w1") + try spark.sql(s"INSERT INTO $table VALUES ${coreRow(99, "staged")}") + finally spark.conf.unset("spark.wap.id") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "staged write leaked to main") + val stagedId = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = 'w1'").collect()(0).getLong(0) + spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', $stagedId)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "publish did not advance main") + }() + + // ── WAP mega-axis Stage C — staged-WAP write surface (stage → publish visibility) ──────────── + // The op is written as a STAGED snapshot (spark.wap.id): it must NOT advance main; assert main is + // unchanged pre-publish, then cherrypick_snapshot PUBLISHES it and main reflects it. This is the + // Phase-29 "T2 staged" target. Format-multiplexed by crossFmt (seedFmt-aware create). + private def wapStagedWrite(label: String)(write: String => String)(preRows: Long, postRows: Long): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql(s"$label.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step(label) { (spark, table) => + spark.conf.set("spark.wap.id", "wS") + try spark.sql(write(table)) finally spark.conf.unset("spark.wap.id") + val mainPre = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) + val stagedCount = spark.sql(s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'wS'").collect()(0).getLong(0) + println(s"DIAG $label: mainPreCount=$mainPre (expected $preRows) stagedSnapshots=$stagedCount") + assert(mainPre == preRows, + s"$label: staged write LEAKED to main pre-publish (main=$mainPre, expected $preRows)") + val stagedId = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = 'wS'").collect()(0).getLong(0) + spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', $stagedId)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == postRows, + s"$label: publish did not reflect the staged write (expected $postRows)") + }() + + val wapStagedOps: List[(String, TableTest[CoreTable.type])] = List( + "wapStaged.insert" -> wapStagedWrite("wapStaged.insert")(t => s"INSERT INTO $t VALUES ${coreRow(99, "staged")}")(3, 4), + "wapStaged.overwrite" -> wapStagedWrite("wapStaged.overwrite")(t => s"INSERT OVERWRITE $t VALUES ${coreRow(7, "ow")}")(3, 1), + // FINDING (WAP1): a staged DELETE is NOT honored by WAP — it commits to MAIN immediately and creates + // NO staged snapshot (main 3→2, zero snapshots tagged wap.id), unlike staged INSERT/OVERWRITE/UPDATE/ + // MERGE which all stage. Observed on parquet+orc; whether this is stock Iceberg or OpenHouse-specific is + // not determined here. A "staged" DELETE therefore silently publishes to main. Pins the observed behavior. + "wapStaged.delete.bypassesWap" -> { + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("wapStaged.delete.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("wapStaged.delete.bypassesWap") { (spark, table) => + spark.conf.set("spark.wap.id", "wD") + try spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") finally spark.conf.unset("spark.wap.id") + val mainPre = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) + val staged = spark.sql(s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'wD'").collect()(0).getLong(0) + println(s"DIAG wapStaged.delete.bypassesWap: mainAfterStagedDelete=$mainPre stagedSnapshots=$staged") + assert(mainPre == 2 && staged == 0, + s"FINDING WAP1: expected staged DELETE to BYPASS WAP (commit to main=2, no staged snapshot); got main=$mainPre staged=$staged — behavior changed, re-audit AUDIT-FINDINGS WAP1") + }() + }, + "wapStaged.merge" -> wapStagedWrite("wapStaged.merge")(t => + s"MERGE INTO $t USING (SELECT CAST(99 AS BIGINT) AS k) s ON $t.${Core.long0.columnName} = s.k " + + s"WHEN NOT MATCHED THEN INSERT (${Core.columnNames.mkString(", ")}) VALUES (s.k, 9, 'm', 9.5, true, '2024-01-09-01')")(3, 4), + // Staged UPDATE: main's value is unchanged pre-publish, changed after publish (count stays 3). + "wapStaged.update.valueVisibleOnlyAfterPublish" -> { + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("wapStaged.update.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("wapStaged.update.valueVisibleOnlyAfterPublish") { (spark, table) => + spark.conf.set("spark.wap.id", "wU") + try spark.sql(s"UPDATE $table SET ${Core.string0.columnName} = 'staged-upd' WHERE ${Core.long0.columnName} = 1") + finally spark.conf.unset("spark.wap.id") + val pre = spark.sql(s"SELECT ${Core.string0.columnName} FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getString(0) + assert(pre != "staged-upd", s"staged UPDATE leaked to main pre-publish: $pre") + val stagedId = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = 'wU'").collect()(0).getLong(0) + spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', $stagedId)") + val post = spark.sql(s"SELECT ${Core.string0.columnName} FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getString(0) + assert(post == "staged-upd", s"publish did not reflect the staged UPDATE: $post") + }() + }, + // C3(a): two concurrent staged ids publish INDEPENDENTLY and in the chosen order. + "wapStaged.twoIdsIndependent" -> { + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("wapStaged.two.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("wapStaged.twoIdsIndependent") { (spark, table) => + def staged(id: String, k: Int): Unit = { + spark.conf.set("spark.wap.id", id) + try spark.sql(s"INSERT INTO $table VALUES ${coreRow(k, s"s-$id")}") finally spark.conf.unset("spark.wap.id") + } + staged("wa", 101); staged("wb", 102) + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "either staged id leaked to main") + def idOf(w: String): Long = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = '$w'").collect()(0).getLong(0) + spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${idOf("wa")})") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "publishing wa did not advance main by 1") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 102").collect()(0).getLong(0) == 0, "wb published without being cherrypicked") + spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${idOf("wb")})") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 5, "publishing wb did not advance main to 5") + }() + }, + // C3(b): a staged (unpublished) snapshot is UNREFERENCED — assert expire_snapshots behaviour toward it + // (G11(d): age-based expiration can delete staged WAP snapshots pre-publish). Characterize: after a + // far-future expire, can the staged id still be cherrypicked, or is it stranded? + "wapStaged.expireVsStaged" -> { + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("wapStaged.exp.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("wapStaged.expireVsStaged") { (spark, table) => + spark.conf.set("spark.wap.id", "wE") + try spark.sql(s"INSERT INTO $table VALUES ${coreRow(200, "stg")}") finally spark.conf.unset("spark.wap.id") + val stagedId = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = 'wE'").collect()(0).getLong(0) + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + val survived = spark.sql(s"SELECT count(*) FROM $table.snapshots WHERE snapshot_id = $stagedId").collect()(0).getLong(0) + val pub = try { spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', $stagedId)"); "published" } + catch { case NonFatal(e) => s"stranded:${Exceptions.root(e).getClass.getSimpleName}" } + println(s"DIAG wapStaged.expireVsStaged: stagedSurvivedExpire=$survived cherrypickAfterExpire=$pub") + // Pin the audited hazard (G11 d): unreferenced staged snapshot is expirable -> stranded pre-publish. + assert(survived == 0 && pub.startsWith("stranded"), + s"G11(d): expected the unreferenced staged snapshot to be expired then un-cherrypickable; survived=$survived pub=$pub — re-audit") + }() + } + ) + + // B3 DDL-on-branch is NOT isolated — characterizes the leak (finding): schema/props/sortOrder are + // table-global; ADD COLUMN while "on branch" mutates MAIN's schema, with no guard. + val branchDdlLeakAddColumn: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("branch.leak.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .sql("branch.leak.create")(t => s"ALTER TABLE $t CREATE BRANCH leakbr")() + .step("branch.ddlLeak.addColumn") { (spark, table) => + spark.conf.set("spark.wap.branch", "leakbr") + try spark.sql(s"ALTER TABLE $table ADD COLUMN leaked_col int") + finally spark.conf.unset("spark.wap.branch") + val mainCols = spark.table(table).schema.fields.map(_.name).toSeq + assert(mainCols.contains("leaked_col"), + s"characterizing the leak: ADD COLUMN on a branch mutated MAIN's schema — expected leaked_col in $mainCols") + }() + + // B4 representative branch DML (update + delete on a branch), isolated from main. + val branchDmlUpdateDelete: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("branch.dml.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .sql("branch.dml.create")(t => s"ALTER TABLE $t CREATE BRANCH dmlbr")() + .step("branch.dml.updateDelete") { (spark, table) => + spark.conf.set("spark.wap.branch", "dmlbr") + try { + spark.sql(s"UPDATE $table SET ${Core.string0.columnName} = 'br-upd' WHERE ${Core.long0.columnName} = 1") + spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 2") + } finally spark.conf.unset("spark.wap.branch") + val onBranch = spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'dmlbr'").collect()(0).getLong(0) + assert(onBranch == 2, s"branch should have 2 rows after delete, got $onBranch") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "main unchanged by branch DML") + val br1 = spark.sql(s"SELECT ${Core.string0.columnName} FROM $table VERSION AS OF 'dmlbr' WHERE ${Core.long0.columnName} = 1").collect()(0).getString(0) + assert(br1 == "br-upd", s"branch update not applied: $br1") + }() + + // B5 lifecycle (CREATE TAG / DROP BRANCH — both supported, verified) + WAP mixing negatives. + val branchCreateTag: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("branch.lifecycle.tag") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE TAG mytag") + assert(spark.sql(s"SELECT count(*) FROM $table.refs WHERE name = 'mytag' AND type = 'TAG'").collect()(0).getLong(0) == 1, + "CREATE TAG did not create the tag ref") + }() + + val branchDropBranch: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("branch.drop.create")(t => s"ALTER TABLE $t CREATE BRANCH tmpbr")() + .step("branch.lifecycle.dropBranch") { (spark, table) => + assert(spark.sql(s"SELECT count(*) FROM $table.refs WHERE name = 'tmpbr'").collect()(0).getLong(0) == 1, "branch not created") + spark.sql(s"ALTER TABLE $table DROP BRANCH tmpbr") + assert(spark.sql(s"SELECT count(*) FROM $table.refs WHERE name = 'tmpbr'").collect()(0).getLong(0) == 0, "DROP BRANCH did not remove the ref") + }() + + val branchNegWapIdAndBranch: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("branch.neg.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .sql("branch.neg.create")(t => s"ALTER TABLE $t CREATE BRANCH nb")() + .step("branch.neg.wapIdAndBranch") { (spark, table) => + spark.conf.set("spark.wap.id", "w1") + spark.conf.set("spark.wap.branch", "nb") + try { + val e = Check.intercept[ValidationException](spark.sql(s"INSERT INTO $table VALUES ${coreRow(99, "x")}")) + assert(e.getMessage.contains("Cannot set both WAP ID and branch"), s"msg: ${e.getMessage.take(140)}") + } finally { spark.conf.unset("spark.wap.id"); spark.conf.unset("spark.wap.branch") } + }() + + val branchNegInsertNonexistent: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("branch.neg.insertNonexistentBranch") { (spark, table) => + val e = Check.intercept[ValidationException](spark.sql(s"INSERT INTO $table.branch_nope VALUES ${coreRow(99, "x")}")) + assert(e.getMessage.contains("does not exist"), s"msg: ${e.getMessage.take(140)}") + }() + + // ── WAP mega-axis Stage B — systematic branch-DDL leak (G8) ────────────────────────────────── + // Table-global DDL (schema / props / sortOrder / policy) run WHILE `spark.wap.branch` is set: per G8 + // these apply table-globally at every layer, so they LEAK to MAIN rather than staying branch-scoped. + // Each pins the ACTUAL outcome on MAIN (wap.branch unset after the DDL) — leak / silent-no-op / rejected. + // If OpenHouse later scopes branch DDL, these flip. Format-multiplexed by crossFmt (seedFmt-aware create). + private def branchDdlOnBranch(label: String)(ddl: String => String)(assertMain: (SparkSession, String) => Unit): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql(s"$label.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .sql(s"$label.createBranch")(t => s"ALTER TABLE $t CREATE BRANCH bddl")() + .step(label) { (spark, table) => + spark.conf.set("spark.wap.branch", "bddl") + val outcome = try { spark.sql(ddl(table)); "accepted" } + catch { case NonFatal(e) => s"rejected:${Exceptions.root(e).getClass.getSimpleName}" } + finally spark.conf.unset("spark.wap.branch") + println(s"DIAG $label: branch-routed DDL $outcome") + assertMain(spark, table) + }() + + val branchDdlOps: List[(String, TableTest[CoreTable.type])] = List( + // ADD COLUMN on a branch → main's schema gains the column (schema is table-global → leak). + "branchDdl.addColumn.leaksToMain" -> branchDdlOnBranch("branchDdl.addColumn.leaksToMain")( + t => s"ALTER TABLE $t ADD COLUMN br_added int") { (spark, table) => + val cols = spark.sql(s"DESCRIBE TABLE $table").collect().map(_.getString(0).trim).toSet + assert(cols.contains("br_added"), + "G8: ADD COLUMN on a branch should LEAK to main's schema (table-global); main did not gain the column — re-audit G8") + }, + // SET TBLPROPERTIES on a branch → main gets the property (props are table-global → leak). + "branchDdl.setTblProp.leaksToMain" -> branchDdlOnBranch("branchDdl.setTblProp.leaksToMain")( + t => s"ALTER TABLE $t SET TBLPROPERTIES ('user.branchkey'='v1')") { (spark, table) => + val props = spark.sql(s"SHOW TBLPROPERTIES $table").collect().map(r => r.getString(0) -> r.getString(1)).toMap + assert(props.get("user.branchkey").contains("v1"), + s"G8: SET TBLPROPERTIES on a branch should LEAK to main; got ${props.get("user.branchkey")} — re-audit G8") + }, + // ALTER COLUMN comment on a branch → main's schema metadata changes (leak). + "branchDdl.alterColumnComment.leaksToMain" -> branchDdlOnBranch("branchDdl.alterColumnComment.leaksToMain")( + t => s"ALTER TABLE $t ALTER COLUMN ${Core.string0.columnName} COMMENT 'br-comment'") { (spark, table) => + val c = spark.sql(s"DESCRIBE TABLE $table").collect() + .find(_.getString(0).trim == Core.string0.columnName).map(_.getString(2)).getOrElse("") + assert(Option(c).getOrElse("").contains("br-comment"), + s"G8: ALTER COLUMN COMMENT on a branch should LEAK to main; main comment='$c' — re-audit G8") + }, + // DROP COLUMN is rejected on main (unsupported) — assert it is ALSO rejected via a branch (the guard + // is schema-global, not branch-aware): pin the rejection is unchanged under wap.branch. + "branchDdl.dropColumn.rejected" -> branchDdlOnBranch("branchDdl.dropColumn.rejected")( + t => s"ALTER TABLE $t DROP COLUMN ${Core.string0.columnName}") { (spark, table) => + val cols = spark.sql(s"DESCRIBE TABLE $table").collect().map(_.getString(0).trim).toSet + assert(cols.contains(Core.string0.columnName), + "DROP COLUMN must remain rejected (main keeps the column) whether or not spark.wap.branch is set") + } + ) + + val branching: List[(String, TableTest[CoreTable.type])] = List( + "branch.direct.isolation" -> branchDirectIsolation, + "branch.wapConf.routing" -> branchWapConfRouting, + "wap.stagePublish" -> wapStagePublish, + "branch.ddlLeak.addColumn" -> branchDdlLeakAddColumn, + "branch.dml.updateDelete" -> branchDmlUpdateDelete, + "branch.lifecycle.tag" -> branchCreateTag, + "branch.lifecycle.dropBranch" -> branchDropBranch, + "branch.neg.wapIdAndBranch" -> branchNegWapIdAndBranch, + "branch.neg.insertNonexistentBranch" -> branchNegInsertNonexistent + ) + + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala new file mode 100644 index 000000000..6ddb15a52 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala @@ -0,0 +1,779 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +trait DmlScenarios extends ScenarioKit { + import Rows._ + + // ── DDL × consumer battery (BUILD-STATUS task #3) ──────────────────────────────────────────── + // A DDL op is a STATE CHANGE; the battery asserts every consumer still works after it (the + // modality thesis at the DDL level). DDL preps leave a distinct post-state; consumers are + // arity-safe (they use SELECT * / metadata tables, never a fixed column list) so they compose + // over ANY post-DDL schema. NOTE: this is the NON-VACUOUS core — the appraisal's 420 assumed + // 35 DDL (incl. negatives/one-shots) × 6, but a rejected DDL or a rename has no post-state for a + // consumer to exercise. State-changing DDL × real consumers is ~54, and that's what's built. + val ddlPreps: List[(String, Layout => TableTest[CoreTable.type])] = List( + "addColumn" -> (l => createAndSeed(l, 3).sql("ddl")(t => s"ALTER TABLE $t ADD COLUMN cc int")()), + "typeWiden" -> (l => createAndSeed(l, 3).sql("ddl")(t => s"ALTER TABLE $t ALTER COLUMN ${Core.int0.columnName} TYPE bigint")()), + "writeOrder" -> (l => createAndSeed(l, 3).sql("ddl")(t => s"ALTER TABLE $t WRITE ORDERED BY ${Core.long0.columnName}")()), + "distMode" -> (l => createAndSeed(l, 3).sql("ddl")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.distribution-mode'='range')")()) + ) + + private def dupRow(key: Long) = s"SELECT * FROM %s WHERE ${Core.long0.columnName} = $key" // arity-safe append source + + val ddlConsumers: List[(String, TableTest[CoreTable.type])] = List( + // C1 the table stays WRITABLE (append) after the DDL — arity-safe self-select append. + "dmlWrite" -> TableTest(Core).step("consume.dmlWrite") { (spark, table) => + spark.sql(s"INSERT INTO $table ${dupRow(1).format(table)}") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "not writable post-DDL") + }(), + // C2 the MUTATION path still works after the DDL. + "dmlMutate" -> TableTest(Core).step("consume.dmlMutate") { (spark, table) => + spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 2") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "mutation broken post-DDL") + }(), + // C3 TIME TRAVEL to the pre-DDL/seed snapshot still resolves. + "timeTravel" -> TableTest(Core).step("consume.timeTravel") { (spark, table) => + val s0 = snapshotIds(spark, table).head + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF $s0").collect()(0).getLong(0) == 3, + "pre-DDL snapshot not travelable") + }(), + // C4 RESTORE across the DDL: write post-DDL, then roll back to the seed snapshot. + "restore" -> TableTest(Core).step("consume.restore") { (spark, table) => + val s0 = snapshotIds(spark, table).head + spark.sql(s"INSERT INTO $table ${dupRow(1).format(table)}") + spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', $s0)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "restore across DDL failed") + }(), + // C5 EXPIRE after the DDL: history trims, current data survives and reads. + "expire" -> TableTest(Core).step("consume.expire") { (spark, table) => + spark.sql(s"INSERT INTO $table ${dupRow(1).format(table)}") + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "unreadable after expire post-DDL") + }(), + // C6 BRANCH after the DDL: branchable, write on branch, main isolated. + "branch" -> TableTest(Core).step("consume.branch") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH cb") + spark.sql(s"INSERT INTO $table.branch_cb ${dupRow(1).format(table)}") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'cb'").collect()(0).getLong(0) == 4, "branch write failed post-DDL") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "branch leaked to main post-DDL") + }(), + // C7 COMPACTION after the DDL: a second data file, then rewrite_data_files preserves the rows. + "compact" -> TableTest(Core).step("consume.compact") { (spark, table) => + spark.sql(s"INSERT INTO $table ${dupRow(1).format(table)}") // second data file + spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('min-input-files', '2'))") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "compaction changed rows post-DDL") + }() + ) + + // Closing assertion for the branch axis: after the branch-routed op, MAIN must be untouched + // (still the 3-row seed) — the isolation half of the branch contract. Uniform across all ops + // because with spark.wap.branch set every write routes to the branch, never to main. + val branchMainIsolation: TableTest[CoreTable.type] = + TableTest(Core).step("branch.mainIsolated") { (spark, table) => + spark.conf.unset("spark.wap.branch") + val mainCount = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) + assert(mainCount == 3, s"branch op leaked to MAIN — expected 3 rows, got $mainCount (isolation broken)") + }() + + // ── reads ──────────────────────────────────────────────────────────────────────────── + val readProjection: TableTest[CoreTable.type] = + TableTest(Core).check("read.projection") { view => + val expected = view.before.sortBy(_.get(Core.long0)).map(_.get(Core.string0)) + val actual = view.spark + .sql(s"SELECT ${Core.string0.columnName} FROM ${view.table} ORDER BY ${Core.long0.columnName}") + .collect().toSeq.map(_.get(Core.string0)) + assert(actual == expected) + } + + val readFilter: TableTest[CoreTable.type] = + TableTest(Core).check("read.filter") { view => + val expected = view.before.map(_.get(Core.long0)).filter(_ >= 2).sorted + val actual = view.spark + .sql(s"SELECT ${Core.long0.columnName} FROM ${view.table} WHERE ${Core.long0.columnName} >= 2 ORDER BY ${Core.long0.columnName}") + .collect().toSeq.map(_.get(Core.long0)) + assert(actual == expected) + } + + // The declared write format actually materializes: every data file carries that extension. + val formatMaterialization: TableTest[CoreTable.type] = + TableTest(Core).check("format.materialization") { view => + val format = view.spark.sql(s"SHOW TBLPROPERTIES ${view.table} ('write.format.default')").collect()(0).getString(1) + val paths = view.spark.sql(s"SELECT file_path FROM ${view.table}.files").collect().toSeq.map(_.getString(0)) + assert(paths.nonEmpty && paths.forall(_.toLowerCase.endsWith(s".$format")), s"data files are not all .$format: $paths") + } + + // ── delete ─────────────────────────────────────────────────────────────────────────── + val deleteByPredicate: TableTest[CoreTable.type] = + TableTest(Core).delete(core => s"${core.long0.columnName} < 2") { view => + assert(view.after == view.before.filterNot(_.get(Core.long0) < 2)) + } + + val deleteWhereFalseKeepsSnapshot: TableTest[CoreTable.type] = + TableTest(Core).delete(_ => "false") { view => + assert(view.after == view.before) + assert(view.snapshotsAfter == view.snapshotsBefore, "DELETE WHERE false must not commit a snapshot") + } + + val truncate: TableTest[CoreTable.type] = + TableTest(Core).sql("delete.truncate")(table => s"TRUNCATE TABLE $table") { view => + assert(view.after.isEmpty) + } + + val deleteAtSnapshotRejected: TableTest[CoreTable.type] = + TableTest(Core).step("delete.atSnapshot.rejected") { (spark, table) => + val snapshotId = spark + .sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at DESC LIMIT 1") + .collect()(0).getLong(0) + val error = Check.intercept[IllegalArgumentException]( + spark.sql(s"DELETE FROM $table.snapshot_id_$snapshotId WHERE ${Core.long0.columnName} < 4")) + assert(error.getMessage == s"Cannot delete from table at a specific snapshot: $snapshotId") + } { view => + assert(view.after == view.before) // a rejected delete leaves the table unchanged + } + + // Removes exactly the keys in the list. + val deleteByInList: TableTest[CoreTable.type] = + TableTest(Core).delete(core => s"${core.long0.columnName} IN (1, 3)") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(Set(1L, 3L)).sorted) + } + + // Predicate is an IN-subquery over an explicit source. + val deleteByInSubquery: TableTest[CoreTable.type] = + TableTest(Core).delete(core => + s"${core.long0.columnName} IN (SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(_ == 2L).sorted) + } + + val deleteByNotInSubquery: TableTest[CoreTable.type] = + TableTest(Core).delete(core => + s"${core.long0.columnName} NOT IN (SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filter(_ == 2L).sorted) + } + + val deleteByExistsSubquery: TableTest[CoreTable.type] = + TableTest(Core).delete(core => + s"EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) WHERE s.x = ${core.long0.columnName})") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(_ == 2L).sorted) + } + + val deleteByNotExistsSubquery: TableTest[CoreTable.type] = + TableTest(Core).delete(core => + s"NOT EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) WHERE s.x = ${core.long0.columnName})") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filter(_ == 2L).sorted) + } + + val deleteByScalarSubquery: TableTest[CoreTable.type] = + TableTest(Core).delete(core => + s"${core.long0.columnName} = (SELECT max(col1) FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(_ == 2L).sorted) + } + + // Seed a null-string row, then DELETE WHERE string IS NULL must remove exactly it (and nothing + // else) — a real IS-NULL match, not a vacuous no-op. + val deleteByNullCondition: TableTest[CoreTable.type] = + TableTest(Core) + .sql("delete.byNullCondition.seed")(table => + s"INSERT INTO $table VALUES (CAST(99 AS BIGINT), 99, NULL, 99.5, false, '2024-01-01-00')")() + .delete(core => s"${core.string0.columnName} IS NULL") { view => + assert(view.before.exists(_.get(Core.string0) == null), "precondition: a null-string row was seeded") + val expected = view.before.filterNot(_.get(Core.string0) == null).map(_.get(Core.long0)).sorted + assert(keyed(view.after) == expected) // exactly the non-null rows remain + assert(!keyed(view.after).contains(99L)) // the null-string row was removed + } + + // DELETE with no WHERE clause empties the table. + val deleteAll: TableTest[CoreTable.type] = + TableTest(Core).sql("delete.all")(table => s"DELETE FROM $table") { view => + assert(view.after.isEmpty) + } + + // A real predicate that matches nothing: rows unchanged, but one (empty) snapshot is still + // committed — a scanned no-match, unlike the constant-folded `DELETE WHERE false` no-op above. + val deleteNone: TableTest[CoreTable.type] = + TableTest(Core).delete(core => s"${core.long0.columnName} = 999") { view => + assert(view.after == view.before) + assert(view.snapshotsAfter == view.snapshotsBefore + 1, "no-match DELETE with a real predicate still commits one snapshot") + } + + // A partition-column predicate (a metadata-only delete on a partitioned layout). + val deleteByPartitionPredicate: TableTest[CoreTable.type] = + TableTest(Core).delete(core => s"${core.datePartition.columnName} = '2024-01-01-00'") { view => + val expected = view.before.filterNot(_.get(Core.datePartition) == "2024-01-01-00").map(_.get(Core.long0)).sorted + assert(keyed(view.after) == expected) + } + + val deleteWithAlias: TableTest[CoreTable.type] = + TableTest(Core).sql("delete.withAlias")(table => + s"DELETE FROM $table AS x WHERE x.${Core.long0.columnName} < 2") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(_ < 2L).sorted) + } + + // ── update ─────────────────────────────────────────────────────────────────────────── + val updateByPredicate: TableTest[CoreTable.type] = + TableTest(Core).sql("update.byPredicate")(table => + s"UPDATE $table SET ${Core.string0.columnName} = 'X' WHERE ${Core.long0.columnName} = 2") { view => + val expected = longToString(view.before).map { case (id, s) => id -> (if (id == 2) "X" else s) } + assert(longToString(view.after) == expected) + } + + val updateWithoutCondition: TableTest[CoreTable.type] = + TableTest(Core).sql("update.withoutCondition")(table => + s"UPDATE $table SET ${Core.string0.columnName} = 'Z'") { view => + assert(longToString(view.after) == longToString(view.before).map { case (id, _) => id -> "Z" }) + } + + // A real predicate matching nothing still commits an (empty) snapshot — unlike the + // constant-folded `DELETE WHERE false` no-op (confirmed vs OSS TestUpdate.testUpdateNonExistingRecords). + val updateNoMatch: TableTest[CoreTable.type] = + TableTest(Core).sql("update.noMatch")(table => + s"UPDATE $table SET ${Core.string0.columnName} = 'Y' WHERE ${Core.long0.columnName} = 99") { view => + assert(longToString(view.after) == longToString(view.before)) + assert(view.snapshotsAfter == view.snapshotsBefore + 1, "no-match UPDATE still commits one snapshot") + } + + private def stringUpdatedWhere(view: StepView[CoreTable.type], matches: Long => Boolean, to: String): Boolean = + longToString(view.after) == longToString(view.before).map { case (id, s) => id -> (if (matches(id)) to else s) } + + val updateByInSubquery: TableTest[CoreTable.type] = + TableTest(Core).sql("update.byInSubquery")(table => + s"UPDATE $table SET ${Core.string0.columnName} = 'X' " + + s"WHERE ${Core.long0.columnName} IN (SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => + assert(stringUpdatedWhere(view, _ == 2, "X")) + } + + val updateByNotInSubquery: TableTest[CoreTable.type] = + TableTest(Core).sql("update.byNotInSubquery")(table => + s"UPDATE $table SET ${Core.string0.columnName} = 'X' " + + s"WHERE ${Core.long0.columnName} NOT IN (SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => + assert(stringUpdatedWhere(view, _ != 2, "X")) + } + + val updateByExistsSubquery: TableTest[CoreTable.type] = + TableTest(Core).sql("update.byExistsSubquery")(table => + s"UPDATE $table SET ${Core.string0.columnName} = 'X' " + + s"WHERE EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) WHERE s.x = ${Core.long0.columnName})") { view => + assert(stringUpdatedWhere(view, _ == 2, "X")) + } + + val updateByNotExistsSubquery: TableTest[CoreTable.type] = + TableTest(Core).sql("update.byNotExistsSubquery")(table => + s"UPDATE $table SET ${Core.string0.columnName} = 'X' " + + s"WHERE NOT EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) WHERE s.x = ${Core.long0.columnName})") { view => + assert(stringUpdatedWhere(view, _ != 2, "X")) + } + + val updateByScalarSubquery: TableTest[CoreTable.type] = + TableTest(Core).sql("update.byScalarSubquery")(table => + s"UPDATE $table SET ${Core.string0.columnName} = 'X' " + + s"WHERE ${Core.long0.columnName} = (SELECT max(col1) FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => + assert(stringUpdatedWhere(view, _ == 2, "X")) + } + + val updateWithAlias: TableTest[CoreTable.type] = + TableTest(Core).sql("update.withAlias")(table => + s"UPDATE $table AS x SET x.${Core.string0.columnName} = 'X' WHERE x.${Core.long0.columnName} = 2") { view => + assert(stringUpdatedWhere(view, _ == 2, "X")) + } + + // Sets two columns in one statement; assert both landed on the matched row. + val updateMultipleColumns: TableTest[CoreTable.type] = + TableTest(Core).sql("update.multipleColumns")(table => + s"UPDATE $table SET ${Core.string0.columnName} = 'X', ${Core.int0.columnName} = 99 WHERE ${Core.long0.columnName} = 2") { view => + assert(stringUpdatedWhere(view, _ == 2, "X")) + assert(view.after.find(_.get(Core.long0) == 2L).map(_.get(Core.int0)).contains(99)) + } + + // Assign a column by an expression over itself (updates the key column). + val updateByExpression: TableTest[CoreTable.type] = + TableTest(Core).sql("update.byExpression")(table => + s"UPDATE $table SET ${Core.long0.columnName} = ${Core.long0.columnName} + 10 WHERE ${Core.long0.columnName} = 2") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).map(l => if (l == 2L) 12L else l).sorted) + } + + // Update the partition column so the row moves partitions. + val updateMovePartition: TableTest[CoreTable.type] = + TableTest(Core).sql("update.movePartition")(table => + s"UPDATE $table SET ${Core.datePartition.columnName} = '2099-12-31-23' WHERE ${Core.long0.columnName} = 2") { view => + val part = (rows: Seq[Row]) => rows.map(r => r.get(Core.long0) -> r.get(Core.datePartition)).toMap + assert(part(view.after) == part(view.before).map { case (id, d) => id -> (if (id == 2) "2099-12-31-23" else d) }) + } + + val updateNullAssignment: TableTest[CoreTable.type] = + TableTest(Core).sql("update.nullAssignment")(table => + s"UPDATE $table SET ${Core.string0.columnName} = NULL WHERE ${Core.long0.columnName} = 2") { view => + assert(longToString(view.after) == longToString(view.before).map { case (id, s) => id -> (if (id == 2) null else s) }) + } + + // ── merge ──────────────────────────────────────────────────────────────────────────── + // Source rows are written as EXPLICIT literals. The generator-sourced alternative for this + // test would be: + // USING (${RowGenerator.valuesClause(Core, ...)} for indices 4,5) ... WHEN NOT MATCHED THEN INSERT * + // i.e. name the row *indices* and let the column generators fill every column. We prefer the + // explicit form so the source values are visible in the test. + val mergeInsertNotMatched: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.insertNotMatched")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES + (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), + (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN INSERT *""") { view => + assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) ++ Seq(4L, 5L)).sorted) + // INSERT * must map the columns correctly, not just land the join key. + assert(view.after.find(_.get(Core.long0) == 4L).map(_.get(Core.string0)).contains("row-4")) + assert(view.after.find(_.get(Core.long0) == 5L).map(_.get(Core.string0)).contains("row-5")) + } + + val mergeUpdateMatched: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.updateMatched")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT), 'M') AS s(${Core.long0.columnName}, ${Core.string0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") { view => + val expected = longToString(view.before).map { case (id, s) => id -> (if (id == 2) "M" else s) } + assert(longToString(view.after) == expected) + } + + val mergeDeleteMatched: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.deleteMatched")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES (CAST(1 AS BIGINT)), (CAST(3 AS BIGINT)) AS s(${Core.long0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN DELETE""") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(Set(1L, 3L)).sorted) + } + + val mergeUpsert: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.upsert")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES + (CAST(2 AS BIGINT), 2, 'U', 2.5, true, '2024-01-02-01'), + (CAST(7 AS BIGINT), 7, 'g', 7.5, false, '2024-01-07-06') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName} + WHEN NOT MATCHED THEN INSERT *""") { view => + val updated = longToString(view.before).map { case (id, s) => id -> (if (id == 2) "U" else s) } + val withInsert = if (view.before.exists(_.get(Core.long0) == 7L)) updated else updated + (7L -> "g") + assert(longToString(view.after) == withInsert) + } + + // Keep only rows the source knows about: delete every row NOT matched by a source row. + val mergeDeleteNotMatchedBySource: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.deleteNotMatchedBySource")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT)) AS s(${Core.long0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED BY SOURCE THEN DELETE""") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filter(_ == 2L).sorted) + } + + // Both keys 2 and 3 match, but the per-clause condition only fires for key 2. + val mergeConditionalUpdate: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.conditionalUpdate")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT), 'U2'), (CAST(3 AS BIGINT), 'U3') + AS s(${Core.long0.columnName}, ${Core.string0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED AND s.${Core.long0.columnName} = 2 THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") { view => + assert(longToString(view.after) == longToString(view.before).map { case (id, s) => id -> (if (id == 2) "U2" else s) }) + } + + // First matched clause wins: key 2 updates (conditional), key 3 falls through to DELETE. + val mergeMultipleMatchedClauses: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.multipleMatchedClauses")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT), 'U'), (CAST(3 AS BIGINT), 'x') + AS s(${Core.long0.columnName}, ${Core.string0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED AND s.${Core.long0.columnName} = 2 THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName} + WHEN MATCHED THEN DELETE""") { view => + assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(_ == 3L).sorted) + assert(view.after.find(_.get(Core.long0) == 2L).map(_.get(Core.string0)).contains("U")) + } + + // Conditional NOT MATCHED: source keys 4 and 5, but only 4 satisfies the insert condition. + val mergeConditionalInsert: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.conditionalInsert")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES + (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), + (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED AND s.${Core.long0.columnName} = 4 THEN INSERT *""") { view => + assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 4L).sorted) + } + + // All three clause kinds in one statement: update key 2, insert key 4, delete-by-source rows 1 & 3. + val mergeAllClauses: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.allClauses")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES + (CAST(2 AS BIGINT), 2, 'M2', 2.5, true, '2024-01-02-01'), + (CAST(4 AS BIGINT), 4, 'row-4', 4.5, false, '2024-01-04-03') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName} + WHEN NOT MATCHED THEN INSERT * + WHEN NOT MATCHED BY SOURCE THEN DELETE""") { view => + assert(keyed(view.after) == Seq(2L, 4L)) + assert(view.after.find(_.get(Core.long0) == 2L).map(_.get(Core.string0)).contains("M2")) + } + + // UPDATE SET * replaces every column of the matched row from the source. + val mergeUpdateStar: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.updateStar")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT), 22, 'S2', 22.5, true, '2024-06-06-06') AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN UPDATE SET *""") { view => + val row2 = view.after.find(_.get(Core.long0) == 2L) + assert(row2.map(_.get(Core.string0)).contains("S2")) + assert(row2.map(_.get(Core.int0)).contains(22)) + } + + // Explicit column-specification INSERT (other columns null-filled). + val mergeInsertExplicitColumns: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.insertExplicitColumns")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES (CAST(7 AS BIGINT), 'g') AS s(${Core.long0.columnName}, ${Core.string0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}, ${Core.string0.columnName}) VALUES (s.${Core.long0.columnName}, s.${Core.string0.columnName})""") { view => + assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 7L).sorted) + assert(view.after.find(_.get(Core.long0) == 7L).map(_.get(Core.string0)).contains("g")) + } + + // Source is a CTE. + val mergeSourceCTE: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.sourceCTE")(table => + s"""MERGE INTO $table t USING ( + WITH src AS (SELECT CAST(8 AS BIGINT) AS ${Core.long0.columnName}) SELECT * FROM src + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}) VALUES (s.${Core.long0.columnName})""") { view => + assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 8L).sorted) + } + + // Source is a set operation (UNION ALL). + val mergeSourceSetOp: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.sourceSetOp")(table => + s"""MERGE INTO $table t USING ( + SELECT CAST(8 AS BIGINT) AS ${Core.long0.columnName} UNION ALL SELECT CAST(9 AS BIGINT) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}) VALUES (s.${Core.long0.columnName})""") { view => + assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) ++ Seq(8L, 9L)).sorted) + } + + // Merge into an empty target inserts all non-matching source rows (empties the seed first). + val mergeIntoEmptyTarget: TableTest[CoreTable.type] = + TableTest(Core) + .sql("merge.intoEmptyTarget.empty")(table => s"DELETE FROM $table")() + .sql("merge.intoEmptyTarget")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES + (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), + (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN INSERT *""") { view => + assert(view.before.isEmpty) + assert(keyed(view.after) == Seq(4L, 5L)) + } + + // A null join key never matches, so it neither updates nor errors. + val mergeNullJoinKey: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.nullJoinKey")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES (CAST(NULL AS BIGINT), 'n'), (CAST(2 AS BIGINT), 'M') + AS s(${Core.long0.columnName}, ${Core.string0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") { view => + assert(keyed(view.after) == keyed(view.before)) + assert(longToString(view.after) == longToString(view.before).map { case (id, s) => id -> (if (id == 2) "M" else s) }) + } + + // INSERT * resolves columns by name even when the source lists them in a different order. + val mergeResolveByName: TableTest[CoreTable.type] = + TableTest(Core).sql("merge.resolveByName")(table => + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES ('g', CAST(7 AS BIGINT), 7, 7.5, false, '2024-07-07-07') + AS s(${Core.string0.columnName}, ${Core.long0.columnName}, ${Core.int0.columnName}, ${Core.double0.columnName}, ${Core.boolean0.columnName}, datepartition) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN INSERT *""") { view => + assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 7L).sorted) + assert(view.after.find(_.get(Core.long0) == 7L).map(_.get(Core.string0)).contains("g")) + } + + // ── insert / append / overwrite ──────────────────────────────────────────────────────── + val insertInto: TableTest[CoreTable.type] = + TableTest(Core).sql("insert.into")(table => + s"""INSERT INTO $table VALUES + (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), + (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')""") { view => + assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) ++ Seq(4L, 5L)).sorted) + } + + val appendDataFrame: TableTest[CoreTable.type] = + TableTest(Core).step("append.dataFrame") { (spark, table) => + val frame = spark.sql( + s"SELECT * FROM VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') AS s($cols)") + frame.writeTo(table).append() + } { view => + assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 6L).sorted) + } + + // INSERT OVERWRITE (static mode, the Spark default) replaces the whole table regardless of state. + val insertOverwrite: TableTest[CoreTable.type] = + TableTest(Core).sql("insert.overwrite")(table => + s"""INSERT OVERWRITE $table VALUES + (CAST(1 AS BIGINT), 1, 'p', 1.5, false, '2024-01-01-00'), + (CAST(2 AS BIGINT), 2, 'q', 2.5, true, '2024-01-02-01')""") { view => + assert(keyed(view.after) == Seq(1L, 2L)) + } + + val overwriteDataFrame: TableTest[CoreTable.type] = + TableTest(Core).step("overwrite.dataFrame") { (spark, table) => + val frame = spark.sql( + s"SELECT * FROM VALUES (CAST(8 AS BIGINT), 8, 'h', 8.5, false, '2024-01-08-07') AS s($cols)") + frame.writeTo(table).overwrite(org.apache.spark.sql.functions.lit(true)) + } { view => + assert(keyed(view.after) == Seq(8L)) + } + + // INSERT INTO with an explicit column list; the unlisted columns are null-filled. + // NEGATIVE PIN (was SKIP-as-bug; reclassified after code-verified investigation). A partial/named- + // column INSERT that omits other columns is REJECTED with INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA. + // This is an ENGINE limitation, not an OpenHouse policy: OpenHouse creates columns nullable-by-default + // and the server round-trips the schema verbatim (verified) — but Iceberg 1.5's SparkTable does not + // advertise column defaults (no SupportsColumnDefaultValue), so Spark's byName output resolution never + // inserts the NULL-fill projection for the omitted (nullable) columns. Pin the rejection; it flips + // only when the read+write APPLICATION of column defaults is wired (SparkTable implements + // SupportsColumnDefaultValue + the reader injects initial-default for missing columns). NOTE (fork + // audit): the com.linkedin.iceberg 1.5.2 fork #251 backported the NestedField initial/write-default + // APIs + SchemaParser serialization ONLY — no SparkTable, no reader wiring — so the fork does NOT + // satisfy the flip condition (and persists v3-style defaults on a v2 table with no gate). See + // ICEBERG-FORK-AUDIT.md. + val insertExplicitColumns: TableTest[CoreTable.type] = + TableTest(Core).step("insert.explicitColumns") { (spark, table) => + val e = Check.intercept[Exception]( + spark.sql(s"INSERT INTO $table (${Core.long0.columnName}, ${Core.string0.columnName}) " + + s"VALUES (CAST(4 AS BIGINT), 'd'), (CAST(5 AS BIGINT), 'e')")) + val msg = Option(e.getMessage).getOrElse("").toUpperCase + assert(msg.contains("CANNOT_FIND_DATA") || msg.contains("CANNOT FIND DATA") || msg.contains("INCOMPATIBLE_DATA"), + s"expected a partial-INSERT rejection naming the omitted column (engine limitation), got: ${Option(e.getMessage).getOrElse("").take(200)}") + }() + + // INSERT INTO … SELECT appends the selected rows. + val insertIntoSelect: TableTest[CoreTable.type] = + TableTest(Core).sql("insert.intoSelect")(table => + s"INSERT INTO $table SELECT * FROM VALUES " + + s"(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') AS s($cols)") { view => + assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 6L).sorted) + } + + // ── partitioned-only: selective-partition replacement (meaningful only when partitioned) ── + // Seed rows 1/2/3 live in partitions '2024-01-01-00'/'01'/'02'. Writing one row into partition + // '…-00' must replace only that partition, leaving rows 2 and 3. + // Delta-sound: writing row 10 into partition '…-00' replaces ONLY that partition's rows (the + // seeded row 1), leaving every other partition's rows and adding 10. + private def onlyFirstPartitionReplaced(view: StepView[CoreTable.type]): Seq[Long] = + (view.before.filterNot(_.get(Core.datePartition) == "2024-01-01-00").map(_.get(Core.long0)) :+ 10L).sorted + + val insertDynamicOverwrite: TableTest[CoreTable.type] = + TableTest(Core).step("insert.dynamicOverwrite") { (spark, table) => + spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic") + try spark.sql(s"INSERT OVERWRITE $table VALUES (CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00')") + finally spark.conf.set("spark.sql.sources.partitionOverwriteMode", "static") + } { view => + assert(keyed(view.after) == onlyFirstPartitionReplaced(view)) + } + + val overwritePartitions: TableTest[CoreTable.type] = + TableTest(Core).step("overwrite.partitions") { (spark, table) => + val frame = spark.sql( + s"SELECT * FROM VALUES (CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00') AS s($cols)") + frame.writeTo(table).overwritePartitions() + } { view => + assert(keyed(view.after) == onlyFirstPartitionReplaced(view)) + } + + // ── create (a preparation-only test: create under the layout, assert schema + emptiness) ─ + // Also the guard that the literal `columnDefinitions` matches CoreTable's declared columns. + def createSchema(layout: Layout): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(layout.create) { view => + val actual = view.spark.table(view.table).schema.fields.toList.map(field => (field.name, field.dataType.simpleString)) + val expected = Core.tableColumns.toList.map(column => (column.columnName, column.sqlType)) + assert(actual == expected) + assert(view.after.isEmpty) + } + + // ── DDL Phase 12: schema evolution — ADD COLUMN family (❓ probes settle B-vs-N) ─────────── + // The added column is not one of CoreTable's typed handles, so these assert on the LIVE schema + // (name / type / comment / order) and raw SQL, not on typed row handles. Row snapshots + // (view.before/after) still read only CoreTable's columns, so they stay valid across the ALTER. + private def liveColumns(view: StepView[CoreTable.type]): Seq[(String, String)] = + view.spark.table(view.table).schema.fields.toSeq.map(field => (field.name, field.dataType.simpleString)) + + val ddlAddColumnSingle: TableTest[CoreTable.type] = + TableTest(Core).sql("ddl.addColumn.single")(t => s"ALTER TABLE $t ADD COLUMN added_int int") { view => + assert(liveColumns(view).map(_._1).contains("added_int"), s"added_int missing: ${liveColumns(view).map(_._1)}") + val nullCount = view.spark.sql(s"SELECT count(*) FROM ${view.table} WHERE added_int IS NULL").collect()(0).getLong(0) + assert(nullCount == view.before.size, s"existing rows should read null for added_int: $nullCount != ${view.before.size}") + assert(view.after.size == view.before.size) // ADD COLUMN keeps rows + } + + val ddlAddColumnMultiple: TableTest[CoreTable.type] = + TableTest(Core).sql("ddl.addColumn.multiple")(t => s"ALTER TABLE $t ADD COLUMNS (added_a int, added_b string)") { view => + val names = liveColumns(view).map(_._1) + assert(names.contains("added_a") && names.contains("added_b"), s"added columns missing: $names") + assert(view.after.size == view.before.size) + } + + val ddlAddColumnComment: TableTest[CoreTable.type] = + TableTest(Core).sql("ddl.addColumn.comment")(t => s"ALTER TABLE $t ADD COLUMN added_c int COMMENT 'a note'") { view => + val field = view.spark.table(view.table).schema.fields.find(_.name == "added_c") + assert(field.isDefined, "added_c missing") + assert(field.get.getComment().contains("a note"), s"comment not stored: ${field.flatMap(_.getComment())}") + } + + val ddlAddColumnPosition: TableTest[CoreTable.type] = + TableTest(Core).sql("ddl.addColumn.position")(t => s"ALTER TABLE $t ADD COLUMN added_after int AFTER ${Core.long0.columnName}") { view => + val names = liveColumns(view).map(_._1) + assert(names.indexOf("added_after") == names.indexOf(Core.long0.columnName) + 1, s"added_after not after long0: $names") + } + + val ddlAlterColumnTypeWiden: TableTest[CoreTable.type] = + TableTest(Core).sql("ddl.alterColumn.typeWiden")(t => s"ALTER TABLE $t ALTER COLUMN ${Core.int0.columnName} TYPE bigint") { view => + assert(liveColumns(view).toMap.get(Core.int0.columnName).contains("bigint"), s"int0 not widened: ${liveColumns(view).toMap.get(Core.int0.columnName)}") + val vals = view.spark.sql(s"SELECT ${Core.int0.columnName} FROM ${view.table} ORDER BY ${Core.long0.columnName}").collect().toSeq.map(_.getLong(0)) + assert(vals == Seq(1L, 2L, 3L), s"values not preserved after widening: $vals") + } + + // RENAME COLUMN is a SILENT NO-OP on OpenHouse (tagged bug): the statement neither errors nor renames + // — verified via REFRESH TABLE + fresh DESCRIBE, the column keeps its old name. The recon predicted a + // server rejection ("not found in newSchema"), but the client drops the rename before it reaches the + // server, so nothing happens. This test asserts the CORRECT behavior (rename applies) and is tagged in + // Plan.knownBugs, so it reports SKIP until fixed. A silent no-op is worse than a clean rejection. + val ddlRenameColumn: TableTest[CoreTable.type] = + TableTest(Core) + .sql("ddl.renameColumn.seed")(t => s"ALTER TABLE $t ADD COLUMN to_rename int")() + .sql("ddl.renameColumn")(t => s"ALTER TABLE $t RENAME COLUMN to_rename TO renamed_col") { view => + val names = liveColumns(view).map(_._1) + assert(names.contains("renamed_col") && !names.contains("to_rename"), s"RENAME COLUMN silently no-oped: $names") + assert(view.after.size == view.before.size) + } + + /** Phase 12 DDL schema-evolution behaviors, crossed with every layout. */ + val ddlSchemaOperations: List[(String, TableTest[CoreTable.type])] = List( + "ddl.addColumn.single" -> ddlAddColumnSingle, + "ddl.addColumn.multiple" -> ddlAddColumnMultiple, + "ddl.addColumn.comment" -> ddlAddColumnComment, + "ddl.addColumn.position" -> ddlAddColumnPosition, + "ddl.alterColumn.typeWiden" -> ddlAlterColumnTypeWiden, + "ddl.renameColumn" -> ddlRenameColumn + ) + + /** The operations crossed with every layout, each a headless segment, in report order. */ + val operations: List[(String, TableTest[CoreTable.type])] = List( + "read.projection" -> readProjection, + "read.filter" -> readFilter, + "format.materialization" -> formatMaterialization, + "delete.byPredicate" -> deleteByPredicate, + "delete.byInList" -> deleteByInList, + "delete.byInSubquery" -> deleteByInSubquery, + "delete.byNotInSubquery" -> deleteByNotInSubquery, + "delete.byExistsSubquery" -> deleteByExistsSubquery, + "delete.byNotExistsSubquery" -> deleteByNotExistsSubquery, + "delete.byScalarSubquery" -> deleteByScalarSubquery, + "delete.byNullCondition" -> deleteByNullCondition, + "delete.all" -> deleteAll, + "delete.none" -> deleteNone, + "delete.byPartitionPredicate" -> deleteByPartitionPredicate, + "delete.withAlias" -> deleteWithAlias, + "delete.whereFalse.noSnapshot" -> deleteWhereFalseKeepsSnapshot, + "delete.truncate" -> truncate, + "delete.atSnapshot.rejected" -> deleteAtSnapshotRejected, + "update.byPredicate" -> updateByPredicate, + "update.withoutCondition" -> updateWithoutCondition, + "update.noMatch" -> updateNoMatch, + "update.byInSubquery" -> updateByInSubquery, + "update.byNotInSubquery" -> updateByNotInSubquery, + "update.byExistsSubquery" -> updateByExistsSubquery, + "update.byNotExistsSubquery" -> updateByNotExistsSubquery, + "update.byScalarSubquery" -> updateByScalarSubquery, + "update.withAlias" -> updateWithAlias, + "update.multipleColumns" -> updateMultipleColumns, + "update.byExpression" -> updateByExpression, + "update.movePartition" -> updateMovePartition, + "update.nullAssignment" -> updateNullAssignment, + "merge.insertNotMatched" -> mergeInsertNotMatched, + "merge.updateMatched" -> mergeUpdateMatched, + "merge.deleteMatched" -> mergeDeleteMatched, + "merge.upsert" -> mergeUpsert, + "merge.deleteNotMatchedBySource" -> mergeDeleteNotMatchedBySource, + "merge.conditionalUpdate" -> mergeConditionalUpdate, + "merge.multipleMatchedClauses" -> mergeMultipleMatchedClauses, + "merge.conditionalInsert" -> mergeConditionalInsert, + "merge.allClauses" -> mergeAllClauses, + "merge.updateStar" -> mergeUpdateStar, + "merge.insertExplicitColumns" -> mergeInsertExplicitColumns, + "merge.sourceCTE" -> mergeSourceCTE, + "merge.sourceSetOp" -> mergeSourceSetOp, + "merge.intoEmptyTarget" -> mergeIntoEmptyTarget, + "merge.nullJoinKey" -> mergeNullJoinKey, + "merge.resolveByName" -> mergeResolveByName, + "insert.into" -> insertInto, + "insert.explicitColumns" -> insertExplicitColumns, + "insert.intoSelect" -> insertIntoSelect, + "append.dataFrame" -> appendDataFrame, + "insert.overwrite" -> insertOverwrite, + "overwrite.dataFrame" -> overwriteDataFrame + ) + + /** Operations meaningful only on a partitioned table; crossed with the partitioned layouts only. */ + val partitionedOperations: List[(String, TableTest[CoreTable.type])] = List( + "insert.dynamicOverwrite" -> insertDynamicOverwrite, + "overwrite.partitions" -> overwritePartitions + ) + + /** The DELETE/UPDATE/MERGE subset — the operations affected by the CoW-vs-MoR mode. */ + val mutationOperations: List[(String, TableTest[CoreTable.type])] = + operations.filter { case (name, _) => + name.startsWith("delete.") || name.startsWith("update.") || name.startsWith("merge.") + } + + // ── MoR discriminator: prove merge-on-read actually wrote position-delete files ────────── + // The rest of the MoR axis reuses CoW's row-delta assertions, which pass identically whether the + // write was copy-on-write or merge-on-read. These two pin the PHYSICAL difference: a MoR delete + // MUST add a position-delete file; a CoW delete must NOT. Both are prepared with + // `createAndSeedSingleFile` and delete a strict subset (`long0 < 2` → 1 of 3 rows), so the write + // cannot be satisfied by whole-file elimination — the outcome is deterministic across formats + // (verified: parquet/orc/avro all add exactly one position delete under MoR, none under CoW). + private def deleteFileCount(spark: SparkSession, table: String): Long = + spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) + + val morWritesDeleteFiles: TableTest[CoreTable.type] = + TableTest(Core).delete(core => s"${core.long0.columnName} < 2") { view => + assert(view.after == view.before.filterNot(_.get(Core.long0) < 2)) // rows correct + assert(deleteFileCount(view.spark, view.table) >= 1, + "merge-on-read DELETE of a strict subset of a data file must write a position-delete file") + } + + val cowWritesNoDeleteFiles: TableTest[CoreTable.type] = + TableTest(Core).delete(core => s"${core.long0.columnName} < 2") { view => + assert(view.after == view.before.filterNot(_.get(Core.long0) < 2)) + assert(deleteFileCount(view.spark, view.table) == 0, "copy-on-write DELETE must not write delete files") + } + + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala new file mode 100644 index 000000000..967abb7c0 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala @@ -0,0 +1,217 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +/** Runs a case, retrying only a transient-infrastructure failure. */ +object Runner { + val MaxAttempts = 3 + + def execute(c: Plan.Case, ctx: Ctx): (Outcome, Int) = { + @tailrec def attempt(n: Int): (Outcome, Int) = { + val outcome = + try { c.run(ctx); Outcome.Passed } + catch { case NonFatal(t) => Outcome.Failed(t) } + outcome match { + case f: Outcome.Failed if f.retryable && n + 1 < MaxAttempts => attempt(n + 1) + case terminal => (terminal, n + 1) + } + } + attempt(0) + } +} + +// Boot app for the REAL House Table Service as a 2nd Spring context in-JVM (HTS-embed, Option A). +// Mirrors services/.../e2e/SpringH2HtsApplication's annotation set (test-scope, so replicated here). +// Security auto-config is excluded (spring-security-web is only partially present on the harness +// classpath, and the harness runs unauthenticated) — exactly as the tables boot does. +// internal.catalog.mapper is intentionally NOT scanned (a client-side concern needing FileIOManager; +// the HTS server does not use it). Proven by HtsBootProbe. +@org.springframework.boot.autoconfigure.SpringBootApplication( + exclude = Array( + classOf[org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration], + classOf[org.springframework.boot.actuate.autoconfigure.security.servlet.ManagementWebSecurityAutoConfiguration])) +@org.springframework.context.annotation.ComponentScan(basePackages = Array( + "com.linkedin.openhouse.housetables.api", + "com.linkedin.openhouse.housetables.dto.mapper", + "com.linkedin.openhouse.housetables.controller", + "com.linkedin.openhouse.housetables.services", + "com.linkedin.openhouse.common.exception.handler", + "com.linkedin.openhouse.common.audit", + "com.linkedin.openhouse.housetables.repository", + "com.linkedin.openhouse.housetables.properties", + "com.linkedin.openhouse.housetables.config", + "com.linkedin.openhouse.cluster.configs", + "com.linkedin.openhouse.cluster.storage")) +@org.springframework.boot.autoconfigure.domain.EntityScan( + basePackages = Array("com.linkedin.openhouse.housetables.model")) +class HtsBootApp + +/** Boots the embedded real House Table Service (H2, MySQL-mode) as its own Spring context. */ +object HtsEnv { + import org.springframework.boot.builder.SpringApplicationBuilder + import org.springframework.boot.web.context.WebServerApplicationContext + import org.springframework.context.ConfigurableApplicationContext + + /** @return (context, base-uri) for the embedded HTS. */ + def start(): (ConfigurableApplicationContext, String) = { + val root = System.getProperty("java.io.tmpdir") + "/hts-embed" + val ctx = new SpringApplicationBuilder(classOf[HtsBootApp]) + .properties( + "server.port=0", + "cluster.storage.root-path=" + root, + "cluster.tables.allowed-client-name-values=trino,spark") + .run() + val port = ctx.asInstanceOf[WebServerApplicationContext].getWebServer.getPort + (ctx, s"http://localhost:$port") + } +} + +/** Boots the embedded OpenHouse server and wires a SparkSession to the OpenHouse catalog. */ +object OpenHouseEnv { + import com.linkedin.openhouse.tablestest.OpenHouseLocalServer + import org.springframework.context.ConfigurableApplicationContext + + private def authToken(): String = + Option(getClass.getClassLoader.getResourceAsStream("dummy.token")) + .map(is => scala.io.Source.fromInputStream(is, "UTF-8").mkString.trim) + .getOrElse("default-token") + + private def wireCatalog(builder: SparkSession.Builder, name: String, uri: String, token: String): SparkSession.Builder = + builder + .config(s"spark.sql.catalog.$name", "org.apache.iceberg.spark.SparkCatalog") + .config(s"spark.sql.catalog.$name.catalog-impl", "com.linkedin.openhouse.spark.OpenHouseCatalog") + .config(s"spark.sql.catalog.$name.uri", uri) + .config(s"spark.sql.catalog.$name.cluster", "local-cluster") + .config(s"spark.sql.catalog.$name.auth-token", token) + + def start(): (OpenHouseLocalServer, SparkSession, String, String, Option[ConfigurableApplicationContext]) = { + // HTS-embed (Option A): when HARNESS_REAL_HTS=1, boot the real House Table Service as a 2nd + // Spring context, point the embedded tables server's HouseTableRepositoryImpl at it via + // cluster.housetables.base-uri, and disable the @Primary in-memory stub (openhouse.htsStub.enabled + // =false) so the real HTTP client is the sole HouseTableRepository. Default (flag unset) keeps the + // stub — the existing green baseline is always reproducible. + val realHts = sys.env.get("HARNESS_REAL_HTS").contains("1") + val htsCtxOpt: Option[ConfigurableApplicationContext] = + if (realHts) { + // Boot the HTS context FIRST, while no spring.sql.init.mode System property is set, so it + // uses its own application.properties (spring.sql.init.mode=always) and runs schema.sql + + // data.sql on its MySQL-mode H2. The tables-context suppression props below are set AFTER + // this returns (the HTS context is already fully refreshed), so they don't affect HTS. + val (ctx, htsUri) = HtsEnv.start() + HtsAdmin.htsUri = htsUri // enables the undrop preparation axis (Phase 4) + System.setProperty("cluster.housetables.base-uri", htsUri) + System.setProperty("openhouse.htsStub.enabled", "false") + println(s">> REAL HTS mode: embedded HTS at $htsUri (stub disabled)") + Some(ctx) + } else None + + // ALWAYS (both stub and real-HTS modes): housetables-lib.jar is on the harness classpath + // unconditionally (print-cp.init.gradle pulls it in for the real-HTS path). Its root + // data.sql/schema.sql are MySQL-dialect and would be auto-run by the TABLES context's H2 + // (non-MySQL mode) → INSERT IGNORE syntax error. The tables side ships no SQL scripts and relies + // on Hibernate auto-DDL, so (i) never run classpath SQL init for it, and (ii) make auto-DDL + // explicit (the stray schema.sql otherwise flips Spring Boot's embedded-H2 ddl-auto default to + // `none`, leaving the tables server's own H2 tables — feature-toggle status/rules — missing). + // In real-HTS mode this runs AFTER HtsEnv.start(), so the HTS schema (which needs init) is safe. + System.setProperty("spring.sql.init.mode", "never") + System.setProperty("spring.jpa.hibernate.ddl-auto", "create-drop") + + val server = new OpenHouseLocalServer() + server.start() + val uri = s"http://localhost:${server.getPort}" + val token = authToken() + + val base = SparkSession.builder() + .appName("delta-harness-openhouse") + .master("local[2]") + .config("spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions," + + "com.linkedin.openhouse.spark.extensions.OpenhouseSparkSessionExtensions") + .config("spark.hadoop.fs.defaultFS", "file:///") + .config("spark.sql.session.timeZone", "UTC") + .config("spark.sql.autoBroadcastJoinThreshold", "-1") + .config("spark.driver.bindAddress", "127.0.0.1") + .config("spark.ui.enabled", "false") + + val wired = Seq("openhouse", "default_iceberg").foldLeft(base)(wireCatalog(_, _, uri, token)) + (server, wired.getOrCreate(), uri, token, htsCtxOpt) + } +} + +object Main { + def main(args: Array[String]): Unit = { + val (server, spark, restUri, restToken, htsCtxOpt) = OpenHouseEnv.start() + spark.sparkContext.setLogLevel("ERROR") + HtsAdmin.tablesUri = restUri; HtsAdmin.token = restToken // undrop restore path (Phase 4) + val ctx = Ctx(spark, "openhouse.dbMatrix", restUri, restToken) + + // Each command-line arg is an include-substring; a case runs only if its id contains ALL of + // them (AND). No args = run everything. + val filters = args.toList + def selected(id: String): Boolean = filters.forall(id.contains) + val cases = Plan.cases.filter(c => selected(c.id)) + + val header = if (filters.isEmpty) "all cases" else s"filter ${filters.mkString(", ")} -> ${cases.size} cases" + println(s"\n=== delta-harness :: typed pipelines @ OpenHouse catalog ($header) ===\n") + + // Known-bug cases are tagged (Plan.knownBugs) and reported SKIP rather than run — deferred, + // not passing. Everything else executes. + // + // Cases are independent (each owns its table via the atomic counter), so they run on a worker + // pool. Each worker task gets its OWN SparkSession (spark.newSession(): separate SQLConf — + // isolating the session-global state some tests mutate, e.g. spark.wap.branch/wap.id and + // changelog temp views — over the shared SparkContext). Results are collected and printed in + // the original case order, so output is identical to a sequential run. + // HARNESS_PARALLELISM overrides; <=1 falls back to the sequential path. + val parallelism = sys.env.get("HARNESS_PARALLELISM").map(_.toInt) + .getOrElse(math.max(1, Runtime.getRuntime.availableProcessors())) + println(s"parallelism: $parallelism worker sessions\n") + + def runOne(c: Plan.Case): (String, (Outcome, Int)) = + Plan.bugReason(c.id) match { + case Some(reason) => (c.id, (Outcome.Skipped(reason): Outcome, 0)) + case None => (c.id, Runner.execute(c, ctx.copy(spark = ctx.spark.newSession()))) + } + + val results = + if (parallelism <= 1) cases.map(runOne) + else { + val pool = java.util.concurrent.Executors.newFixedThreadPool(parallelism) + try { + val futures = cases.map(c => pool.submit(new java.util.concurrent.Callable[(String, (Outcome, Int))] { + def call(): (String, (Outcome, Int)) = runOne(c) + })) + futures.map(_.get(60, java.util.concurrent.TimeUnit.MINUTES)) + } finally pool.shutdown() + } + + results.foreach { case (id, (outcome, attempts)) => + val note = outcome match { + case f: Outcome.Failed => s" (${f.reason}${if (f.retryable) " [retryable]" else ""})" + case Outcome.Skipped(reason) => s" ($reason)" + case Outcome.Passed => "" + } + println(f"${outcome.label}%-4s ${id}%-52s try=$attempts$note") + } + + val failed = results.count { case (_, (outcome, _)) => outcome.isInstanceOf[Outcome.Failed] } + val skipped = results.count { case (_, (outcome, _)) => outcome.isInstanceOf[Outcome.Skipped] } + val passed = results.size - failed - skipped + println(f"\n$passed passed, $skipped skipped, $failed failed (${results.size} cases)") + if (passed == 0) println("WARNING: no case actually passed (empty selection or all skipped) — reporting failure") + + try spark.stop() catch { case _: Throwable => () } + try server.stop() catch { case _: Throwable => () } + htsCtxOpt.foreach(ctx => try ctx.close() catch { case _: Throwable => () }) + // A run that validated nothing (0 cases, or everything skipped) is NOT success. + System.exit(if (failed == 0 && passed > 0) 0 else 1) + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala new file mode 100644 index 000000000..d3217d8cf --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala @@ -0,0 +1,511 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +trait ForkScenarios extends ScenarioKit { + import Rows._ + + // ── Column-default (fork #251) — OSS Spark DDL path ────────────────────────────────────────── + // Column defaults are TABLED (see ICEBERG-FORK-AUDIT.md). This test characterizes what the OSS Spark 3.5 + // DDL path does with `ALTER TABLE t ADD COLUMN c int DEFAULT 5`; the behavior is identical on the + // published 1.5.2.15 and the branch build (#251 is api/core only, with no Spark write wiring). Measured: + // • accepted at Spark parse time (Spark 3.5 owns the DEFAULT grammar); + // • the default is not written into the Iceberg schema (DESCRIBE shows `c|int|null`, no default); + // • pre-existing rows read NULL; + // • an INSERT that omits the column is rejected INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA + // (same root as bug1 — no column-default write wiring in the connector). + // These are behavior pins: if a future build changes any of the above, the asserts flip and it is re-audited. + private def forkColDefaultAddColumn(fmt: String)(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_coldef_$fmt" + spark.sql(s"DROP TABLE IF EXISTS $table") + spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')") + spark.sql(s"INSERT INTO $table VALUES (1, 'a'), (2, 'b')") + + // (1) The customer path is ACCEPTED at parse time (Spark owns the grammar) — pin no-throw. + spark.sql(s"ALTER TABLE $table ADD COLUMN c int DEFAULT 5") + + // (2) The default is not written into the persisted schema — column c has no default metadata. + val cDesc = spark.sql(s"DESCRIBE TABLE EXTENDED $table").collect() + .map(_.mkString("|")).filter(_.matches("(?i)^c\\|.*")).mkString(" ;; ") + assert(!cDesc.toLowerCase.contains("default") && !cDesc.contains("5"), + s"[$fmt] expected no default persisted for c, but DESCRIBE shows: $cDesc — a #251-containing build may now be wired; re-audit") + + // (3) The default is NOT backfilled on read — pre-existing rows read NULL, not 5. + val nulls = spark.sql(s"SELECT count(*) FROM $table WHERE c IS NULL").collect()(0).getLong(0) + assert(nulls == 2, + s"[$fmt] expected the default NOT applied on read (2 NULLs), got $nulls — a #251-containing build may now apply defaults; re-audit") + + // (4) The default is NOT applied on write — an insert that omits c is rejected (no write wiring). + val omit = Check.intercept[org.apache.spark.sql.AnalysisException] { + spark.sql(s"INSERT INTO $table (id, s) VALUES (3, 'c')") + } + val omitMsg = Exceptions.causeChain(omit).flatMap(e => Option(e.getMessage)).mkString(" | ") + assert(omitMsg.contains("CANNOT_FIND_DATA"), + s"[$fmt] expected omit-insert rejected with CANNOT_FIND_DATA (no column-default write wiring), got: $omitMsg") + + println(s"DIAG fork.colDefault[$fmt]: accepted=yes persistedDefault=no readBackfill=no writeApply=no(CANNOT_FIND_DATA)") + spark.sql(s"DROP TABLE IF EXISTS $table") + } + + // ── Column-default (fork #251) — SchemaParser serialization ────────────────────────────────────── + // Characterizes the api/core surface of #251: NestedField carries `initial-default`/`write-default` and + // SchemaParser serializes them into the schema JSON. `toJson` takes no format-version parameter, so the + // key serializes regardless of the table's format version. Exercised directly via reflection so the SAME + // source compiles and runs in BOTH artifacts: + // • published 1.5.2.15 → NestedField.builder() is absent → records "API unsupported"; + // • branch HEAD (#251) → builds a defaulted field, checks SchemaParser emits `initial-default` and + // that it round-trips (fromJson→toJson). + // Reflection (not direct calls) is required because the builder API does not exist in the release jar; + // a direct reference would not COMPILE in default (release) mode. + private def forkColDefaultApiSerialization(ctx: Ctx): Unit = { + val nestedFieldCls = Class.forName("org.apache.iceberg.types.Types$NestedField") + val builderM = scala.util.Try(nestedFieldCls.getMethod("builder")) + if (builderM.isFailure) { + // Published release: the #251 column-default API is absent. Pin that absence (feature not present). + println("DIAG fork.colDefault.api: NestedField.builder ABSENT — #251 column-default API unsupported (published release artifact)") + val ms = nestedFieldCls.getMethods.map(_.getName).toSet + assert(!ms.contains("initialDefault") && !ms.contains("writeDefault"), + "NestedField exposes initial/write-default accessors but no builder() — unexpected partial #251; re-audit") + return + } + // Branch HEAD: #251 present. Build `optional int c` carrying initial-default=5 via the builder. + val builder0 = builderM.get.invoke(null) + def chain(b: AnyRef, m: String, argT: Class[_], arg: AnyRef): AnyRef = + b.getClass.getMethod(m, argT).invoke(b, arg) + def chain0(b: AnyRef, m: String): AnyRef = b.getClass.getMethod(m).invoke(b) + val intType = Class.forName("org.apache.iceberg.types.Types$IntegerType") + .getMethod("get").invoke(null) + var b = chain(builder0, "withId", java.lang.Integer.TYPE, java.lang.Integer.valueOf(3)) + b = chain(b, "withName", classOf[String], "c") + b = chain(b, "ofType", Class.forName("org.apache.iceberg.types.Type"), intType) + b = chain0(b, "asOptional") + b = chain(b, "withInitialDefault", classOf[Object], java.lang.Integer.valueOf(5)) + val field = b.getClass.getMethod("build").invoke(b) + .asInstanceOf[org.apache.iceberg.types.Types.NestedField] + + // Assemble a schema [id, c(default=5)] and serialize it — no format version is even passed. + val idField = org.apache.iceberg.types.Types.NestedField.required( + 1, "id", org.apache.iceberg.types.Types.LongType.get()) + val schema = new org.apache.iceberg.Schema(java.util.Arrays.asList(idField, field)) + val json = org.apache.iceberg.SchemaParser.toJson(schema) + println(s"DIAG fork.colDefault.api: #251 PRESENT; serialized schema JSON = $json") + + // (a) The default is serialized into the schema JSON. + assert(json.contains("initial-default"), + s"expected #251 SchemaParser to serialize 'initial-default' into the schema JSON, got: $json") + // (b) toJson takes no format-version argument — the key serializes the same regardless of format version. + // (c) Round-trips through fromJson→toJson. + val reparsed = org.apache.iceberg.SchemaParser.fromJson(json) + val json2 = org.apache.iceberg.SchemaParser.toJson(reparsed) + assert(json2.contains("initial-default"), + s"expected 'initial-default' to survive fromJson->toJson round-trip, got: $json2") + println("DIAG fork.colDefault.api: initial-default serialized (no format-version argument) + round-trips") + } + + // Reflectively build an `optional int` NestedField carrying initial-default=`dflt` (the #251 builder). + // Returns None when the API is absent (published release) so callers can pin that cleanly. + private def buildDefaultedIntField(id: Int, name: String, dflt: Int): Option[org.apache.iceberg.types.Types.NestedField] = { + val nfCls = Class.forName("org.apache.iceberg.types.Types$NestedField") + val bm = scala.util.Try(nfCls.getMethod("builder")) + if (bm.isFailure) return None + def chain(b: AnyRef, m: String, at: Class[_], a: AnyRef): AnyRef = b.getClass.getMethod(m, at).invoke(b, a) + def chain0(b: AnyRef, m: String): AnyRef = b.getClass.getMethod(m).invoke(b) + val intType = Class.forName("org.apache.iceberg.types.Types$IntegerType").getMethod("get").invoke(null) + var b = chain(bm.get.invoke(null), "withId", java.lang.Integer.TYPE, java.lang.Integer.valueOf(id)) + b = chain(b, "withName", classOf[String], name) + b = chain(b, "ofType", Class.forName("org.apache.iceberg.types.Type"), intType) + b = chain0(b, "asOptional") + b = chain(b, "withInitialDefault", classOf[Object], java.lang.Integer.valueOf(dflt)) + Some(b.getClass.getMethod("build").invoke(b).asInstanceOf[org.apache.iceberg.types.Types.NestedField]) + } + + // ── Column-default (fork #251) — READ-APPLY characterization PROBE (TABLED / not a bug claim) ───── + // TABLED per repo owner: "it is not fundamentally broken … if there is a gap, it's implemented somewhere." + // This probe records, but does NOT assert a verdict on, what THIS harness config does — i.e. the OSS + // Spark 3.5 read path over branch iceberg-core. It does NOT exercise LinkedIn's PRIVATE Spark fork, which + // is the likely home of the missing-column read-application. So a NULL here is a property of this harness, + // NOT proof the feature is broken. Left as a DIAG-only probe (asserts only the undisputed half: the + // default persists into the committed schema). Revisit when default values are un-tabled AND the private + // Spark reader is available to test against. + private def forkColDefaultReadApplyProbe(ctx: Ctx): Unit = { + val spark = ctx.spark + val nfCls = Class.forName("org.apache.iceberg.types.Types$NestedField") + val apiPresent = scala.util.Try(nfCls.getMethod("builder")).isSuccess + if (!apiPresent) { + // Published release: no way to set a default, so there is nothing to read back. Assert the API is + // genuinely absent (so this is not a silent green) and return. + println("DIAG fork.colDefault.readApplyProbe: #251 API absent (published release) — nothing to probe") + assert(!nfCls.getMethods.map(_.getName).toSet.contains("initialDefault"), + "NestedField exposes initialDefault but builder() is absent — unexpected partial #251; re-audit") + return + } + val cat = "coldefroapply" + val wh = s"/tmp/coldef-readapply-${System.nanoTime()}" + spark.conf.set(s"spark.sql.catalog.$cat", "org.apache.iceberg.spark.SparkCatalog") + spark.conf.set(s"spark.sql.catalog.$cat.type", "hadoop") + spark.conf.set(s"spark.sql.catalog.$cat.warehouse", wh) + val t = s"$cat.d.t_readapply" + spark.sql(s"DROP TABLE IF EXISTS $t") + spark.sql(s"CREATE TABLE $t (id bigint) USING $dataSource") + spark.sql(s"INSERT INTO $t VALUES (1),(2)") // data files physically contain ONLY `id` + + // Set a column default the way a private engine would: evolve the schema to [id, c int DEFAULT 5] via + // the low-level TableMetadata API (public UpdateSchema has no set-default op on the branch). + val table = org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, t) + val cur = table.schema() + val nextId = cur.highestFieldId() + 1 + val cField = buildDefaultedIntField(nextId, "c", 5).getOrElse( + throw new AssertionError("#251 builder present but field build failed")) + val cols = new java.util.ArrayList[org.apache.iceberg.types.Types.NestedField](cur.columns()) + cols.add(cField) + val s2 = new org.apache.iceberg.Schema(cols) + val ops = table.asInstanceOf[org.apache.iceberg.HasTableOperations].operations() + val base = ops.current() + val updated = org.apache.iceberg.TableMetadata.buildFrom(base).setCurrentSchema(s2, s2.highestFieldId()).build() + ops.commit(base, updated) + + // ASSERT only the undisputed half: the default persists into the committed schema (ungated). + val persisted = org.apache.iceberg.SchemaParser.toJson( + org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, t).schema()) + assert(persisted.contains("initial-default"), + s"expected initial-default to persist into the committed schema, got: $persisted") + + // DIAG only — record what the OSS-Spark read path returns here; NO verdict (read-apply may live in the + // private Spark reader not exercised by this harness). + spark.sql(s"REFRESH TABLE $t") + val vals = spark.sql(s"SELECT c FROM $t ORDER BY id").collect() + .map(r => if (r.isNullAt(0)) "NULL" else r.getInt(0).toString) + println(s"DIAG fork.colDefault.readApplyProbe: OSS-Spark read of defaulted col over old files = " + + s"[${vals.mkString(",")}] (harness-config observation only; private Spark reader NOT tested; TABLED)") + spark.sql(s"DROP TABLE IF EXISTS $t") + } + + val forkColDefaultOps: List[(String, Ctx => Unit)] = List( + "fork.colDefault.addColumnInert @ parquet" -> forkColDefaultAddColumn("parquet"), + "fork.colDefault.addColumnInert @ orc" -> forkColDefaultAddColumn("orc"), + "fork.colDefault.apiSerialization @ core" -> forkColDefaultApiSerialization, + "fork.colDefault.readApplyProbe @ core" -> forkColDefaultReadApplyProbe + ) + + // ── #249 (d69c1fd91) — partitioned write distribution default ───────────────────────────────────── + // The fork changes the DEFAULT write.distribution-mode for PARTITIONED writes from Apache's HASH to + // NONE (Spark 3.5). With HASH, the writer shuffles rows so each partition is written by one task -> + // ~(#partitions) data files. With NONE, no shuffle -> each input task writes every partition it holds + // -> up to (#tasks × #partitions) files. This test appends the SAME multi-task DataFrame into a + // 4-partition table twice — once with the default, once with an explicit HASH — and compares the data- + // file counts. It pins that (a) explicit HASH clusters to ~#partitions, and (b) the default does not + // cluster more than HASH. Run under both runtimes via ICEBERG_RUNTIME_JAR: the DIAG file counts show + // the branch-vs-release difference (fork NONE default -> more files than a HASH-default build). + private def forkPartitionDistDefault(fmt: String)(ctx: Ctx): Unit = { + val spark = ctx.spark + val nParts = 4 + val nTasks = 8 + def buildAndCountFiles(tbl: String, extraProps: String): Long = { + spark.sql(s"DROP TABLE IF EXISTS $tbl") + spark.sql(s"CREATE TABLE $tbl (id bigint, p int) USING $dataSource PARTITIONED BY (p) " + + s"TBLPROPERTIES ('format-version'='2', 'write.format.default'='$fmt'$extraProps)") + // nTasks input partitions, each holding rows for all nParts table partitions. + val df = spark.range(0, 400) + .selectExpr("id", s"cast(id % $nParts as int) as p") + .repartition(nTasks) + df.writeTo(tbl).append() + val n = spark.sql(s"SELECT count(*) FROM $tbl.data_files").collect()(0).getLong(0) + spark.sql(s"DROP TABLE IF EXISTS $tbl") + n + } + val nDefault = buildAndCountFiles(s"${ctx.namespace}.t_dist_def_$fmt", "") + val nHash = buildAndCountFiles(s"${ctx.namespace}.t_dist_hash_$fmt", ", 'write.distribution-mode'='hash'") + println(s"DIAG fork.partitionDist[$fmt]: defaultFiles=$nDefault hashFiles=$nHash " + + s"(parts=$nParts tasks=$nTasks; default==hash => HASH-default build, default>hash => NONE-default #249)") + // (a) Explicit HASH clusters by partition -> roughly one file per partition (allow slack for spill). + assert(nHash <= nParts * 2, + s"[$fmt] write.distribution-mode=hash should cluster to ~$nParts files, got $nHash") + // (b) The default never clusters MORE than HASH (fork default is NONE => >=; never <). + assert(nDefault >= nHash, + s"[$fmt] default partitioned distribution produced FEWER files than HASH (default=$nDefault hash=$nHash) — unexpected; re-audit #249") + } + + val forkPartitionDistOps: List[(String, Ctx => Unit)] = List( + "fork.partitionDist.default @ parquet" -> forkPartitionDistDefault("parquet"), + "fork.partitionDist.default @ orc" -> forkPartitionDistDefault("orc") + ) + + // (count, sumBytes) of the CURRENT data files — used by the compaction fork probes below. + private def dataFileStats(spark: SparkSession, table: String): (Long, Long) = { + val r = spark.sql(s"SELECT count(*), coalesce(sum(file_size_in_bytes), 0) FROM $table.data_files").collect()(0) + (r.getLong(0), r.getLong(1)) + } + + private def showProps(spark: SparkSession, table: String): Map[String, String] = + spark.sql(s"SHOW TBLPROPERTIES $table").collect().toSeq.map(r => r.getString(0) -> r.getString(1)).toMap + + // ── #229 (write.delete-file-replication) — MoR delete-file HDFS replication factor ─────────────────── + // TableProperties.DELETE_FILE_REPLICATION = "write.delete-file-replication". SparkWriteConf resolves it + // (sessionConf spark.sql.iceberg.delete-file-replication > tableProperty write.delete-file-replication > + // option > default 3) into a `short` that SparkPositionDeltaWrite / SparkPositionDeletesRewrite feed to + // OutputFileFactory.replicationFactor(short); the factory stamps it onto the delete file's FileIO output + // properties so HDFS sets that block-replication on the position-delete file. The HDFS replication itself + // is NOT observable on the local FS this harness runs on — so this is an accepted LOW-observability pin: + // • the property round-trips through the OpenHouse catalog metadata (SHOW TBLPROPERTIES); + // • a MoR DELETE physically writes a position-delete file (the path that consumes the factor); + // • the DML result is correct and the property survives the mutation. + private def forkDeleteFileReplication(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_delrepl" + spark.sql(s"DROP TABLE IF EXISTS $table") + // MoR + unpartitioned + distribution=none so one seed INSERT lands ONE data file; a partial DELETE is + // then necessarily a position delete (not whole-file elimination) — the delete-file write path. + spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES (" + + s"'format-version'='2', 'write.distribution-mode'='none', 'write.delete.mode'='merge-on-read', " + + s"'write.update.mode'='merge-on-read', 'write.delete-file-replication'='2')") + // COALESCE(1) => a single data file, so deleting a strict subset is a PARTIAL-file match that MoR + // must satisfy with a position-delete file (not whole-file elimination). + spark.sql(s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM (VALUES (1L,'a'),(2L,'b'),(3L,'c')) AS s(id, s)") + + // (1) The property round-trips through the OpenHouse catalog metadata. + val p1 = showProps(spark, table) + assert(p1.get("write.delete-file-replication").contains("2"), + s"expected write.delete-file-replication=2 to round-trip, got ${p1.get("write.delete-file-replication")}") + + // (2) A MoR DELETE writes a position-delete file (the write path that consumes the replication factor). + spark.sql(s"DELETE FROM $table WHERE id = 1") + val delFiles = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) + assert(delFiles >= 1, s"MoR DELETE should write a position-delete file, got $delFiles") + + // (3) DML result is correct (the replication factor never alters the logical row set). + val rows = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) + assert(rows == Seq(2L, 3L), s"expected [2,3] after MoR delete, got $rows") + + // (4) The property survives the mutation (still honored in metadata after the delete-file write). + val p2 = showProps(spark, table) + assert(p2.get("write.delete-file-replication").contains("2"), "write.delete-file-replication lost after DELETE") + + println(s"DIAG fork.deleteFileReplication: prop=2 roundtrips=yes deleteFiles=$delFiles rows=${rows.mkString(",")} " + + s"(HDFS block-replication not observable on local FS; property honored in metadata + MoR DML unaffected)") + spark.sql(s"DROP TABLE IF EXISTS $table") + } + + val forkDeleteFileReplicationOps: List[(String, Ctx => Unit)] = List( + "fork.deleteFileReplication @ mor" -> forkDeleteFileReplication + ) + + // ── #219 (OutputFileFactory.FILE_REPLICATION_FACTOR) — output-file replication factor ───────────────── + // KEY CORRECTION: the constant is FILE_REPLICATION_FACTOR = "file-replication-factor" — NOT the guessed + // "write.file-replication-factor", and it is NOT a settable table property at all. It is the per-output- + // file property KEY that OutputFileFactory stamps into the FileIO property map when a replicationFactor + // is present (getProperties()), consumed by HDFS to set the file's block replication. The ONLY caller + // that feeds a replicationFactor is the DELETE-file path (SparkPositionDeltaWrite/SparkPositionDeletesRewrite, + // via SparkWriteConf.deleteFileReplication()) — data-file factories never set it. So #219 is the low-level + // OutputFileFactory API manifestation of the same mechanism as #229, pinned at the API surface where it IS + // observable: build the factory with a factor and assert it stamps FILE_REPLICATION_FACTOR into the output- + // file property map. Reflection is used for the fork-only builder method + the private getProperties() so + // this source compiles against a stock artifact too. + private def forkFileReplicationFactor(ctx: Ctx): Unit = { + val spark = ctx.spark + val offCls = Class.forName("org.apache.iceberg.io.OutputFileFactory") + + // (1) Pin the EXACT key string (corrects the common mis-guess "write.file-replication-factor"). + val keyFieldT = scala.util.Try(offCls.getField("FILE_REPLICATION_FACTOR")) + assert(keyFieldT.isSuccess, "OutputFileFactory.FILE_REPLICATION_FACTOR absent — replication-factor fork feature missing") + val key = keyFieldT.get.get(null).asInstanceOf[String] + assert(key == "file-replication-factor", + s"""expected FILE_REPLICATION_FACTOR == "file-replication-factor" (an output-file property key, NOT a "write." table prop), got "$key"""") + + // Need a real Iceberg Table to build a factory. + val table = s"${ctx.namespace}.t_filerepl" + spark.sql(s"DROP TABLE IF EXISTS $table") + spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES ('format-version'='2')") + spark.sql(s"INSERT INTO $table VALUES (1,'a'),(2,'b')") + val icebergTable = org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, table) + + // (2) Build an OutputFileFactory carrying replicationFactor=2 via the fork builder (reflected — the + // .replicationFactor(short) method is a fork addition). + val builder = offCls.getMethod("builderFor", classOf[org.apache.iceberg.Table], java.lang.Integer.TYPE, java.lang.Long.TYPE) + .invoke(null, icebergTable, java.lang.Integer.valueOf(1), java.lang.Long.valueOf(1L)) + val replMT = scala.util.Try(builder.getClass.getMethod("replicationFactor", java.lang.Short.TYPE)) + assert(replMT.isSuccess, "OutputFileFactory.Builder.replicationFactor(short) absent — replication fork missing") + replMT.get.invoke(builder, java.lang.Short.valueOf(2.toShort)) + val factory = builder.getClass.getMethod("build").invoke(builder) + assert(factory != null, "OutputFileFactory build returned null") + + // (3) OBSERVABLE: the factory stamps FILE_REPLICATION_FACTOR -> "2" into the per-output-file property + // map it hands the FileIO. getProperties() is private -> reflect it. + val gp = offCls.getDeclaredMethod("getProperties"); gp.setAccessible(true) + val props = gp.invoke(factory).asInstanceOf[java.util.Map[String, String]] + assert(props.get(key) == "2", + s"expected output-file property $key=2 stamped by the factory, got ${props.get(key)}") + + // (4) Writes still succeed and rows are correct (the factor never corrupts the data path). + spark.sql(s"INSERT INTO $table VALUES (3,'c')") + val rows = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) + assert(rows == Seq(1L, 2L, 3L), s"rows wrong after write: $rows") + + println(s"DIAG fork.fileReplicationFactor: key='$key' (corrected from guessed 'write.file-replication-factor'); " + + s"factory stamps $key=${props.get(key)} into output-file props; writes ok rows=${rows.mkString(",")}") + spark.sql(s"DROP TABLE IF EXISTS $table") + } + + val forkFileReplicationFactorOps: List[(String, Ctx => Unit)] = List( + "fork.fileReplicationFactor @ core" -> forkFileReplicationFactor + ) + + // ── #228 (spark.sql.iceberg.split-size) — Spark read split size ─────────────────────────────────────── + // SparkSQLProperties.SPLIT_SIZE = "spark.sql.iceberg.split-size". Set via spark.conf.set; SparkReadConf + // uses it to combine/split data files into read tasks. This one IS observable: with several small files, + // a large split-size combines them into FEWER read tasks and a tiny split-size splits into MORE — visible + // via rdd.getNumPartitions — while the row set is invariant. × parquet+orc (planning is over both). + private def forkSplitSize(fmt: String)(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_splitsize_$fmt" + spark.sql(s"DROP TABLE IF EXISTS $table") + // distribution=none + several separate INSERTs => several distinct data files. open-file-cost=1 so + // per-file planning weight is the file's byte LENGTH (not the 4MB default that would swamp small + // files) — that makes split-size the governing knob, so the task-count effect is actually visible. + spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$fmt', 'write.distribution-mode'='none', 'read.split.open-file-cost'='1')") + val nFiles = 6 + for (i <- 0 until nFiles) spark.sql(s"INSERT INTO $table SELECT ${i}L, repeat('r$i', 4000)") + val fileCount = spark.sql(s"SELECT count(*) FROM $table.data_files").collect()(0).getLong(0) + assert(fileCount >= 2, s"[$fmt] expected multiple data files for a split test, got $fileCount") + + val key = org.apache.iceberg.spark.SparkSQLProperties.SPLIT_SIZE // "spark.sql.iceberg.split-size" + val saved = spark.conf.getOption(key) + def keys(): Seq[Long] = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) + def rddParts(): Int = spark.sql(s"SELECT * FROM $table").rdd.getNumPartitions + val expected = (0 until nFiles).map(_.toLong) + try { + // (a) The prompt's core path: set spark.sql.iceberg.split-size via spark.conf.set and read the + // multi-file table under a large and a tiny split-size — the row set must be invariant. + spark.conf.set(key, (512L * 1024 * 1024).toString) + val bigRows = keys(); val bigRdd = rddParts() + spark.conf.set(key, "1") + val smallRows = keys(); val smallRdd = rddParts() + assert(bigRows == expected && smallRows == expected, + s"[$fmt] split-size must not change the row set: big=$bigRows small=$smallRows expected=$expected") + assert(smallRdd >= bigRdd, + s"[$fmt] a smaller split-size must not DECREASE the read RDD partition count: small=$smallRdd big=$bigRdd") + + // (b) DETERMINISTIC observability of the same knob at the planner: with open-file-cost=1 the per- + // file planning weight is its byte length, so a split-size below one file combines nothing + // (nFiles task groups) while a split-size above the whole table combines everything (1 group). + val ice = org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, table) + val szKey = org.apache.iceberg.TableProperties.SPLIT_SIZE // "read.split.target-size" + def planGroups(splitBytes: Long): Int = { + val it = ice.newScan().option(szKey, splitBytes.toString).planTasks().iterator() + var n = 0; while (it.hasNext) { it.next(); n += 1 } + n + } + val bigGroups = planGroups(512L * 1024 * 1024) // one combined group + val smallGroups = planGroups(1L) // one group per file + assert(bigGroups == 1, s"[$fmt] a split-size above the whole table should plan 1 task group, got $bigGroups") + assert(smallGroups == fileCount, + s"[$fmt] a split-size below one file should plan one task group per file ($fileCount), got $smallGroups") + + println(s"DIAG fork.splitSize[$fmt]: key='$key' files=$fileCount rows-correct(big+small)=yes " + + s"rddParts(big=$bigRdd,small=$smallRdd) plannedTaskGroups(bigSplit=$bigGroups,smallSplit=$smallGroups) " + + s"(split-size governs read task-group count: 1 vs $fileCount)") + } finally { + saved match { case Some(v) => spark.conf.set(key, v); case None => spark.conf.unset(key) } + spark.sql(s"DROP TABLE IF EXISTS $table") + } + } + + val forkSplitSizeOps: List[(String, Ctx => Unit)] = List( + "fork.splitSize @ parquet" -> forkSplitSize("parquet"), + "fork.splitSize @ orc" -> forkSplitSize("orc") + ) + + // ── #233 (bin-pack by data-file length) — rewrite_data_files compaction ────────────────────────────── + // The fork's bin-pack rewrite weights data files by their LENGTH (file_size_in_bytes) when packing them + // into rewrite groups. That weighting is an internal planner detail — not locally observable via SQL — so + // this is a CHARACTERIZATION: create several UNEVENLY-sized data files, run rewrite_data_files(rewrite-all), + // assert the row set is preserved, and DIAG the before/after file count + total bytes. × parquet+orc (the + // compaction decodes + re-encodes file bytes, so the format is not vacuous). + private def forkBinPackByLength(fmt: String)(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_binpack_$fmt" + spark.sql(s"DROP TABLE IF EXISTS $table") + spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$fmt', 'write.distribution-mode'='none')") + // Unevenly-sized data files: a tiny one, a small one, and a big one. + spark.sql(s"INSERT INTO $table VALUES (1,'a')") + spark.sql(s"INSERT INTO $table VALUES (2,'b'),(3,'c')") + spark.sql(s"INSERT INTO $table SELECT id, repeat('x', 200) FROM range(100, 400)") + val before = dataFileStats(spark, table) + assert(before._1 >= 3, s"[$fmt] expected >=3 uneven data files, got ${before._1}") + val totalRows = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) + + spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") + + val after = dataFileStats(spark, table) + val totalRows2 = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) + assert(totalRows2 == totalRows, s"[$fmt] rewrite_data_files changed the row count: $totalRows -> $totalRows2") + val probe = spark.sql(s"SELECT s FROM $table WHERE id = 1").collect()(0).getString(0) + assert(probe == "a", s"[$fmt] rewrite altered a row: id=1 s=$probe") + + println(s"DIAG fork.binPackByLength[$fmt]: beforeFiles=${before._1} beforeBytes=${before._2} " + + s"afterFiles=${after._1} afterBytes=${after._2} rows=$totalRows " + + s"(bin-pack weights by data-file length; characterization only — rows preserved)") + spark.sql(s"DROP TABLE IF EXISTS $table") + } + + val forkBinPackByLengthOps: List[(String, Ctx => Unit)] = List( + "fork.binPackByLength @ parquet" -> forkBinPackByLength("parquet"), + "fork.binPackByLength @ orc" -> forkBinPackByLength("orc") + ) + + // ── #189 (budgeted rewrite ordering by file-sequence-number) — rewrite_data_files ───────────────────── + // The fork's budgeted rewrite ORDERS candidate files by their file-sequence-number when spending a rewrite + // budget. The ordering decision is metadata-level and NOT locally observable via SQL, and it shares the + // rewrite_data_files execution path with #233 (fork.binPackByLength) — so rather than duplicate that, this + // pins the DISTINCT, observable half: the ordering KEY (file_sequence_number, on the .entries metadata + // table) is exposed and monotonic across commits, and rewrite-all preserves the row set. Ordering is over + // sequence numbers (not file bytes) => format-vacuous => single format (parquet). + private def forkCompactionOrder(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_compord" + spark.sql(s"DROP TABLE IF EXISTS $table") + spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$seedFmt', 'write.distribution-mode'='none')") + // Several commits => several data files with DISTINCT, increasing file-sequence-numbers (the ordering key). + val nCommits = 4 + for (i <- 0 until nCommits) spark.sql(s"INSERT INTO $table VALUES (${i}L, 'c$i')") + val seqs = spark.sql( + s"SELECT file_sequence_number FROM $table.entries WHERE status != 2 AND data_file.content = 0 " + + s"ORDER BY file_sequence_number").collect().toSeq.map(_.getLong(0)) + assert(seqs.size >= nCommits, s"expected >= $nCommits live data-file entries with sequence numbers, got ${seqs.size}: $seqs") + assert(seqs == seqs.sorted, s"file sequence numbers not monotonic: $seqs") + assert(seqs.distinct.size >= 2, s"expected multiple distinct file sequence numbers (the ordering key), got ${seqs.distinct}") + val totalRows = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) + + spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") + + val totalRows2 = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) + assert(totalRows2 == totalRows, s"rewrite changed the row count: $totalRows -> $totalRows2") + val filesAfter = spark.sql(s"SELECT count(*) FROM $table.data_files").collect()(0).getLong(0) + val keys = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) + assert(keys == (0 until nCommits).map(_.toLong), s"rewrite altered the row set: $keys") + + println(s"DIAG fork.compactionOrder: fileSeqNumbers=${seqs.mkString(",")} (ordering key for budgeted rewrite) " + + s"filesBefore=${seqs.size} filesAfter=$filesAfter rows=$totalRows " + + s"(ordering is metadata-level/not locally observable; pin: seq-numbers exposed+monotonic, rewrite preserves rows; " + + s"shares the rewrite path with fork.binPackByLength #233)") + spark.sql(s"DROP TABLE IF EXISTS $table") + } + + val forkCompactionOrderOps: List[(String, Ctx => Unit)] = List( + "fork.compactionOrder @ parquet" -> forkCompactionOrder + ) + + + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala new file mode 100644 index 000000000..caa8dab5d --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala @@ -0,0 +1,332 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// ===================================================================================== +// Delta-test harness against the real OpenHouse catalog. +// +// A test is a TYPED PIPELINE: `TableTest[S <: Schema]`. The type parameter declares which +// table implementation the test depends on, and every step references that schema's columns +// through typed handles — so the compiler forbids mixing schemas or naming a column the +// schema doesn't declare. +// +// Preparations and operations are BOTH pipeline segments of the same schema, composed with +// `andThen`: +// * a preparation prefix (create+seed, and later RTAS / drop+undrop) yields a known state, +// * an operation suffix (delete / update / merge / insert ...) runs on that state. +// The test set is `preparations x operations`. RTAS wires into every DML test by joining the +// preparations list; no operation changes. (RTAS is not built yet — only the seam is.) +// +// Catalog wiring is copied from OpenHouseLocalServer + TestSparkSessionUtil (composed, not +// extended); no OpenHouse test is altered. +// ===================================================================================== + +final case class Ctx(spark: SparkSession, namespace: String, restUri: String = "", restToken: String = "") + +// Minimal REST client to the embedded OpenHouse server (control-plane ops with no SQL surface: +// lock/unlock). Uses JDK 17's java.net.http; auth is the same Bearer token the Spark catalog uses. +object Rest { + import java.net.http.{HttpClient, HttpRequest, HttpResponse} + import java.net.URI + private lazy val client = HttpClient.newHttpClient() + private def base(ctx: Ctx, path: String): HttpRequest.Builder = + HttpRequest.newBuilder(URI.create(ctx.restUri + path)) + .header("Authorization", s"Bearer ${ctx.restToken}") + .header("Content-Type", "application/json") + def post(ctx: Ctx, path: String, body: String): (Int, String) = { + val r = client.send(base(ctx, path).POST(HttpRequest.BodyPublishers.ofString(body)).build(), HttpResponse.BodyHandlers.ofString()) + (r.statusCode(), r.body()) + } + def delete(ctx: Ctx, path: String): (Int, String) = { + val r = client.send(base(ctx, path).DELETE().build(), HttpResponse.BodyHandlers.ofString()) + (r.statusCode(), r.body()) + } + def put(ctx: Ctx, path: String, body: String): (Int, String) = { + val r = client.send(base(ctx, path).PUT(HttpRequest.BodyPublishers.ofString(body)).build(), HttpResponse.BodyHandlers.ofString()) + (r.statusCode(), r.body()) + } + def get(ctx: Ctx, path: String): (Int, String) = { + val r = client.send(base(ctx, path).GET().build(), HttpResponse.BodyHandlers.ofString()) + (r.statusCode(), r.body()) + } +} + +// Drives the soft-delete / list / restore lifecycle for the UNDROP preparation axis (Phase 4). +// The customer DROP hard-codes purge=true (a hard delete), so soft-delete is unreachable via the +// Tables API — we trigger it directly on the EMBEDDED real HTS (only available under HARNESS_REAL_HTS=1), +// then restore via the customer-facing Tables API. Endpoints are process-global (one HTS, one tables +// server for the whole run) so they are held here and set once at startup; TableTest steps see only +// (spark, table) and reach the endpoints through this holder. +object HtsAdmin { + import java.net.http.{HttpClient, HttpRequest, HttpResponse} + import java.net.URI + @volatile var htsUri: String = "" // embedded HTS base (soft-delete + querySoftDeleted) + @volatile var tablesUri: String = "" // tables server base (restore, customer-facing) + @volatile var token: String = "" // Bearer token for the tables server + def enabled: Boolean = htsUri.nonEmpty + + private lazy val client = HttpClient.newHttpClient() + private def send(b: HttpRequest.Builder): (Int, String) = { + val r = client.send(b.header("Content-Type", "application/json").build(), HttpResponse.BodyHandlers.ofString()) + (r.statusCode(), r.body()) + } + + /** Soft-delete on the embedded HTS (V1 endpoint carries the isSoftDelete flag). No auth (HTS security excluded). */ + def softDelete(db: String, tbl: String): (Int, String) = + send(HttpRequest.newBuilder(URI.create(s"$htsUri/v1/hts/tables?databaseId=$db&tableId=$tbl&isSoftDelete=true")).DELETE()) + + /** Recover the deletedAtMs of a soft-deleted table (needed to restore) from the HTS querySoftDeleted view. */ + def softDeletedAtMs(db: String, tbl: String): Option[Long] = { + val (code, body) = send(HttpRequest.newBuilder(URI.create(s"$htsUri/hts/tables/querySoftDeleted?databaseId=$db&tableId=$tbl")).GET()) + if (code < 200 || code >= 300) None + else "\"deletedAtMs\"\\s*:\\s*(\\d+)".r.findFirstMatchIn(body).map(_.group(1).toLong) + } + + /** Restore via the customer-facing Tables API (PUT .../restore?deletedAtMs=). Requires the Bearer token. */ + def restore(db: String, tbl: String, deletedAtMs: Long): (Int, String) = + send(HttpRequest.newBuilder(URI.create(s"$tablesUri/v1/databases/$db/tables/$tbl/restore?deletedAtMs=$deletedAtMs")) + .header("Authorization", s"Bearer $token") + .PUT(HttpRequest.BodyPublishers.ofString(""))) +} + +sealed trait Outcome { def label: String } +object Outcome { + case object Passed extends Outcome { val label = "PASS" } + final case class Failed(cause: Throwable) extends Outcome { + val label = "FAIL" + def retryable: Boolean = Exceptions.isTransient(cause) + def reason: String = s"${Exceptions.root(cause).getClass.getSimpleName}: ${cause.getMessage}" + } + final case class Skipped(reason: String) extends Outcome { val label = "SKIP" } +} + +object Exceptions { + def causeChain(t: Throwable): List[Throwable] = { + val chain = scala.collection.mutable.ListBuffer[Throwable]() + var current = t + while (current != null && !chain.contains(current)) { chain += current; current = current.getCause } + chain.toList + } + def root(t: Throwable): Throwable = causeChain(t).last + + /** + * Retry ONLY errors we positively recognize as transient. A bare IOException is NOT assumed + * transient — a FileNotFoundException, an EOFException on a corrupt file, or a permission error + * is an IOException too, and those are real failures that must surface rather than be retried + * away. When in doubt, an error is terminal. + */ + def isTransient(t: Throwable): Boolean = causeChain(t).exists { + case _: java.net.SocketTimeoutException => true + case _: java.net.ConnectException => true + case e: java.net.SocketException => Option(e.getMessage).exists(_.toLowerCase.contains("reset")) + case _ => false + } +} + +// Tests assert with plain `assert`; a failed assertion throws AssertionError, which is NonFatal +// and so is caught at the Runner edge and reported as a (terminal) failure. +object Check { + /** + * Require `op` to throw exactly `E` — the ACTUAL thrown type is asserted, not merely that + * *something* threw — and return it so the caller can assert on its message. NonFatal only; a + * wrong type, or no throw at all, is itself an assertion failure. + */ + def intercept[E <: Throwable: ClassTag](op: => Unit): E = { + val expected = classTag[E].runtimeClass + val caught: Option[Throwable] = try { op; None } catch { case NonFatal(t) => Some(t) } + caught match { + case Some(t) if expected.isInstance(t) => t.asInstanceOf[E] + case Some(t) => throw new AssertionError(s"expected ${expected.getName} but got ${t.getClass.getName}: ${t.getMessage}", t) + case None => throw new AssertionError(s"expected ${expected.getName} to be thrown, but nothing was") + } + } +} + +// ── Schema: columns only. A column owns its deterministic value generator; no stored seed. ── +// +// `Column[T]` carries a phantom type `T` — the Scala type the column reads back as — so typed +// row access (`row.get(CoreTable.long0): Long`) is compiler-checked. `literalAt(rowIndex)` is a +// pure function of the row index, so generated data is reproducible. Value generation lives on +// the column, which keeps RowGenerator a plain iteration with no knowledge of types. +final case class Column[T](columnName: String, sqlType: String, literalAt: Int => String) + +sealed trait Schema { + def tableColumns: Seq[Column[_]] + def columnNames: Seq[String] = tableColumns.map(_.columnName) +} + +/** Typed row access, keyed by the column's name: `row.get(CoreTable.long0)` returns a `Long`. */ +object Rows { + implicit class TypedRow(val row: Row) extends AnyVal { + def get[T](column: Column[T]): T = row.getAs[T](column.columnName) + } +} + +// A representative "core" table: one column per common data type. Column NAMES are arbitrary +// literals (decoupled from the Scala handle) — tests reference columns through the handle, so a +// rename here propagates everywhere. Plus an explicit string date-partition field in the widely +// used YYYY-MM-DD-HH form. Columns only; each carries a deterministic generator. +object CoreTable extends Schema { + val long0: Column[Long] = Column("foo_col_long", "bigint", rowIndex => rowIndex.toString) + val int0: Column[Int] = Column("foo_col_int", "int", rowIndex => rowIndex.toString) + val string0: Column[String] = Column("foo_col_string", "string", rowIndex => s"'row-$rowIndex'") + val double0: Column[Double] = Column("foo_col_double", "double", rowIndex => s"$rowIndex.5") + val boolean0: Column[Boolean] = Column("foo_col_boolean", "boolean", rowIndex => if (rowIndex % 2 == 0) "true" else "false") + val datePartition: Column[String] = Column("datepartition", "string", rowIndex => s"'${CoreTable.datePartitionLiteral(rowIndex)}'") + def tableColumns: Seq[Column[_]] = Seq(long0, int0, string0, double0, boolean0, datePartition) + + private val DatePartitionFormat = DateTimeFormatter.ofPattern("yyyy-MM-dd-HH") + private val DatePartitionEpoch = LocalDateTime.of(2024, 1, 1, 0, 0) + + /** Deterministic YYYY-MM-DD-HH partition value (one hour per row), formatted via java.time. */ + def datePartitionLiteral(rowIndex: Int): String = + DatePartitionEpoch.plusHours((rowIndex - 1).toLong).format(DatePartitionFormat) +} + +// A schema exercising complex/nested types: a struct, an array, a map, and a struct-in-struct. +// Struct/array read back as Row/Seq; map as a Map. `id` is first so it is the ordering key. +object NestedTable extends Schema { + val id: Column[Long] = Column("id", "bigint", rowIndex => rowIndex.toString) + val s: Column[Row] = Column("s", "struct", rowIndex => s"named_struct('x', $rowIndex, 'y', 'row-$rowIndex')") + val arr: Column[Seq[Int]] = Column("arr", "array", rowIndex => s"array($rowIndex, ${rowIndex + 1})") + val m: Column[Map[String, Int]] = Column("m", "map", rowIndex => s"map('k', $rowIndex)") + val nested: Column[Row] = Column("nested", "struct>", rowIndex => s"named_struct('inner', named_struct('z', $rowIndex))") + def tableColumns: Seq[Column[_]] = Seq(id, s, arr, m, nested) + + val columnDefinitions: String = + "id bigint, s struct, arr array, m map, nested struct>" +} + +// A schema for type-edge coverage: the common scalar types, exercised with nulls, special float +// values, boundary values, and unicode/empty strings. +object TypesTable extends Schema { + val id: Column[Long] = Column("id", "bigint", rowIndex => rowIndex.toString) + val n: Column[Int] = Column("n", "int", rowIndex => rowIndex.toString) + val x: Column[Double] = Column("x", "double", rowIndex => s"$rowIndex.5") + val dec: Column[java.math.BigDecimal] = Column("dec", "decimal(10,2)", rowIndex => s"CAST($rowIndex.50 AS decimal(10,2))") + val str: Column[String] = Column("str", "string", rowIndex => s"'row-$rowIndex'") + val bin: Column[Array[Byte]] = Column("bin", "binary", rowIndex => s"CAST('bin-$rowIndex' AS binary)") + val dt: Column[java.sql.Date] = Column("dt", "date", rowIndex => s"DATE '2024-01-0$rowIndex'") + val ts: Column[java.sql.Timestamp] = Column("ts", "timestamp", rowIndex => s"TIMESTAMP '2024-01-01 0$rowIndex:00:00'") + val tsntz: Column[java.time.LocalDateTime] = Column("tsntz", "timestamp_ntz", rowIndex => s"TIMESTAMP_NTZ '2024-01-01 0$rowIndex:00:00'") + def tableColumns: Seq[Column[_]] = Seq(id, n, x, dec, str, bin, dt, ts, tsntz) + + val columnDefinitions: String = + "id bigint, n int, x double, dec decimal(10,2), str string, bin binary, dt date, ts timestamp, tsntz timestamp_ntz" +} + +object RowGenerator { + /** VALUES clause for `numberOfRows` deterministic rows, one literal per column. */ + def valuesClause(schema: Schema, numberOfRows: Int): String = + (1 to numberOfRows).map { rowIndex => + schema.tableColumns.map(column => column.literalAt(rowIndex)).mkString("(", ", ", ")") + }.mkString("VALUES ", ", ", "") +} + +/** + * What a step's validation thunk sees: the live table, its rows before and after the step, and + * the table's snapshot (commit) count before and after — so a test can assert the delta in both + * data and commits (e.g. "a no-match UPDATE still commits exactly one snapshot"). + */ +final case class StepView[S <: Schema]( + spark: SparkSession, + table: String, + schema: S, + before: Seq[Row], + after: Seq[Row], + snapshotsBefore: Long, + snapshotsAfter: Long +) + +/** One pipeline step: mutate the live table, then validate it against before/after. */ +final case class Step[S <: Schema]( + label: String, + execute: (SparkSession, String, S) => Unit, + validate: StepView[S] => Unit +) + +/** + * An immutable, typed pipeline. Build a preparation prefix and an operation suffix, then + * `run` executes the steps in order on one fresh, always-dropped table, validating each step. + */ +final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Step[S]]) { + private def add(step: Step[S]): TableTest[S] = new TableTest(schema, steps :+ step) + + /** Append another same-schema pipeline (this is how prep prefixes join operation suffixes). */ + def andThen(next: TableTest[S]): TableTest[S] = new TableTest(schema, steps ++ next.steps) + + // The default validator asserts the seed actually appended `numberOfRows` rows. This defends the + // relative-delta operation assertions from a vacuous pass on an empty/short baseline. + def insert(numberOfRows: Int)( + validate: StepView[S] => Unit = view => assert( + view.after.size == view.before.size + numberOfRows, + s"seed insert($numberOfRows) expected ${view.before.size + numberOfRows} rows, got ${view.after.size}") + ): TableTest[S] = + add(Step(s"insert($numberOfRows)", (spark, table, schema) => + spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(schema, numberOfRows)}"), validate)) + + def delete(predicate: S => String)(validate: StepView[S] => Unit = _ => ()): TableTest[S] = + add(Step("delete", (spark, table, schema) => + spark.sql(s"DELETE FROM $table WHERE ${predicate(schema)}"), validate)) + + /** General operation step: run an arbitrary mutation on the table, then validate the delta. */ + def step(label: String)(mutate: (SparkSession, String) => Unit) + (validate: StepView[S] => Unit = _ => ()): TableTest[S] = + add(Step(label, (spark, table, _) => mutate(spark, table), validate)) + + /** Operation step whose mutation is a single SQL statement (the table name is supplied). */ + def sql(label: String)(statement: String => String) + (validate: StepView[S] => Unit = _ => ()): TableTest[S] = + step(label)((spark, table) => spark.sql(statement(table)))(validate) + + /** Read/assert-only step: no mutation, so before == after; used for the read paths. */ + def check(label: String)(validate: StepView[S] => Unit): TableTest[S] = + step(label)((_, _) => ())(validate) + + // Execute the pipeline on a fresh, always-dropped table. Each step's `before` is the previous + // step's `after` (an empty/zero baseline for the first step), so rows and commits are only ever + // read AFTER a step has run — on a table a prior step created. There is no existence guard: a + // query against a missing table loudly fails, which is the correct behavior. + def run(ctx: Ctx): Unit = withTable(ctx) { table => + steps.foldLeft((Seq.empty[Row], 0L)) { case ((beforeRows, beforeSnapshots), step) => + step.execute(ctx.spark, table, schema) + val afterRows = currentRows(ctx.spark, table) + val afterSnapshots = snapshotCount(ctx.spark, table) + step.validate(StepView(ctx.spark, table, schema, beforeRows, afterRows, beforeSnapshots, afterSnapshots)) + (afterRows, afterSnapshots) + } + } + + // The one table-lifecycle primitive: hand `use` a fresh table name and always drop it afterward. + // The teardown drop is guarded so a drop failure can't mask the real failure from `use`. + private def withTable(ctx: Ctx)(use: String => Unit): Unit = { + val table = s"${ctx.namespace}.t_${TableTest.counter.incrementAndGet()}" + ctx.spark.sql(s"DROP TABLE IF EXISTS $table") // ensure absent + try use(table) + finally try ctx.spark.sql(s"DROP TABLE IF EXISTS $table") catch { case NonFatal(_) => () } + } + + // Rows selected by the schema's columns, ordered by the key (first) column for deterministic + // comparison. Ordering by the key (not all columns) keeps this valid for schemas with columns + // that aren't orderable, e.g. a map. + private def currentRows(spark: SparkSession, table: String): Seq[Row] = { + val columns = schema.columnNames.mkString(", ") + spark.sql(s"SELECT $columns FROM $table ORDER BY ${schema.columnNames.head}").collect().toSeq + } + + private def snapshotCount(spark: SparkSession, table: String): Long = + spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) +} + +object TableTest { + private val counter = new java.util.concurrent.atomic.AtomicInteger(0) + def apply[S <: Schema](schema: S): TableTest[S] = new TableTest(schema, Vector.empty) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala new file mode 100644 index 000000000..ccfbd13d3 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala @@ -0,0 +1,340 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +trait HazardReaderWriterScenarios extends ScenarioKit { + import Rows._ + + val hazardStreamExpiredCheckpoint: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("hazard.stream.expiredCheckpoint") { (spark, table) => + // memory sink cannot recover from a checkpoint — stream into a second Iceberg table. + val dst = s"${table}_sink" + spark.sql(s"DROP TABLE IF EXISTS $dst") + spark.sql(coreCreateParquet(dst)) + val ckpt = java.nio.file.Files.createTempDirectory("ck-hazard").toString + def runStream(): Unit = { + val q = spark.readStream.table(table) + .writeStream.format("iceberg").outputMode("append") + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", ckpt).toTable(dst) + assert(q.awaitTermination(120000), "stream did not finish"); q.stop() + } + try { + runStream() // act 1: offset -> s1 + assert(countOf(spark, s"SELECT count(*) FROM $dst") == "3", "initial stream delivered the seed") + spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") // s2 + runStream() // act 2: CONTROL restart + assert(countOf(spark, s"SELECT count(*) FROM $dst") == "4", + "control restart must deliver exactly the incremental row (restart mechanics work)") + spark.sql(s"INSERT INTO $table VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") // s3 + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + // act 3: the checkpointed offset (s2) is expired -> restart bricked, typed. + val e = Check.intercept[Exception](runStream()) + assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(m => + m.contains("expired or removed") || m.contains("Cannot load current offset") || m.contains("Cannot find snapshot"))), + s"H1 appears FIXED — stream restarted across the expired offset; update MODALITY-RECON H1: " + + s"${e.getClass.getName} ${Option(e.getMessage).getOrElse("").take(200)}") + } finally spark.sql(s"DROP TABLE IF EXISTS $dst") + }() + + // H2 — CDC/changelog over expired lineage: expired explicit bound → hard typed error; + // timestamp bound → SILENT under-report (the truth was 5 changes; the view shows fewer). + val hazardCdcExpiredRange: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() // s1: 3 rows + .step("hazard.cdc.expiredRange") { (spark, table) => + spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") // s2 + spark.sql(s"INSERT INTO $table VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") // s3 + val snaps = snapshotIds(spark, table) + val ts0 = spark.sql(s"SELECT committed_at FROM $table.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getTimestamp(0) + val tsMid = spark.sql(s"SELECT committed_at FROM $table.snapshots WHERE snapshot_id = ${snaps(1)}").collect()(0).getTimestamp(0) + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + // Characterize each bound placement over the punctured lineage. FULL truth would mean fixed. + def changelog(optKey: String, optVal: String, truth: Long): String = try { + val v = spark.sql( + s"CALL openhouse.system.create_changelog_view(table => '${catalogRelative(table)}', " + + s"options => map('$optKey', '$optVal'))").collect()(0).getString(0) + val n = spark.sql(s"SELECT count(*) FROM $v").collect()(0).getLong(0) + if (n < truth) s"SILENT under-report: $n of $truth true changes" else s"FULL: $n of $truth" + } catch { case t: Throwable => + s"TYPED: ${t.getClass.getSimpleName} :: ${Option(t.getMessage).getOrElse("").take(140)}" } + val a = changelog("start-snapshot-id", snaps.head.toString, 5) // explicit expired bound + val b1 = changelog("start-timestamp", (ts0.getTime - 1000).toString, 5) // before all history + val b2 = changelog("start-timestamp", (tsMid.getTime - 1).toString, 2) // mid-history, expired region + println(s"DIAG cdc.explicitExpiredId: $a") + println(s"DIAG cdc.tsBeforeHistory: $b1") + println(s"DIAG cdc.tsMidExpired: $b2") + Seq("explicitId" -> a, "tsBeforeHistory" -> b1, "tsMidExpired" -> b2).foreach { case (k, o) => + assert(!o.startsWith("FULL"), + s"H2 appears FIXED for $k — changelog reported the full truth over expired lineage; update MODALITY-RECON H2: $o") + assert(!o.toLowerCase.contains("expir"), + s"H2 error now NAMES expiration for $k (readability improved) — update MODALITY-RECON H2/Audit B: $o") + } + }() + + // H3 — RTAS wipes column tags (same policies plane as G10) and column comments (new schema from SELECT). + val hazardRtasWipesColumnTags: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("enableReplace")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('replace.enabled'='true')")() + .sql("tagPii")(t => s"ALTER TABLE $t MODIFY COLUMN ${Core.string0.columnName} SET TAG = (PII)")() + .step("hazard.rtas.wipesColumnTags") { (spark, table) => + spark.sql(s"ALTER TABLE $table ALTER COLUMN ${Core.string0.columnName} COMMENT 'contains-pii'") + val before = tableProps(spark, table).getOrElse("policies", "") + assert(before.toLowerCase.contains("pii") || before.toLowerCase.contains("columntags"), + s"PII tag not stored in policies before replace: '$before'") + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + val after = tableProps(spark, table).getOrElse("policies", "") + assert(!(after.toLowerCase.contains("pii")), + s"H3 appears FIXED — PII column tag survived RTAS; update MODALITY-RECON H3 / AUDIT-FINDINGS: '$after'") + val comment = spark.sql(s"DESCRIBE TABLE $table").collect().toSeq + .find(_.getString(0) == Core.string0.columnName).map(_.getString(2)).getOrElse("") + println(s"DIAG rtas.columnComment after replace: '${comment}' (was 'contains-pii')") + }() + + // H5 — retention × branches: the DEFENDED path (positive invariant): main-side TTL delete + + // expiration + orphan removal leave a live branch fully readable. + val hazardRetentionBranchDefended: TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource PARTITIONED BY (${Core.datePartition.columnName}) TBLPROPERTIES ('write.format.default'='$seedFmt')")() + .insert(3)() + .step("hazard.retentionBranch.defended") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH rbb") + spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} <= 2") // retention-shaped main delete + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + spark.sql(s"CALL openhouse.system.remove_orphan_files(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2020-01-01 00:00:00')") + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'rbb'") == "3", + "H5 invariant: branch must remain fully readable after retention-delete + expire + orphan removal") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "1", "main reflects the TTL delete") + }() + + // H6 — rename × consumers: metadata continuity (branch refs, history, writability survive rename). + val hazardRenameConsumers: TableTest[CoreTable.type] = + coreTwoSnapshots.step("hazard.rename.consumers") { (spark, table) => + val snaps = snapshotIds(spark, table) + spark.sql(s"ALTER TABLE $table CREATE BRANCH rnb") + spark.sql(s"INSERT INTO $table.branch_rnb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val renamed = s"${table}_rn" + spark.sql(s"ALTER TABLE $table RENAME TO $renamed") + try { + assert(countOf(spark, s"SELECT count(*) FROM $renamed VERSION AS OF 'rnb'") == "6", + "branch ref must survive rename (metadata is continuous)") + assert(countOf(spark, s"SELECT count(*) FROM $renamed VERSION AS OF ${snaps.head}") == "3", + "time travel must survive rename (same snapshot log)") + spark.sql(s"INSERT INTO $renamed VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert(countOf(spark, s"SELECT count(*) FROM $renamed") == "6", "renamed table writable") + } finally spark.sql(s"ALTER TABLE $renamed RENAME TO $table") // restore for teardown + }() + + // H7 — wap.enabled=false does NOT strand named branches (only staged wap.id snapshots — G4). + val hazardWapToggleBranchesSurvive: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("hazard.wapToggle.branchesSurvive") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH wtb") + spark.sql(s"INSERT INTO $table.branch_wtb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='false')") + spark.sql(s"INSERT INTO $table.branch_wtb VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'wtb'") == "5", + "named branches must survive the WAP toggle (branch surface is not wap-gated)") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "main untouched") + }() + + // H8 — ADD COLUMN breaks every existing explicit-column writer (composition with the + // partial-INSERT rejection): schema evolution is NOT writer-backward-compatible here, + // contrary to ANSI SQL (omitted columns default to NULL). + val hazardAddColumnBreaksWriters: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("hazard.addColumn.breaksWriters") { (spark, table) => + val allCols = Core.tableColumns.map(_.columnName).mkString(", ") + val writerStatement = s"INSERT INTO $table ($allCols) VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')" + spark.sql(writerStatement) // the fleet's writer: green today + assert(countOf(spark, s"SELECT count(*) FROM $table") == "4", "writer works pre-evolution") + spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") + val e = Check.intercept[AnalysisException](spark.sql(writerStatement)) // IDENTICAL statement + assert(e.getMessage.contains("extra_col") && + (e.getMessage.contains("CANNOT_FIND_DATA") || e.getMessage.toLowerCase.contains("cannot find data")), + s"H8 appears FIXED — the pre-evolution writer survived ADD COLUMN (ANSI behavior!); update MODALITY-RECON H8 and BUGS.md: ${e.getMessage.take(200)}") + }() + + // ── Reader × writer-class battery (BUILD-STATUS task #4) ───────────────────────────────────── + // A reader (CDC changelog / incremental read / streaming) must correctly REPRESENT each writer + // class (append / overwrite / delete / update / merge), and the physical mode (CoW vs MoR) must + // not change what the reader reports. Bound each reader to the seed snapshot so only the writer's + // change is under test. Non-vacuous core; the appraisal's 120 assumed every bound-shape crossed — + // this builds the writer-class × reader core (~16), the part that actually varies by writer. + // Format is a parameter (default parquet) so reader×writer blocks can multiplex across formats. + private def cowCreate(t: String, fmt: String): String = + s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')" + private def cowCreate(t: String): String = cowCreate(t, "parquet") + private def morCreate(t: String, fmt: String): String = + s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (${morPropsFmt(fmt)})" + private def morCreate(t: String): String = morCreate(t, "parquet") + + private val writerClasses: List[(String, String => String)] = List( + "append" -> (t => s"INSERT INTO $t VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')"), + "overwrite" -> (t => s"INSERT OVERWRITE $t SELECT * FROM $t WHERE ${Core.long0.columnName} <= 2"), + "delete" -> (t => s"DELETE FROM $t WHERE ${Core.long0.columnName} = 1"), + "update" -> (t => s"UPDATE $t SET ${Core.string0.columnName} = 'upd' WHERE ${Core.long0.columnName} = 2"), + "merge" -> (t => s"MERGE INTO $t t USING (SELECT CAST(2 AS BIGINT) k UNION ALL SELECT CAST(9 AS BIGINT)) s " + + s"ON t.${Core.long0.columnName} = s.k WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + + s"WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}, ${Core.int0.columnName}, ${Core.string0.columnName}, " + + s"${Core.double0.columnName}, ${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + + s"VALUES (s.k, 9, 'row-9', 9.5, true, '2024-01-09-01')") + ) + + // CDC changelog must represent each writer class; assert the defining change-type + print the map. + private def changelogWriterTest(cls: String, mor: Boolean, fmt: String): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(t => if (mor) morCreate(t, fmt) else cowCreate(t, fmt))().insert(3)() + .step(s"readerWriter.changelog.$cls${if (mor) ".mor" else ""}") { (spark, table) => + val s0 = snapshotIds(spark, table).head + spark.sql(writerClasses.toMap.apply(cls)(table)) + // FINDING (G13): a changelog scan REJECTS a MoR table whose update/merge wrote position-delete + // files ("Delete files are currently not supported in changelog scans"). MoR delete-only and + // all CoW writers work; MoR update/merge do NOT — CDC silently unavailable for that shape. + val expectRejected = mor && (cls == "update" || cls == "merge") + def buildView(): String = spark.sql( + s"CALL openhouse.system.create_changelog_view(table => '${catalogRelative(table)}', " + + s"options => map('start-snapshot-id', '$s0'))").collect()(0).getString(0) + if (expectRejected) { + val e = Check.intercept[Exception] { val v = buildView(); spark.sql(s"SELECT * FROM $v").collect() } + assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(_.contains("Delete files are currently not supported"))), + s"G13 appears FIXED — changelog over MoR $cls no longer rejects delete files; update AUDIT-FINDINGS: ${e.getMessage.take(160)}") + println(s"DIAG changelog.$cls.mor: REJECTED (G13 - delete files unsupported in changelog scans)") + } else { + val v = buildView() + val types = spark.sql(s"SELECT _change_type, count(*) AS c FROM $v GROUP BY _change_type") + .collect().toSeq.map(r => r.getString(0) -> r.getLong(1)).toMap + println(s"DIAG changelog.$cls${if (mor) ".mor" else ""}: $types") + cls match { + case "append" => assert(types.getOrElse("INSERT", 0L) == 1 && !types.contains("DELETE"), + s"append changelog must be a single INSERT, no DELETE: $types") + case "delete" => assert(types.getOrElse("DELETE", 0L) == 1 && !types.contains("INSERT"), + s"delete changelog must be a single DELETE, no INSERT: $types") + case "update" => assert(types.getOrElse("DELETE", 0L) >= 1 && types.getOrElse("INSERT", 0L) >= 1, + s"update changelog must decompose to DELETE(old)+INSERT(new): $types") + case _ => assert(types.values.sum >= 1, s"$cls changelog must be non-empty: $types") + } + } + }() + + // Incremental read (append scan) must reflect the writer: appends add rows; a delete/overwrite + // changes the incremental row set. Bound start=seed. + private def incrementalWriterTest(cls: String, fmt: String): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(t => cowCreate(t, fmt))().insert(3)() + .step(s"readerWriter.incremental.$cls") { (spark, table) => + val s0 = snapshotIds(spark, table).head + spark.sql(writerClasses.toMap.apply(cls)(table)) + val s1 = snapshotIds(spark, table).last + val added = spark.read.format("iceberg").option("start-snapshot-id", s0).option("end-snapshot-id", s1) + .load(table).count() + println(s"DIAG incremental.$cls: added=$added") + cls match { + case "append" => assert(added == 1, s"append incremental must scan the 1 appended row: $added") + case _ => assert(added >= 0, s"$cls incremental read must not error: $added") + } + }() + + // Streaming read must represent the writer: an append is delivered; a delete/overwrite snapshot is + // rejected by the stream unless streaming-skip-* is set (characterize the two paths). + def readerWriterStreamAppend(fmt: String): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(t => cowCreate(t, fmt))().insert(3)() + .step("readerWriter.stream.append") { (spark, table) => + val dst = s"${table}_s"; spark.sql(s"DROP TABLE IF EXISTS $dst"); spark.sql(cowCreate(dst, fmt)) + val ckpt = java.nio.file.Files.createTempDirectory("ck-rw").toString + def run(): Unit = { val q = spark.readStream.table(table).writeStream.format("iceberg") + .outputMode("append").trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", ckpt).toTable(dst); assert(q.awaitTermination(120000)); q.stop() } + try { + run(); assert(countOf(spark, s"SELECT count(*) FROM $dst") == "3", "seed not streamed") + spark.sql(writerClasses.toMap.apply("append")(table)) + run(); assert(countOf(spark, s"SELECT count(*) FROM $dst") == "4", "append not streamed incrementally") + } finally spark.sql(s"DROP TABLE IF EXISTS $dst") + }() + + def readerWriterStreamDelete(fmt: String): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(t => cowCreate(t, fmt))().insert(3)() + .step("readerWriter.stream.deleteRejected") { (spark, table) => + val dst = s"${table}_sd"; spark.sql(s"DROP TABLE IF EXISTS $dst"); spark.sql(cowCreate(dst, fmt)) + val ckpt = java.nio.file.Files.createTempDirectory("ck-rwd").toString + def run(): Unit = { val q = spark.readStream.table(table).writeStream.format("iceberg") + .outputMode("append").trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", ckpt).toTable(dst); assert(q.awaitTermination(120000)); q.stop() } + try { + run() // consume the seed + spark.sql(writerClasses.toMap.apply("delete")(table)) // a delete snapshot + val e = Check.intercept[Exception](run()) + println(s"DIAG stream.afterDelete: ${e.getClass.getSimpleName} :: ${Option(e.getMessage).getOrElse("").take(140)}") + assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(m => + m.toLowerCase.contains("delete") || m.toLowerCase.contains("overwrite"))), + s"append-only stream must reject a delete snapshot (streaming-skip-* needed): ${e.getMessage.take(140)}") + } finally spark.sql(s"DROP TABLE IF EXISTS $dst") + }() + + def readerWriterOps(fmt: String): List[(String, TableTest[CoreTable.type])] = { + val changelog = for { + (cls, _) <- writerClasses + mor <- List(false, true) + } yield (s"readerWriter.changelog.$cls${if (mor) ".mor" else ""}", changelogWriterTest(cls, mor, fmt)) + val incremental = List("append", "delete", "overwrite", "update").map(c => + (s"readerWriter.incremental.$c", incrementalWriterTest(c, fmt))) + changelog ++ incremental ++ List( + "readerWriter.stream.append" -> readerWriterStreamAppend(fmt), + "readerWriter.stream.deleteRejected" -> readerWriterStreamDelete(fmt)) + } + + val hazardOps: List[(String, TableTest[CoreTable.type])] = List( + "hazard.stream.expiredCheckpoint" -> hazardStreamExpiredCheckpoint, + "hazard.cdc.expiredRange" -> hazardCdcExpiredRange, + "hazard.rtas.wipesColumnTags" -> hazardRtasWipesColumnTags, + "hazard.retentionBranch.defended" -> hazardRetentionBranchDefended, + "hazard.rename.consumers" -> hazardRenameConsumers, + "hazard.wapToggle.branchesSurvive" -> hazardWapToggleBranchesSurvive, + "hazard.addColumn.breaksWriters" -> hazardAddColumnBreaksWriters + ) + + // H4 — lock starves maintenance (needs the REST lock → Ctx-based). The same gate G2 shows the + // replace path SKIPS is hit by every maintenance commit: upkeep is blocked, replacement is not. + def hazardLockStarvesMaintenance(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_lockmaint" + val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) + spark.sql(s"DROP TABLE IF EXISTS $table") + spark.sql(coreCreateParquet(table)) + spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") + spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + try { + val (lockStatus, lockBody) = Rest.post(ctx, s"/v1/databases/$db/tables/$tbl/lock", """{"locked":true}""") + assert(lockStatus >= 200 && lockStatus < 300, s"lock POST failed: $lockStatus $lockBody") + val snapsBefore = spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) + val e = Check.intercept[Exception](spark.sql( + s"CALL openhouse.system.expire_snapshots(table => '${table.stripPrefix("openhouse.")}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)")) + assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(_.toLowerCase.contains("locked"))), + s"expected LOCKED rejection for the maintenance commit: ${e.getClass.getName} ${Option(e.getMessage).getOrElse("").take(180)}") + spark.sql(s"REFRESH TABLE $table") + val snapsAfter = spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) + assert(snapsAfter == snapsBefore, "locked table must accumulate snapshots (maintenance starved)") + val (unlockStatus, _) = Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/lock") + assert(unlockStatus >= 200 && unlockStatus < 300, "unlock failed") + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${table.stripPrefix("openhouse.")}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + spark.sql(s"REFRESH TABLE $table") + assert(spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) < snapsBefore, + "maintenance must proceed after unlock") + } finally { + Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/lock") + spark.sql(s"DROP TABLE IF EXISTS $table") + } + } + + val hazardCtxOps: List[(String, Ctx => Unit)] = List( + "hazard.lock.starvesMaintenance" -> hazardLockStarvesMaintenance + ) + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala new file mode 100644 index 000000000..3f227548c --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala @@ -0,0 +1,472 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +trait InteractionScenarios extends ScenarioKit { + import Rows._ + + + // ── DDL × history ────────────────────────────────────────────────────────────────────────── + val interactTtAfterAddColumn: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("interact.ddl.ttAfterAddColumn") { (spark, table) => + val s0 = snapshotIds(spark, table).last + spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") + spark.sql(s"INSERT INTO $table VALUES $extraColInsert9") + val current = spark.sql(s"SELECT * FROM $table LIMIT 1").columns.toSeq + val travel = spark.sql(s"SELECT * FROM $table VERSION AS OF $s0 LIMIT 1").columns.toSeq + assert(current.contains("extra_col"), s"current read missing evolved column: $current") + assert(!travel.contains("extra_col") && travel.size == Core.tableColumns.size, + s"time travel must read with the SNAPSHOT's schema (no extra_col): $travel") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF $s0").collect()(0).getLong(0) == 3, + "pre-DDL snapshot row count wrong") + }() + + val interactRestoreAfterAddColumn: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("interact.ddl.restoreAfterAddColumn") { (spark, table) => + val s0 = snapshotIds(spark, table).last + spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") + spark.sql(s"INSERT INTO $table VALUES $extraColInsert9") + spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', $s0)") + val cols = spark.sql(s"SELECT * FROM $table LIMIT 1").columns.toSeq + assert(cols.contains("extra_col"), s"rollback rolls back DATA only — schema keeps the evolved column: $cols") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "data not rolled back") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE extra_col IS NOT NULL").collect()(0).getLong(0) == 0, + "rolled-back rows must read the evolved column as null") + spark.sql(s"INSERT INTO $table VALUES $extraColInsert10") // table stays writable at the evolved arity + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "post-rollback insert failed") + }() + + // E1: data in the evolved column, then the (currently pinned-rejected) DROP — table stays intact. + // Gating pin: if DROP COLUMN support ever lands this fails → extend to full post-drop coverage. + val interactDropColAfterData: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("interact.ddl.dropColAfterData") { (spark, table) => + spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") + spark.sql(s"INSERT INTO $table VALUES $extraColInsert9") + val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table DROP COLUMN extra_col")) + assert(e.getMessage.contains("not found in newSchema"), s"drop rejection message changed: ${e.getMessage.take(200)}") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE extra_col = 42").collect()(0).getLong(0) == 1, + "rejected drop must leave the column's data readable") + spark.sql(s"INSERT INTO $table VALUES $extraColInsert10") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 5, + "rejected drop must leave the table writable") + }() + + // ── RTAS × history / lineage ─────────────────────────────────────────────────────────────── + + val interactRtasHistoryPreserved: TableTest[CoreTable.type] = + rtasPrep.step("interact.rtas.historyPreserved") { (spark, table) => + val pre = snapshotIds(spark, table).last + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + assert(spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) == 2, + "pre-RTAS snapshots must survive the replace") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF $pre").collect()(0).getLong(0) == 3, + "time travel to a pre-RTAS snapshot must work") + }() + + val interactRtasRestoreRejected: TableTest[CoreTable.type] = + rtasPrep.step("interact.rtas.restoreRejected") { (spark, table) => + val pre = snapshotIds(spark, table).last + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + val e = Check.intercept[ValidationException]( + spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', $pre)")) + assert(e.getMessage.contains("not an ancestor"), + s"rollback across RTAS: expected the new-lineage/ancestry rejection, got: ${e.getMessage.take(200)}") + }() + + // The recovery path rollback can't provide: set_current_snapshot has no ancestry requirement. + val interactRtasSetCurrentRecovery: TableTest[CoreTable.type] = + rtasPrep.step("interact.rtas.setCurrentRecovery") { (spark, table) => + val pre = snapshotIds(spark, table).last + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + spark.sql(s"CALL openhouse.system.set_current_snapshot('${catalogRelative(table)}', $pre)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, + "set_current_snapshot must recover the pre-RTAS state (no ancestry requirement)") + }() + + val interactRtasWriteAfter: TableTest[CoreTable.type] = + rtasPrep.step("interact.rtas.writeAfter") { (spark, table) => + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, + "replaced table must stay writable (DML-after-RTAS)") + }() + + // G9 (partition half): the replace path skips checkPartitionSpecEvolution — RTAS CAN change the + // spec where ALTER is pinned-rejected. Characterizes the bypass; if this ever fails, the guard + // was extended to the replace path — update AUDIT-FINDINGS G9. + val interactRtasPartitionSpecChange: TableTest[CoreTable.type] = + rtasPrep.step("interact.rtas.partitionSpecChange") { (spark, table) => + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource PARTITIONED BY (datepartition) AS SELECT * FROM $table") + val desc = spark.sql(s"DESCRIBE TABLE $table").collect().toSeq + // Confirmed live: the table gains a "# Partition Information" section (datepartition listed + // both as a column and as a partition field) — the spec changed where ALTER is pinned-rejected. + assert(desc.exists(_.getString(0) == "# Partition Information") && + desc.count(_.getString(0) == "datepartition") == 2, + s"G9 appears FIXED — RTAS no longer changes the partition spec; update AUDIT-FINDINGS G9. DESCRIBE:\n" + + desc.map(_.mkString(" | ")).mkString("\n")) + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "rows lost in re-spec RTAS") + }() + + // G9 (schema half): column drop via RTAS projection, where ALTER DROP COLUMN is pinned-rejected. + // Confirmed live (first run failed on the harness's own read-back because the column was GONE). + // Runs on a side table so the pipeline's implicit full-schema read-back stays valid. + val interactRtasDropsColumn: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("interact.rtas.dropsColumn") { (spark, table) => + val side = s"${table}_dropcol" + spark.sql(s"DROP TABLE IF EXISTS $side") + try { + spark.sql(s"CREATE TABLE $side USING $dataSource TBLPROPERTIES ('replace.enabled'='true') AS SELECT * FROM $table") + spark.sql(s"CREATE OR REPLACE TABLE $side USING $dataSource AS " + + s"SELECT ${Core.long0.columnName}, ${Core.string0.columnName} FROM $side") + val cols = spark.sql(s"SELECT * FROM $side LIMIT 1").columns.toSeq + assert(cols == Seq(Core.long0.columnName, Core.string0.columnName), + s"G9 appears FIXED — RTAS no longer drops columns (ALTER DROP stays rejected); update AUDIT-FINDINGS G9: $cols") + assert(spark.sql(s"SELECT count(*) FROM $side").collect()(0).getLong(0) == 3, "rows lost in column-drop RTAS") + } finally spark.sql(s"DROP TABLE IF EXISTS $side") + }() + + // ── RTAS × table-property merge semantics (the THIRD property path beside CREATE and ALTER) ── + val interactRtasPropsUserSurvival: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$seedFmt', 'replace.enabled'='true', 'user.key'='v1')")() + .insert(3)() + .step("interact.rtas.props.userSurvival") { (spark, table) => + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + val p = tableProps(spark, table) + assert(p.get("user.key").contains("v1"), s"user prop lost across RTAS: user.key=${p.get("user.key")}") + assert(p.get("replace.enabled").contains("true"), s"replace.enabled lost across RTAS: ${p.get("replace.enabled")}") + }() + + val interactRtasPropsStatementWins: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$seedFmt', 'replace.enabled'='true', 'user.key'='v1')")() + .insert(3)() + .step("interact.rtas.props.statementWins") { (spark, table) => + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource TBLPROPERTIES ('user.key'='v2') " + + s"AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + val p = tableProps(spark, table) + assert(p.get("user.key").contains("v2"), s"statement TBLPROPERTIES must win over the old value: ${p.get("user.key")}") + assert(p.get("replace.enabled").contains("true"), + s"props NOT named in the statement must still survive (merge, not wholesale replace): ${p.get("replace.enabled")}") + }() + + val interactRtasPropsCreateDefaulting: TableTest[CoreTable.type] = + rtasPrep.step("interact.rtas.props.createDefaulting") { (spark, table) => + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource TBLPROPERTIES ('write.format.default'='orc') " + + s"AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + val p = tableProps(spark, table) + assert(p.get("write.format.default").contains("orc"), + s"RTAS can change the storage format where ALTER can't rewrite: ${p.get("write.format.default")}") + assert(p.get("format-version").forall(_ == "2"), s"forced format-version drifted: ${p.get("format-version")}") + spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "orc-format table not writable") + }() + + val interactRtasPropsReservedPlane: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource PARTITIONED BY (datepartition) TBLPROPERTIES (" + + s"'write.format.default'='$seedFmt', 'replace.enabled'='true')")() + .insert(3)() + .sql("setRetention")(t => s"ALTER TABLE $t SET POLICY (RETENTION = 30d ON COLUMN datepartition WHERE pattern = 'yyyy-MM-dd-HH')")() + .step("interact.rtas.props.reservedPlane") { (spark, table) => + val uuidBefore = tableProps(spark, table).getOrElse("openhouse.tableUUID", "") + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource PARTITIONED BY (datepartition) " + + s"AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + val p = tableProps(spark, table) + assert(p.getOrElse("openhouse.tableUUID", "") == uuidBefore, + s"tableUUID must be preserved across RTAS: $uuidBefore -> ${p.get("openhouse.tableUUID")}") + // G10 (confirmed live): RTAS silently WIPES the policies plane — the retention policy set + // before the replace is gone after it (while tableUUID survives). Characterizes the bug; + // if this fails, G10 was fixed — flip to a survival assertion and update AUDIT-FINDINGS. + val policiesAfter = p.get("policies") + assert(policiesAfter.forall(b => !b.toLowerCase.contains("retention")), + s"G10 appears FIXED — retention policy survived RTAS; update AUDIT-FINDINGS G10 and flip this test: $policiesAfter") + }() + + // RTAS on a table with an existing branch: refs travel in the replace payload — branch survives, + // still readable at its (old-lineage) head. + val interactRtasWithBranch: TableTest[CoreTable.type] = + rtasPrep.step("interact.rtas.withBranch") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH keepbr") + spark.sql(s"INSERT INTO $table.branch_keepbr VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + val refs = spark.sql(s"SELECT name FROM $table.refs").collect().toSeq.map(_.getString(0)).toSet + assert(refs.contains("keepbr"), s"branch ref lost across RTAS: $refs") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'keepbr'").collect()(0).getLong(0) == 4, + "branch head (old lineage) unreadable after RTAS") + }() + + // ── branch × history / maintenance ───────────────────────────────────────────────────────── + val interactBranchTtBeforeBranchPoint: TableTest[CoreTable.type] = + coreTwoSnapshots.step("interact.branch.ttBeforeBranchPoint") { (spark, table) => + val snaps = snapshotIds(spark, table) + val ts0 = spark.sql(s"SELECT committed_at FROM $table.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getTimestamp(0) + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") + spark.sql(s"ALTER TABLE $table CREATE BRANCH tb") + spark.sql(s"INSERT INTO $table.branch_tb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'tb'").collect()(0).getLong(0) == 6, "branch head") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF ${snaps.head}").collect()(0).getLong(0) == 3, + "snapshot-id travel to a pre-branch-point ancestor must work") + spark.conf.set("spark.wap.branch", "tb") + try { + assert(spark.sql(s"SELECT count(*) FROM $table TIMESTAMP AS OF '$ts0'").collect()(0).getLong(0) == 3, + "explicit TIMESTAMP AS OF must override spark.wap.branch and resolve against main history") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF ${snaps.head}").collect()(0).getLong(0) == 3, + "explicit VERSION AS OF must override spark.wap.branch") + } finally spark.conf.unset("spark.wap.branch") + }() + + // E5 characterization (mirror of G8): DDL on MAIN hits branches immediately — schema is + // table-global, and an old-arity branch writer is broken mid-flight. + val interactBranchMainDdlImmediate: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("interact.branch.mainDdlImmediate") { (spark, table) => + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") + spark.sql(s"ALTER TABLE $table CREATE BRANCH mb") + spark.sql(s"INSERT INTO $table.branch_mb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") // DDL on MAIN + val branchCols = spark.sql(s"SELECT * FROM $table VERSION AS OF 'mb' LIMIT 1").columns.toSeq + assert(branchCols.contains("extra_col"), s"main DDL is table-global — branch reads see it immediately: $branchCols") + val e = Check.intercept[AnalysisException]( + spark.sql(s"INSERT INTO $table.branch_mb VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')")) + assert(e.getMessage.toLowerCase.contains("not enough data columns"), + s"old-arity branch writer must break after main DDL (characterizes the hazard): ${e.getMessage.take(200)}") + spark.sql(s"INSERT INTO $table.branch_mb VALUES (CAST(8 AS BIGINT), 8, 'row-8', 8.5, true, '2024-01-08-07', 44)") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'mb'").collect()(0).getLong(0) == 5, + "new-arity branch write after main DDL") + }() + + // E10: expiration is ref-aware — branch heads survive, shared ancestry prunes. + val interactBranchExpireProtectsRefs: TableTest[CoreTable.type] = + coreTwoSnapshots.step("interact.branch.expireProtectsRefs") { (spark, table) => + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") + spark.sql(s"ALTER TABLE $table CREATE BRANCH eb") + spark.sql(s"INSERT INTO $table.branch_eb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + spark.sql(s"INSERT INTO $table VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert(spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) == 4, "expected 4 snapshots pre-expire") + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + val refs = spark.sql(s"SELECT name FROM $table.refs").collect().toSeq.map(_.getString(0)).toSet + assert(refs == Set("main", "eb"), s"branch/tag refs must survive expiration: $refs") + assert(spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) == 2, + "shared ancestry prunes to the two ref heads") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'eb'").collect()(0).getLong(0) == 6, "branch readable post-expire") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 6, "main readable post-expire") + }() + + // C4: restore procedures target MAIN even while spark.wap.branch is set (procedures are not + // branch-conf-routed) — the branch is untouched. + val interactBranchRollbackWhileWapConf: TableTest[CoreTable.type] = + coreTwoSnapshots.step("interact.branch.rollbackWhileWapConf") { (spark, table) => + val s0 = snapshotIds(spark, table).head + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") + spark.sql(s"ALTER TABLE $table CREATE BRANCH rb") + spark.sql(s"INSERT INTO $table.branch_rb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + spark.conf.set("spark.wap.branch", "rb") + try spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', $s0)") + finally spark.conf.unset("spark.wap.branch") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, + "rollback under wap.branch conf still targets MAIN (procedures are not branch-routed)") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'rb'").collect()(0).getLong(0) == 6, + "branch untouched by the main rollback") + }() + + // C1: rolled-past snapshots are unreferenced — expiration makes the rollback permanent. + val interactRestoreExpireAfterRollback: TableTest[CoreTable.type] = + coreTwoSnapshots.step("interact.restore.expireAfterRollback") { (spark, table) => + val snaps = snapshotIds(spark, table) + spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', ${snaps.head})") + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + assert(spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) == 1, + "the rolled-past snapshot must be expired (unreferenced)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "current state intact") + val e = Check.intercept[Exception]( + spark.sql(s"SELECT count(*) FROM $table VERSION AS OF ${snaps(1)}").collect()) + assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(_.toLowerCase.contains("snapshot"))), + s"travel to the expired snapshot must fail (rollback is now PERMANENT): ${e.getMessage.take(200)}") + }() + + // ── THE COMPOSITE DEFECT: branch × expiration × merge (G11; INTERACTION-AUDIT §6) ─────────── + // Bytecode-confirmed mechanism: RemoveSnapshots retention is per-ref and head-anchored (no + // protection for the ancestry BETWEEN live refs), and SnapshotUtil's ancestry walk SILENTLY + // TRUNCATES at an expired hole and returns false. So policy-driven expiration between branch + // work and the merge makes fast_forward spuriously reject with "not an ancestor" — even when + // main never advanced — and, with no rebase in Iceberg, the branch is permanently stranded. + // The pair test (branch × expire) PASSES because reads don't consume ancestry; only the merge does. + val interactExpireMergeSpuriousReject: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("interact.branch.expireMerge.spuriousReject") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH mb") + spark.sql(s"INSERT INTO $table.branch_mb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") // B1 + spark.sql(s"INSERT INTO $table.branch_mb VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") // B2 (head) + assert(countOf(spark, s"SELECT count(*) FROM $table.snapshots") == "3", "expected P, B1, B2") + // main NEVER advances. This merge is valid right now (branch.fastForward.merge is the + // no-expiration control proving it). Interpose the destroyer: + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + // P2 VIOLATED: retention is per-ref head-anchored — the intermediate branch commit B1 + // (merge connectivity) is expired even though both refs are alive. + assert(countOf(spark, s"SELECT count(*) FROM $table.snapshots") == "2", + "retention keeps only the two ref heads; the intermediate branch snapshot is expired") + // The pair-test ILLUSION: refs alive, branch fully readable — nothing looks broken. + val refs = spark.sql(s"SELECT name FROM $table.refs").collect().toSeq.map(_.getString(0)).toSet + assert(refs == Set("main", "mb"), s"both refs alive: $refs") + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'mb'") == "5", "branch readable") + // P1 VIOLATED: the merge is now spuriously rejected — the ancestry walk from B2 hits the + // B1 hole, silently truncates, and concludes main's head "is not an ancestor" of the branch. + val e = Check.intercept[Exception]( + spark.sql(s"CALL openhouse.system.fast_forward('${catalogRelative(table)}', 'main', 'mb')")) + assert(Option(e.getMessage).exists(_.contains("not an ancestor")), + s"G11 appears FIXED — fast_forward survived expiration (or failed differently); update AUDIT-FINDINGS G11: " + + s"${e.getClass.getName} ${Option(e.getMessage).getOrElse("").take(180)}") + // P6 VIOLATED: no recovery path merges the branch. Characterize the cherry-pick fallback: + val b2 = spark.sql(s"SELECT snapshot_id FROM $table.refs WHERE name = 'mb'").collect()(0).getLong(0) + val cherry = try { + spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${b2}L)") + s"SUCCEEDED — main now ${countOf(spark, s"SELECT count(*) FROM $table")} rows (B1's commit silently LOST in the 'merge')" + } catch { case t: Throwable => s"REJECTED ${t.getClass.getName} :: ${Option(t.getMessage).getOrElse("").take(160)}" } + println(s"DIAG expireMerge.cherrypickFallback: $cherry") + val mainCount = countOf(spark, s"SELECT count(*) FROM $table").toLong + assert(mainCount == 3 || mainCount == 4, s"main must stay consistent (3, or 4 if cherry-pick half-merged): $mainCount") + // Copy-out is the ONLY full recovery (data files survive: expiration ran cleanExpiredFiles(false)). + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'mb'") == "5", + "branch data must remain readable for copy-out recovery") + }() + + // P3 VIOLATED: WAP-staged snapshots are UNREFERENCED, so age-based expiration silently deletes + // them before publish; the loss only becomes loud at publish time ("Cannot find snapshot"). + // OpenHouse's scheduled expiration job (default 3-day TTL) makes this automatic, not hypothetical. + val interactExpireMergeStagedWapLoss: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("interact.branch.expireMerge.stagedWapLoss") { (spark, table) => + spark.conf.set("spark.wap.id", "w2") + try spark.sql(s"INSERT INTO $table VALUES (CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + finally spark.conf.unset("spark.wap.id") + assert(countOf(spark, s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'w2'") == "1", "staged") + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + // The SILENT loss: expiration reports nothing about the staged work it destroyed. + assert(countOf(spark, s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'w2'") == "0", + "P3 appears FIXED — staged WAP snapshot survived expiration; update AUDIT-FINDINGS G11") + // Loud only NOW, at publish — after the work is unrecoverable: + val e = Check.intercept[Exception]( + spark.sql(s"CALL openhouse.system.publish_changes(table => '${catalogRelative(table)}', wap_id => 'w2')")) + println(s"DIAG stagedWapLoss.publish: ${e.getClass.getName} :: ${Option(e.getMessage).getOrElse("").take(180)}") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "main unchanged; the staged write is gone") + }() + + // ── flags at CREATE + ALTER-to-MoR + compaction over evolved schema ──────────────────────── + val interactFlagsWapReplaceAtCreate: TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$seedFmt', 'write.wap.enabled'='true', 'replace.enabled'='true')")() + .insert(3)() + .step("interact.flags.wapReplaceAtCreate") { (spark, table) => + val p = tableProps(spark, table) + assert(p.get("write.wap.enabled").contains("true") && p.get("replace.enabled").contains("true"), + s"flags set at CREATE must be honored: wap=${p.get("write.wap.enabled")} replace=${p.get("replace.enabled")}") + spark.sql(s"ALTER TABLE $table CREATE BRANCH cb") // wap-at-create usable immediately + val e = Check.intercept[BadRequestException]( + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table")) + assert(e.getMessage.contains("while WAP"), + s"RTAS-while-WAP guard must fire from create-time flags too: ${e.getMessage.take(200)}") + }() + + val interactMorAlterToMor: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)() + .sql("seed(3, one-file)")(t => + s"INSERT INTO $t SELECT /*+ COALESCE(1) */ * FROM (${RowGenerator.valuesClause(Core, 3)}) AS seed")() + .step("interact.mor.alterToMor") { (spark, table) => + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')") + spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") + val deleteFiles = spark.sql(s"SELECT count(*) FROM $table.all_delete_files").collect()(0).getLong(0) + assert(deleteFiles == 1, + s"ALTER-to-MoR must govern subsequent deletes (expected 1 position-delete file, got $deleteFiles)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "row not deleted") + }() + + val interactMaintCompactEvolved: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("interact.maint.compactEvolved") { (spark, table) => + spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") + spark.sql(s"INSERT INTO $table VALUES $extraColInsert9") + spark.sql(s"INSERT INTO $table VALUES $extraColInsert10") + spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}')") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 5, "compaction changed row count") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE extra_col IN (42, 43)").collect()(0).getLong(0) == 2, + "compaction over mixed-schema files must preserve evolved-column values") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE extra_col IS NULL").collect()(0).getLong(0) == 3, + "pre-evolution rows must stay null in the evolved column") + }() + + val interactions: List[(String, TableTest[CoreTable.type])] = List( + "interact.ddl.ttAfterAddColumn" -> interactTtAfterAddColumn, + "interact.ddl.restoreAfterAddColumn" -> interactRestoreAfterAddColumn, + "interact.ddl.dropColAfterData" -> interactDropColAfterData, + "interact.rtas.historyPreserved" -> interactRtasHistoryPreserved, + "interact.rtas.restoreRejected" -> interactRtasRestoreRejected, + "interact.rtas.setCurrentRecovery" -> interactRtasSetCurrentRecovery, + "interact.rtas.writeAfter" -> interactRtasWriteAfter, + "interact.rtas.partitionSpecChange" -> interactRtasPartitionSpecChange, + "interact.rtas.dropsColumn" -> interactRtasDropsColumn, + "interact.rtas.props.userSurvival" -> interactRtasPropsUserSurvival, + "interact.rtas.props.statementWins" -> interactRtasPropsStatementWins, + "interact.rtas.props.createDefaulting" -> interactRtasPropsCreateDefaulting, + "interact.rtas.props.reservedPlane" -> interactRtasPropsReservedPlane, + "interact.rtas.withBranch" -> interactRtasWithBranch, + "interact.branch.ttBeforeBranchPoint" -> interactBranchTtBeforeBranchPoint, + "interact.branch.mainDdlImmediate" -> interactBranchMainDdlImmediate, + "interact.branch.expireProtectsRefs" -> interactBranchExpireProtectsRefs, + "interact.branch.rollbackWhileWapConf" -> interactBranchRollbackWhileWapConf, + "interact.restore.expireAfterRollback" -> interactRestoreExpireAfterRollback, + "interact.branch.expireMerge.spuriousReject" -> interactExpireMergeSpuriousReject, + "interact.branch.expireMerge.stagedWapLoss" -> interactExpireMergeStagedWapLoss, + "interact.flags.wapReplaceAtCreate" -> interactFlagsWapReplaceAtCreate, + "interact.mor.alterToMor" -> interactMorAlterToMor, + "interact.maint.compactEvolved" -> interactMaintCompactEvolved + ) + + // G2 characterization needs the REST lock (no SQL surface) → Ctx-based like controlPlane. + // Sanity-checks the lock DOES block a normal write, then demonstrates RTAS sails through it. + def interactRtasOnLockedTable(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_lockrtas" + val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) + spark.sql(s"DROP TABLE IF EXISTS $table") + spark.sql(coreCreateParquet(table)) + spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')") + try { + val (lockStatus, lockBody) = Rest.post(ctx, s"/v1/databases/$db/tables/$tbl/lock", """{"locked":true}""") + assert(lockStatus >= 200 && lockStatus < 300, s"lock POST failed: $lockStatus $lockBody") + val blocked = Check.intercept[Exception](spark.sql( + s"UPDATE $table SET ${Core.string0.columnName} = 'x' WHERE ${Core.long0.columnName} = 1")) + assert(Exceptions.causeChain(blocked).exists(t => Option(t.getMessage).exists(_.toLowerCase.contains("locked"))), + s"lock not enforced on UPDATE: ${blocked.getMessage.take(160)}") + // G2: the replace branches never reach the isTableLocked check — RTAS replaces a LOCKED table. + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, + "G2 characterization: RTAS bypassed the lock (if a locked-table rejection landed here, G2 is FIXED — update AUDIT-FINDINGS)") + } finally { + Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/lock") + spark.sql(s"DROP TABLE IF EXISTS $table") + } + } + + val interactionCtxOps: List[(String, Ctx => Unit)] = List( + "interact.rtas.onLockedTable" -> interactRtasOnLockedTable + ) + + // ═══ Surface-completion axis: queued follow-ups + untested Iceberg surface ═══════════════════ + + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala new file mode 100644 index 000000000..026650c38 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala @@ -0,0 +1,214 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +trait MaintControlScenarios extends ScenarioKit { + import Rows._ + + // ── time travel + restore/rollback ────────────────────────────────────────────────────── + // A two-snapshot base: seed 3 rows (snapshot A), then insert 2 more (snapshot B). + // Format is a PARAMETER, not baked in — so any block built on this base can multiplex across formats. + + def timeTravelVersionAsOf(fmt: String): TableTest[CoreTable.type] = + coreTwoSnapshots(fmt).check("timeTravel.versionAsOf") { view => + val snaps = snapshotIds(view.spark, view.table) + assert(view.spark.sql(s"SELECT count(*) FROM ${view.table} VERSION AS OF ${snaps(0)}").collect()(0).getLong(0) == 3) + assert(view.spark.sql(s"SELECT count(*) FROM ${view.table} VERSION AS OF ${snaps(1)}").collect()(0).getLong(0) == 5) + } + + def timeTravelTimestampAsOf(fmt: String): TableTest[CoreTable.type] = + coreTwoSnapshots(fmt).check("timeTravel.timestampAsOf") { view => + val ts0 = view.spark.sql(s"SELECT committed_at FROM ${view.table}.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getTimestamp(0) + assert(view.spark.sql(s"SELECT count(*) FROM ${view.table} TIMESTAMP AS OF '$ts0'").collect()(0).getLong(0) == 3) + } + + def timeTravelMetadataTables(fmt: String): TableTest[CoreTable.type] = + coreTwoSnapshots(fmt).check("timeTravel.metadataTables") { view => + def count(meta: String): Long = view.spark.sql(s"SELECT count(*) FROM ${view.table}.$meta").collect()(0).getLong(0) + assert(count("snapshots") == 2) + assert(count("history") == 2) + assert(count("files") >= 1 && count("manifests") >= 1) + } + + def timeTravelIncrementalRead(fmt: String): TableTest[CoreTable.type] = + coreTwoSnapshots(fmt).check("timeTravel.incrementalRead") { view => + val snaps = snapshotIds(view.spark, view.table) + val added = view.spark.read.format("iceberg") + .option("start-snapshot-id", snaps(0)).option("end-snapshot-id", snaps(1)) + .load(view.table).count() + assert(added == 2) // only the rows added between snapshot A and B + } + + def timeTravelOps(fmt: String): List[(String, TableTest[CoreTable.type])] = List( + "timeTravel.versionAsOf" -> timeTravelVersionAsOf(fmt), + "timeTravel.timestampAsOf" -> timeTravelTimestampAsOf(fmt), + "timeTravel.metadataTables" -> timeTravelMetadataTables(fmt), + "timeTravel.incrementalRead" -> timeTravelIncrementalRead(fmt) + ) + + // Restore/rollback via stored procedures (gated: OpenHouse may not expose CALL procedures). + + def restoreRollbackToSnapshot(fmt: String): TableTest[CoreTable.type] = + coreTwoSnapshots(fmt).step("restore.rollbackToSnapshot") { (spark, table) => + val first = snapshotIds(spark, table).head + spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', $first)") + } { view => + assert(view.after.size == 3) // rolled back to the 3-row snapshot + } + + def restoreSetCurrentSnapshot(fmt: String): TableTest[CoreTable.type] = + coreTwoSnapshots(fmt).step("restore.setCurrentSnapshot") { (spark, table) => + val first = snapshotIds(spark, table).head + spark.sql(s"CALL openhouse.system.set_current_snapshot('${catalogRelative(table)}', $first)") + } { view => + assert(view.after.size == 3) + } + + def restoreRollbackOps(fmt: String): List[(String, TableTest[CoreTable.type])] = List( + "restore.rollbackToSnapshot" -> restoreRollbackToSnapshot(fmt), + "restore.setCurrentSnapshot" -> restoreSetCurrentSnapshot(fmt) + ) + + // ── Maintenance OPERATIONS (Iceberg CALL procedures; jobs merely orchestrate these) ────────── + // SE / OFD / compaction are stored procedures, reachable from Spark SQL like rollback/set_current. + // Each mutates physical state; we assert the current DATA is preserved and observe the metadata delta. + def maintenanceExpireSnapshots(fmt: String): TableTest[CoreTable.type] = + coreTwoSnapshots(fmt).step("maintenance.expireSnapshots") { (spark, table) => + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + } { view => + assert(view.after.size == 5, "expire_snapshots changed the current data") + assert(view.snapshotsAfter < view.snapshotsBefore, s"expire did not drop a snapshot: ${view.snapshotsBefore} -> ${view.snapshotsAfter}") + } + + def maintenanceRewriteDataFiles(fmt: String): TableTest[CoreTable.type] = + coreTwoSnapshots(fmt).step("maintenance.rewriteDataFiles") { (spark, table) => + spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}')") + } { view => + assert(view.after.size == 5, "compaction changed rows") // rows preserved + } + + def maintenanceRemoveOrphanFiles(fmt: String): TableTest[CoreTable.type] = + coreTwoSnapshots(fmt).step("maintenance.removeOrphanFiles") { (spark, table) => + // older_than must be ≥24h in the past (a safety guard); a far-past ts is a valid no-op that + // still exercises the procedure end-to-end without corrupting live files. + spark.sql(s"CALL openhouse.system.remove_orphan_files(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2020-01-01 00:00:00')") + } { view => + assert(view.after.size == 5, "orphan removal changed rows") + } + + def maintenanceOps(fmt: String): List[(String, TableTest[CoreTable.type])] = List( + "maintenance.expireSnapshots" -> maintenanceExpireSnapshots(fmt), + "maintenance.rewriteDataFiles" -> maintenanceRewriteDataFiles(fmt), + "maintenance.removeOrphanFiles" -> maintenanceRemoveOrphanFiles(fmt) + ) + + // ── Control-plane (REST) ops with no SQL surface — driven via the embedded server's HTTP API ── + // Lock enforcement: POST /lock (a real public entry), then a Spark mutation is rejected server-side + // (LOCKED_TABLE_OPERATION); DELETE /lock restores mutability. High-fidelity — the embedded server + // runs the real TablesController/TablesServiceImpl (see REST-FIDELITY-EVAL.md). + def controlLockEnforcement(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_lock" + val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) + spark.sql(s"DROP TABLE IF EXISTS $table") + spark.sql(coreCreateParquet(table)) + spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") + try { + val (lockStatus, lockBody) = Rest.post(ctx, s"/v1/databases/$db/tables/$tbl/lock", """{"locked":true}""") + assert(lockStatus >= 200 && lockStatus < 300, s"lock POST failed: $lockStatus $lockBody") + val e = Check.intercept[Exception](spark.sql( + s"UPDATE $table SET ${Core.string0.columnName} = 'locked-write' WHERE ${Core.long0.columnName} = 1")) + assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(_.toLowerCase.contains("locked"))), + s"expected a locked-table rejection, got: ${e.getMessage.take(200)}") + val (unlockStatus, unlockBody) = Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/lock") + assert(unlockStatus >= 200 && unlockStatus < 300, s"unlock DELETE failed: $unlockStatus $unlockBody") + spark.sql(s"UPDATE $table SET ${Core.string0.columnName} = 'unlocked-write' WHERE ${Core.long0.columnName} = 1") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.string0.columnName} = 'unlocked-write'").collect()(0).getLong(0) == 1, + "post-unlock update did not apply") + } finally spark.sql(s"DROP TABLE IF EXISTS $table") + } + + // Undrop lifecycle — TAGGED SKIP (Plan.knownBugs). Not runnable at fidelity in the embedded harness: + // (1) the embedded HouseTableRepository is a @Primary in-memory STUB (HouseTablesH2Repository) — a + // test here would exercise the shim's own reimplementation, not the real HTS soft-delete logic; + // (2) the public Tables DELETE hard-codes purge=true, so drop→soft-delete is unreachable via the + // customer API in ANY environment (undrop is HTS-admin-only — a product finding). + // Real fidelity needs an embedded HTS (SpringH2HtsApplication) + de-@Primary-ing the stub. The body + // documents the intended list→restore flow for that future harness. + def controlUndropLifecycle(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_undrop" + val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) + spark.sql(s"DROP TABLE IF EXISTS $table") + spark.sql(coreCreateParquet(table)) + spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") + // (intended, once a real HTS soft-deletes the table:) + val (listStatus, listBody) = Rest.get(ctx, s"/v1/databases/$db/softDeletedTables") + assert(listStatus == 200 && listBody.contains(tbl), "soft-deleted table should be listed") + val (restoreStatus, _) = Rest.put(ctx, s"/v1/databases/$db/tables/$tbl/restore?deletedAtMs=0", "") + assert(restoreStatus >= 200 && restoreStatus < 300, "restore should succeed") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "restored table keeps its rows") + spark.sql(s"DROP TABLE IF EXISTS $table") + } + + val controlPlane: List[(String, Ctx => Unit)] = List( + "control.lock.enforcement" -> controlLockEnforcement, + "control.undrop.lifecycle" -> controlUndropLifecycle + ) + + // ── Undrop admin-lifecycle block (Phase 5 — REAL HTS only, HtsAdmin.enabled) ───────────────── + // With an embedded real HTS the full soft-delete → list → restore / purge lifecycle is exercisable + // (the customer DROP still hard-deletes — soft-delete is driven directly on HTS). These are the + // HTS-admin lifecycle cases that sit ALONGSIDE the surface-doubling undrop battery. + + // Soft-delete → the customer softDeletedTables listing shows it → restore → rows intact. + def undropAdminRestoreRoundTrip(ctx: Ctx): Unit = { + val (table, db, tbl) = undropSeed(ctx, "t_undrop_rt") + val (sd, sdb) = HtsAdmin.softDelete(db, tbl); assert(sd >= 200 && sd < 300, s"soft-delete failed ($sd): $sdb") + val (ls, lb) = Rest.get(ctx, s"/v1/databases/$db/softDeletedTables") + assert(ls == 200 && lb.contains(tbl), s"soft-deleted table not listed via Tables API ($ls): $lb") + val ms = HtsAdmin.softDeletedAtMs(db, tbl).getOrElse(throw new AssertionError(s"no deletedAtMs for $db.$tbl")) + val (rs, rb) = HtsAdmin.restore(db, tbl, ms); assert(rs >= 200 && rs < 300, s"restore failed ($rs): $rb") + assert(ctx.spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "restored table lost rows") + ctx.spark.sql(s"DROP TABLE IF EXISTS $table") + } + + // Two soft-deleted tables both appear in the listing (paging/enumeration works). + def undropAdminListSoftDeleted(ctx: Ctx): Unit = { + val (_, db, t1) = undropSeed(ctx, "t_undrop_l1") + val (_, _, t2) = undropSeed(ctx, "t_undrop_l2") + assert(HtsAdmin.softDelete(db, t1)._1 / 100 == 2, "soft-delete t1 failed") + assert(HtsAdmin.softDelete(db, t2)._1 / 100 == 2, "soft-delete t2 failed") + val (ls, lb) = Rest.get(ctx, s"/v1/databases/$db/softDeletedTables") + assert(ls == 200 && lb.contains(t1) && lb.contains(t2), s"both soft-deleted tables should list ($ls): $lb") + } + + // Restore AFTER purge must be rejected — purge is permanent. Pin whatever the real HTS returns + // (a 4xx; the point is that restore no longer succeeds once the row is purged). + def undropAdminRestoreAfterPurgeRejected(ctx: Ctx): Unit = { + val (_, db, tbl) = undropSeed(ctx, "t_undrop_purge") + assert(HtsAdmin.softDelete(db, tbl)._1 / 100 == 2, "soft-delete failed") + val ms = HtsAdmin.softDeletedAtMs(db, tbl).getOrElse(throw new AssertionError("no deletedAtMs")) + // purge everything deleted before a far-future instant → removes this row permanently + val (ps, _) = Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/purge?purgeAfterMs=${Long.MaxValue}") + assert(ps / 100 == 2, s"purge should succeed ($ps)") + val (rs, _) = HtsAdmin.restore(db, tbl, ms) + assert(rs >= 400, s"restore after purge must be rejected, got $rs") + } + + val undropAdminOps: List[(String, Ctx => Unit)] = List( + "undropAdmin.restoreRoundTrip" -> undropAdminRestoreRoundTrip, + "undropAdmin.listSoftDeleted" -> undropAdminListSoftDeleted, + "undropAdmin.restoreAfterPurgeRejected" -> undropAdminRestoreAfterPurgeRejected + ) + + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala new file mode 100644 index 000000000..f006cf1c0 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala @@ -0,0 +1,241 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +trait MorMaintScenarios extends ScenarioKit { + import Rows._ + + // ── MoR delete-file coexistence battery (BUILD-STATUS task #5, the NON-vacuous core) ───────── + // The appraisal's "core DML → L×M=12" is ~90% vacuous: a read/insert on a DELETE-FREE MoR table + // is byte-identical to CoW (no delete files to apply; append is mode-independent). The mutation + // ops ARE crossed with MoR already (the `mor` bucket, 264). The genuinely-new MoR surface is + // operating on a table that ALREADY carries a live position-delete file — data-file/delete-file + // COEXISTENCE. `createAndSeedMorDeleted` leaves 2 rows (keys 2,3) with a live delete for key 1; + // these ops then act on that state. + val morCoexistOps: List[(String, TableTest[CoreTable.type])] = List( + // A new data file must coexist with the existing delete file; the read applies the delete to + // OLD data only, not the appended rows. + "coexist.append" -> TableTest(Core).step("coexist.append") { (spark, table) => + spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "append over live delete file wrong count") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 0, "deleted row resurrected by append") + }(), + // A second delete adds a second position-delete file over the same data file. + "coexist.secondDelete" -> TableTest(Core).step("coexist.secondDelete") { (spark, table) => + spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 2") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 1, "second delete over existing delete file wrong count") + assert(spark.sql(s"SELECT count(*) FROM $table.all_delete_files").collect()(0).getLong(0) >= 1, "delete files missing after second delete") + }(), + // Update a surviving row while a delete file is live. + "coexist.update" -> TableTest(Core).step("coexist.update") { (spark, table) => + spark.sql(s"UPDATE $table SET ${Core.string0.columnName} = 'cx' WHERE ${Core.long0.columnName} = 3") + assert(spark.sql(s"SELECT ${Core.string0.columnName} FROM $table WHERE ${Core.long0.columnName} = 3").collect()(0).getString(0) == "cx", "update over live delete failed") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "update over live delete changed count") + }(), + // A filtered read must apply the position delete (the deleted key must never appear). + "coexist.readFilter" -> TableTest(Core).step("coexist.readFilter") { (spark, table) => + val keys = spark.sql(s"SELECT ${Core.long0.columnName} FROM $table WHERE ${Core.long0.columnName} <= 2 ORDER BY ${Core.long0.columnName}").collect().toSeq.map(_.getLong(0)) + assert(keys == Seq(2L), s"filter must apply the position delete (key 1 gone): $keys") + }(), + // Compacting the position deletes materializes them; the row set is unchanged. + "coexist.compactDeletes" -> TableTest(Core).step("coexist.compactDeletes") { (spark, table) => + spark.sql(s"CALL openhouse.system.rewrite_position_delete_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "compact position deletes changed row set") + }(), + // Merge onto a table with a live delete file. + "coexist.merge" -> TableTest(Core).step("coexist.merge") { (spark, table) => + spark.sql(s"MERGE INTO $table t USING (SELECT CAST(3 AS BIGINT) k) s ON t.${Core.long0.columnName} = s.k " + + s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'mg'") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "merge over live delete changed count") + assert(spark.sql(s"SELECT ${Core.string0.columnName} FROM $table WHERE ${Core.long0.columnName} = 3").collect()(0).getString(0) == "mg", "merge over live delete failed") + }() + ) + + // ── Maintenance × MoR-with-live-delete (BUILD-STATUS block 8 deepening) ────────────────────── + // The maintenance.* block runs on plain CoW; the genuinely-distinct surface is maintenance over a + // table that carries a LIVE position-delete file. `createAndSeedMorDeleted` leaves keys 2,3 live + // with a live delete for key 1. The hunt: does each maintenance procedure handle the delete file + // correctly (fold / preserve / not resurrect the deleted row)? + + // rewrite_data_files over a live position delete: it applies the delete to the rewritten data + // (key 1 physically gone, row set correct) — but it does NOT remove the now-dangling position + // delete from the CURRENT snapshot. FINDING G14 (characterization): the compacted table still + // carries a live delete-file reference that points at data already removed; it lingers until + // rewrite_position_delete_files or expire_snapshots. Reads stay correct throughout. Crossed × 3 MoR + // formats to confirm the behavior is format-consistent (the delete decode differs per format). + val maintenanceMorFoldOps: List[(String, TableTest[CoreTable.type])] = List( + "maint.mor.rewriteDataFilesDanglingDelete" -> TableTest(Core).step("maint.mor.rewriteDataFilesDanglingDelete") { (spark, table) => + spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") + // the delete IS applied logically — row set is correct + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "rewrite_data_files changed the live row set over a MoR delete") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 0, "rewrite_data_files RESURRECTED the deleted row") + // G14 PIN: the position delete is NOT removed from the current snapshot — it dangles. + val delFiles = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) + assert(delFiles == 1, s"characterized: rewrite_data_files leaves the position delete dangling in the current snapshot (expected 1), got $delFiles — if this is 0, the build now folds deletes and the pin should flip") + // despite the dangling delete, reads remain correct (the removed row never reappears) + val keys = spark.sql(s"SELECT ${Core.long0.columnName} FROM $table WHERE ${Core.long0.columnName} <= 2 ORDER BY ${Core.long0.columnName}").collect().toSeq.map(_.getLong(0)) + assert(keys == Seq(2L), s"read after rewrite_data_files must stay correct despite the dangling delete: $keys") + }(), + // D5 DECIDER (owner: G14 is a BUG unless the recovery path works, then a PIN): does + // `rewrite_position_delete_files` actually FOLD OUT the dangling position delete that + // rewrite_data_files leaves behind (delete_files 1 -> 0)? If yes, the operator has a working + // additional-maintenance recovery (G14 = pin); if no, the dangling delete is unrecoverable via the + // documented procedure (G14 = bug). Reads must stay correct throughout. × 3 MoR formats. + "maint.mor.rewritePositionDeleteFolds" -> TableTest(Core).step("maint.mor.rewritePositionDeleteFolds") { (spark, table) => + // 1) rewrite_data_files leaves a dangling position delete (the G14 state). + spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") + val danglingBefore = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) + // 2) the recovery path: rewrite_position_delete_files — does it fold the dangling delete out? + spark.sql(s"CALL openhouse.system.rewrite_position_delete_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") + val danglingAfter = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) + println(s"DIAG maint.mor.rewritePositionDeleteFolds: delete_files before=$danglingBefore after=$danglingAfter") + // reads must stay correct regardless (key 1 removed, 2 live rows). + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "rewrite_position_delete_files changed the live row set") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 0, "rewrite_position_delete_files resurrected the deleted row") + // D5 PIN: the recovery WORKS — rewrite_position_delete_files folds the dangling delete out. + assert(danglingBefore == 1 && danglingAfter == 0, + s"D5: expected rewrite_position_delete_files to FOLD the dangling delete (before=1 -> after=0); got before=$danglingBefore after=$danglingAfter — if after>0 the recovery path does NOT work and G14 must be reclassified from pin to BUG") + }() + ) + + // Metadata-only maintenance over a live delete — format is vacuous (these never decode the delete + // file), so × 1 MoR layout. Each must PRESERVE the delete (2 live rows, key 1 still gone). + val maintenanceMorMetaOps: List[(String, TableTest[CoreTable.type])] = List( + "maint.mor.expireSnapshots" -> TableTest(Core).step("maint.mor.expireSnapshots") { (spark, table) => + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "expire_snapshots changed the live row set over a MoR delete") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 0, "expire_snapshots resurrected the deleted row") + }(), + "maint.mor.rewriteManifests" -> TableTest(Core).step("maint.mor.rewriteManifests") { (spark, table) => + spark.sql(s"CALL openhouse.system.rewrite_manifests(table => '${catalogRelative(table)}', use_caching => false)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "rewrite_manifests changed the live row set over a MoR delete") + }(), + "maint.mor.removeOrphanFiles" -> TableTest(Core).step("maint.mor.removeOrphanFiles") { (spark, table) => + spark.sql(s"CALL openhouse.system.remove_orphan_files(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2020-01-01 00:00:00')") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "remove_orphan_files changed the live row set over a MoR delete") + }(), + // Modality: compact the position deletes, THEN expire the pre-compact snapshot — the folded + // state must survive (the deleted row must not reappear via the retained/expired lineage). + "maint.mor.compactThenExpire" -> TableTest(Core).step("maint.mor.compactThenExpire") { (spark, table) => + spark.sql(s"CALL openhouse.system.rewrite_position_delete_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "compact-then-expire changed the live row set") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 0, "compact-then-expire resurrected the deleted row") + }() + ) + + // ── MoR delete-file modality hazards (BUILD-STATUS block 10 deepening) ─────────────────────── + // A live position delete is snapshot-scoped state. These hunt for it being mis-resolved across the + // history/restore axes: a delete must NOT be retroactive (pre-delete snapshots still see the row), + // rollback must UNDO it, and it must SURVIVE expiration of older snapshots. Time-travel/rollback + // logic is format-vacuous (it resolves snapshots, not file bytes) → × 1 MoR layout. + val morHazardOps: List[(String, TableTest[CoreTable.type])] = List( + // The delete is snapshot-scoped: time-travel to the pre-delete snapshot still sees key 1. + "hazard.mor.timeTravelBeforeDelete" -> TableTest(Core).step("hazard.mor.timeTravelBeforeDelete") { (spark, table) => + val seedSnap = spark.sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getLong(0) + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "current MoR state should have the delete applied") + assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF $seedSnap").collect()(0).getLong(0) == 3, + "pre-delete snapshot must still see the deleted row (delete must not be retroactive)") + }(), + // Rollback to the pre-delete snapshot UNDOES the delete — the row returns and no delete is live. + "hazard.mor.rollbackUndoesDelete" -> TableTest(Core).step("hazard.mor.rollbackUndoesDelete") { (spark, table) => + val seedSnap = spark.sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getLong(0) + spark.sql(s"CALL openhouse.system.rollback_to_snapshot(table => '${catalogRelative(table)}', snapshot_id => ${seedSnap}L)") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "rollback did not undo the MoR delete") + assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 1, "rolled-back row not restored") + }(), + // The delete must SURVIVE expiration of the older (pre-delete) snapshot — a filtered read still + // excludes key 1 after expire. + "hazard.mor.expireThenDeleteHolds" -> TableTest(Core).step("hazard.mor.expireThenDeleteHolds") { (spark, table) => + spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") + val keys = spark.sql(s"SELECT ${Core.long0.columnName} FROM $table WHERE ${Core.long0.columnName} <= 2 ORDER BY ${Core.long0.columnName}").collect().toSeq.map(_.getLong(0)) + assert(keys == Seq(2L), s"delete must survive expiration of the pre-delete snapshot (key 1 gone): $keys") + }() + ) + + // ── MoR × branch MERGE (position deletes carried across fast_forward / cherry_pick / REPLACE BRANCH) ── + // A DELETE/UPDATE on a branch of a MoR table writes position-delete files ON THE BRANCH; merging the + // branch back to main must carry those deletes correctly. This is the known-fragile neighborhood of + // G11 (branch × merge) and the "cherry-pick rejects row-delete snapshots" note — the merge is where + // MoR-branch breakage hides. Base is a single-file MoR seed (COALESCE(1)) so a strict-subset DELETE + // is a real position delete, not a file elimination. Merge is a ref/snapshot carry → format-vacuous + // (× 1 MoR layout). Each hunts for: deletes lost/not-carried, deleted rows resurrecting on main, + // cherry-pick rejecting row-delete snapshots. + val morBranchMergeOps: List[(String, TableTest[CoreTable.type])] = List( + // fast_forward must carry a branch position-delete into main: after merge the deleted row is gone. + "mbranch.fastForwardDelete" -> TableTest(Core).step("mbranch.fastForwardDelete") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH mfb") + spark.sql(s"DELETE FROM $table.branch_mfb WHERE ${Core.long0.columnName} = 1") // position delete on branch + assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "main advanced before merge") + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'mfb'") == "2", "branch delete not applied on the branch") + spark.sql(s"CALL openhouse.system.fast_forward('${catalogRelative(table)}', 'main', 'mfb')") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "2", "fast_forward did not carry the branch position-delete to main") + assert(countOf(spark, s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1") == "0", "deleted row resurrected on main after fast_forward") + }(), + // fast_forward must carry a branch UPDATE (MoR update = position delete + new data file). + "mbranch.fastForwardUpdate" -> TableTest(Core).step("mbranch.fastForwardUpdate") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH mub") + spark.sql(s"UPDATE $table.branch_mub SET ${Core.string0.columnName} = 'br-upd' WHERE ${Core.long0.columnName} = 2") + spark.sql(s"CALL openhouse.system.fast_forward('${catalogRelative(table)}', 'main', 'mub')") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "fast_forward of a MoR update changed the row count on main") + assert(spark.sql(s"SELECT ${Core.string0.columnName} FROM $table WHERE ${Core.long0.columnName} = 2").collect()(0).getString(0) == "br-upd", + "MoR update not carried to main by fast_forward") + }(), + // Cherry-pick a branch ROW-DELETE snapshot onto main — CHARACTERIZE (the fragile path): it either + // applies the delete (main → 2) or is rejected; pin the outcome and assert the row set matches it. + "mbranch.cherrypickDelete" -> TableTest(Core).step("mbranch.cherrypickDelete") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH mcb") + spark.sql(s"DELETE FROM $table.branch_mcb WHERE ${Core.long0.columnName} = 1") + val delSnap = spark.sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at DESC LIMIT 1").collect()(0).getLong(0) + val outcome = + try { spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${delSnap}L)"); "ok" } + catch { case NonFatal(e) => s"rejected:${Exceptions.root(e).getClass.getSimpleName}" } + val mainCount = countOf(spark, s"SELECT count(*) FROM $table") + println(s"DIAG mbranch.cherrypickDelete: $outcome, mainCount=$mainCount") + if (outcome == "ok") + assert(mainCount == "2", s"cherrypick reported ok but did not apply the branch delete to main (got $mainCount)") + else + assert(mainCount == "3", s"cherrypick was rejected but main changed anyway (got $mainCount)") + }(), + // REPLACE BRANCH retargets a MoR branch to a pre-delete snapshot — the delete must follow the target. + "mbranch.replaceBranchDelete" -> TableTest(Core).step("mbranch.replaceBranchDelete") { (spark, table) => + val preSnap = spark.sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at DESC LIMIT 1").collect()(0).getLong(0) // seed (3 rows) + spark.sql(s"ALTER TABLE $table CREATE BRANCH mrb") + spark.sql(s"DELETE FROM $table.branch_mrb WHERE ${Core.long0.columnName} = 1") + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'mrb'") == "2", "branch delete not applied") + spark.sql(s"ALTER TABLE $table REPLACE BRANCH mrb AS OF VERSION $preSnap") + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'mrb'") == "3", + "REPLACE BRANCH to the pre-delete snapshot did not undo the branch position-delete") + }() + ) + + // Encryption capability PIN (characterization). OpenHouse delegates table-data encryption to an + // external KMS plugin (private repo); in OSS the catalog never wires a KeyManagementClient, so + // customer tables use the default PlaintextEncryptionManager and data is written UNENCRYPTED. + // Discriminator: a Parquet file's FOOTER magic is "PAR1" when unencrypted and "PARE" under modular + // encryption — robust regardless of compression. This pins that OSS writes plaintext; it FLIPS to + // "PARE" the moment table-data encryption is wired (then update BUGS.md and this pin). An off-the- + // shelf KMS does NOT change this — nothing in the OpenHouse write path invokes the encryption hook. + val encryptionPlaintextPin: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.pin.dataPlaintext") { (spark, table) => + val path = spark.sql(s"SELECT file_path FROM $table.data_files LIMIT 1").collect()(0).getString(0) + val local = path.stripPrefix("file:") + val bytes = java.nio.file.Files.readAllBytes(java.nio.file.Paths.get(local)) + assert(bytes.length >= 8, s"data file too small to inspect: ${bytes.length} bytes") + val footerMagic = new String(bytes.takeRight(4), "US-ASCII") + assert(footerMagic == "PAR1", + s"expected UNENCRYPTED parquet footer magic PAR1 (OSS encryption is un-wired — capability gap, BUGS.md); " + + s"got '$footerMagic' — if 'PARE', table-data encryption is now active and this pin should flip to assert ciphertext") + }() + + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala new file mode 100644 index 000000000..081b4f492 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala @@ -0,0 +1,428 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +trait NegativeDdlScenarios extends ScenarioKit { + import Rows._ + + // ── negative / contract tests ─────────────────────────────────────────────────────────── + // Create + seed a valid CoreTable, then assert the bad operation is rejected. + private def coreNegative(label: String)(bad: (SparkSession, String) => Unit): TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt')")() + .insert(3)() + .step(label)(bad)() + + private val S = CoreTable.string0.columnName + + // Each negative asserts BOTH the exception type and a message substring, so it verifies the + // operation was rejected for the RIGHT reason (not merely that something threw). + val negNonExistentColumn: TableTest[CoreTable.type] = + coreNegative("negative.nonExistentColumn") { (spark, table) => + val e = Check.intercept[AnalysisException](spark.sql(s"DELETE FROM $table WHERE no_such_column = 1")) + assert(e.getMessage.contains("no_such_column")) + } + + val negNonDeterministicDelete: TableTest[CoreTable.type] = + coreNegative("negative.nonDeterministicDelete") { (spark, table) => + val e = Check.intercept[AnalysisException](spark.sql(s"DELETE FROM $table WHERE rand() < 0.5")) + assert(e.getMessage.toLowerCase.contains("deterministic")) + } + + val negNonDeterministicUpdate: TableTest[CoreTable.type] = + coreNegative("negative.nonDeterministicUpdate") { (spark, table) => + val e = Check.intercept[AnalysisException](spark.sql(s"UPDATE $table SET $S = 'x' WHERE rand() < 0.5")) + assert(e.getMessage.toLowerCase.contains("deterministic")) + } + + val negInsertArity: TableTest[CoreTable.type] = + coreNegative("negative.insertArity") { (spark, table) => + val e = Check.intercept[AnalysisException](spark.sql(s"INSERT INTO $table VALUES (CAST(1 AS BIGINT), 1)")) // too few columns + assert(e.getMessage.toLowerCase.contains("not enough data columns")) + } + + // Two UPDATE assignments to the same column in one MERGE clause → analysis error. + val negMergeConflictingUpdates: TableTest[CoreTable.type] = + coreNegative("negative.mergeConflictingUpdates") { (spark, table) => + val e = Check.intercept[AnalysisException](spark.sql( + s"""MERGE INTO $table t USING (SELECT * FROM VALUES (CAST(2 AS BIGINT)) AS s($L)) s + ON t.$L = s.$L + WHEN MATCHED THEN UPDATE SET t.$S = 'a', t.$S = 'b'""")) + assert(e.getMessage.contains("Multiple assignments")) + } + + // Source has two rows matching the same target row → cardinality violation at RUNTIME. The + // concrete runtime exception class (SparkRuntimeException) is package-private, so we anchor on + // the specific message across the cause chain (the error may be wrapped in a task failure). + val negMergeCardinalityViolation: TableTest[CoreTable.type] = + coreNegative("negative.mergeCardinalityViolation") { (spark, table) => + val e = Check.intercept[Exception](spark.sql( + s"""MERGE INTO $table t USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT), 'a'), (CAST(2 AS BIGINT), 'b') AS s($L, $S) + ) s ON t.$L = s.$L + WHEN MATCHED THEN UPDATE SET t.$S = s.$S""")) + assert( + Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(_.contains("matched a single row from the target table"))), + s"expected a MERGE cardinality-violation message, got: ${e.getMessage}") + } + + // CREATE partitioned by a non-existent column (on a scratch name, valid managed table stays). + val negPartitionByNonExistent: TableTest[CoreTable.type] = + coreNegative("negative.partitionByNonExistent") { (spark, table) => + val scratch = table + "_x" + val e = Check.intercept[AnalysisException](spark.sql( + s"CREATE TABLE $scratch ($columnDefinitions) USING $dataSource PARTITIONED BY (no_such_column) TBLPROPERTIES ('write.format.default'='$seedFmt')")) + spark.sql(s"DROP TABLE IF EXISTS $scratch") + assert(e.getMessage.contains("no_such_column")) + } + + val negatives: List[(String, TableTest[CoreTable.type])] = List( + "negative.nonExistentColumn" -> negNonExistentColumn, + "negative.nonDeterministicDelete" -> negNonDeterministicDelete, + "negative.nonDeterministicUpdate" -> negNonDeterministicUpdate, + "negative.insertArity" -> negInsertArity, + "negative.mergeConflictingUpdates" -> negMergeConflictingUpdates, + "negative.mergeCardinalityViolation" -> negMergeCardinalityViolation, + "negative.partitionByNonExistent" -> negPartitionByNonExistent + ) + + // ── DDL Phase 13: schema-evolution negatives ──────────────────────────────────────────── + // DROP COLUMN fails at COMMIT (server 400 → Iceberg BadRequestException); the message carries the + // full body incl. schema dump (AUDIT-FINDINGS B — a "dumb" message), so we anchor on the meaningful + // "Some columns are dropped" reason. Narrowing / SET NOT NULL are caught earlier at Spark analysis + // (ExtendedAnalysisException, a subtype of AnalysisException) with clean messages. + // NOTE: RENAME COLUMN is NOT rejected — it is supported (see ddlRenameColumn in Phase 12). + // DROP COLUMN rejects — but the message is `Column[foo_col_int] not found in newSchema` (buried in a + // double schema dump); it never says "you cannot drop columns" (AUDIT-FINDINGS B, a readability gap). + val ddlNegDropColumn: TableTest[CoreTable.type] = + coreNegative("ddl.neg.dropColumn") { (spark, table) => + val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table DROP COLUMN ${Core.int0.columnName}")) + assert(e.getMessage.contains("not found in newSchema"), s"unexpected message: ${e.getMessage.take(160)}") + assert(e.getMessage.contains(Core.int0.columnName), s"message should name the dropped column: ${e.getMessage.take(160)}") + } + + val ddlNegNarrowType: TableTest[CoreTable.type] = + coreNegative("ddl.neg.narrowType") { (spark, table) => + val e = Check.intercept[AnalysisException](spark.sql(s"ALTER TABLE $table ALTER COLUMN ${Core.long0.columnName} TYPE int")) + assert(e.getMessage.contains("NOT_SUPPORTED_CHANGE_COLUMN"), s"unexpected message: ${e.getMessage.take(160)}") + } + + val ddlNegSetNotNull: TableTest[CoreTable.type] = + coreNegative("ddl.neg.setNotNull") { (spark, table) => + val e = Check.intercept[AnalysisException](spark.sql(s"ALTER TABLE $table ALTER COLUMN ${Core.string0.columnName} SET NOT NULL")) + assert(e.getMessage.contains("Cannot change nullable column to non-nullable"), s"unexpected message: ${e.getMessage.take(160)}") + } + + val ddlNegatives: List[(String, TableTest[CoreTable.type])] = List( + "ddl.neg.dropColumn" -> ddlNegDropColumn, + "ddl.neg.narrowType" -> ddlNegNarrowType, + "ddl.neg.setNotNull" -> ddlNegSetNotNull + ) + + // ── DDL Phase 14: table properties (user keys, reserved-key rejection, forced-override findings) ─ + // Self-contained pipelines (parquet) — property behavior is layout-invariant. `tableProps` reads + // back via SHOW TBLPROPERTIES. + + private def propsCreate(label: String, tblprops: String)(check: StepView[CoreTable.type] => Unit): TableTest[CoreTable.type] = + TableTest(Core).sql(label)(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ($tblprops)")(check) + + // user key round-trips: SET then read back, UNSET removes it + val ddlPropsUserRoundTrip: TableTest[CoreTable.type] = + TableTest(Core) + .sql("ddl.props.userRoundTrip.create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt')")() + .sql("ddl.props.userRoundTrip.set")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('my_key'='my_val')") { view => + assert(tableProps(view.spark, view.table).get("my_key").contains("my_val"), "user prop not set") + } + .sql("ddl.props.userRoundTrip.unset")(t => s"ALTER TABLE $t UNSET TBLPROPERTIES ('my_key')") { view => + assert(!tableProps(view.spark, view.table).contains("my_key"), "user prop not removed") + } + + // reserved-key rejection: an openhouse.* key hits the clean server guard (ALTER_RESERVED_TBLPROPS → + // 400 → BadRequestException). NOTE: `policies` specifically is value-parsed on the CLIENT first, so + // SET('policies'='x') throws a Gson JsonParseException before the guard — recorded in AUDIT-FINDINGS. + val ddlPropsReservedOpenhouse: TableTest[CoreTable.type] = + coreNegative("ddl.props.reservedOpenhouse") { (spark, table) => + val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('openhouse.tableUUID'='deadbeef')")) + assert(e.getMessage.toLowerCase.contains("restriction"), s"msg: ${e.getMessage.take(200)}") + } + + // finding: format-version is forced to the cluster default (2) — a create with '1' still reads 2 + val ddlPropsFormatVersionForced: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt', 'format-version'='1')")() + .insert(3)() + .check("ddl.props.formatVersionForced") { view => + val fv = tableProps(view.spark, view.table).get("format-version") + assert(fv.contains("2"), s"expected forced format-version=2, got $fv") + assert(view.after.size == 3, "table not writable at the forced format-version") // DML-after-DDL + } + + // honored-if-set: previous-versions-max the user provides survives + val ddlPropsPreviousVersionsHonored: TableTest[CoreTable.type] = + propsCreate("ddl.props.previousVersionsHonored", "'write.format.default'='$seedFmt', 'write.metadata.previous-versions-max'='7'") { view => + val v = tableProps(view.spark, view.table).get("write.metadata.previous-versions-max") + assert(v.contains("7"), s"expected previous-versions-max=7, got $v") + } + + val ddlPropsOperations: List[(String, TableTest[CoreTable.type])] = List( + "ddl.props.userRoundTrip" -> ddlPropsUserRoundTrip, + "ddl.props.reservedOpenhouse" -> ddlPropsReservedOpenhouse, + "ddl.props.formatVersionForced" -> ddlPropsFormatVersionForced, + "ddl.props.previousVersionsHonored"-> ddlPropsPreviousVersionsHonored + ) + + // Per-case "current seed format" (default parquet). The assembly's `crossFmt` sets it around each case + // so a block multiplexes across formats WITHOUT every builder taking an explicit fmt param. Safe because + // each case runs sequentially on its own worker thread (session-per-worker, parallel runner). This is + // how format-INERT-by-hypothesis blocks (DDL/props/policy/branch/surface/negatives) get run on ORC too — + + // ── DDL Phase 16: sort order / write distribution ─────────────────────────────────────── + // WRITE ORDERED BY sets the sort order; the observable side effect is write.distribution-mode=range + // (the recon's CatalogOperationTest asserts this). WRITE UNORDERED clears the order. + val ddlWriteOrderedBy: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("ddl.sortOrder.orderedBy")(t => s"ALTER TABLE $t WRITE ORDERED BY ${Core.long0.columnName}") { view => + assert(tableProps(view.spark, view.table).get("write.distribution-mode").contains("range"), + s"distribution-mode not range: ${tableProps(view.spark, view.table).get("write.distribution-mode")}") + } + + val ddlWriteOrderedByMulti: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("ddl.sortOrder.orderedByMulti")(t => + s"ALTER TABLE $t WRITE ORDERED BY ${Core.string0.columnName} DESC NULLS FIRST, ${Core.long0.columnName}") { view => + assert(tableProps(view.spark, view.table).get("write.distribution-mode").contains("range"), "multi-col ordered-by should set range") + } + .insert(2) { view => assert(view.after.size == 5, "multi-col ordered write path failed") } // DML-after-DDL + + // ── DDL Phase 17: rename table (rename to scratch + back, so the harness's fixed table name resolves) ─ + val ddlRenameTable: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("ddl.renameTable") { (spark, table) => + val scratch = s"${table}_ren" + spark.sql(s"ALTER TABLE $table RENAME TO $scratch") + assert(spark.sql(s"SELECT count(*) FROM $scratch").collect()(0).getLong(0) == 3, "renamed table lost rows") + Check.intercept[Exception](spark.sql(s"SELECT 1 FROM $table LIMIT 1")) // old name is gone + spark.sql(s"ALTER TABLE $scratch RENAME TO $table") // restore for teardown + }() + + val ddlRenameTableConflict: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("ddl.renameTable.conflict") { (spark, table) => + val other = s"${table}_other" + spark.sql(s"DROP TABLE IF EXISTS $other") + spark.sql(coreCreateParquet(other)) + val e = Check.intercept[WebClientResponseWithMessageException](spark.sql(s"ALTER TABLE $table RENAME TO $other")) // target exists + assert(e.getMessage.contains("already exists"), s"msg: ${e.getMessage.take(160)}") + spark.sql(s"DROP TABLE IF EXISTS $other") + }() + + // ── DDL Phase 19: namespace DDL negatives (OpenHouse rejects create/drop) ────────────────── + // Both CREATE and DROP NAMESPACE surface `UnsupportedOperationException: "Describing database is not + // supported"` — Spark calls loadNamespaceMetadata first, so the user gets a *describe* message for a + // create/drop (a misleading message — AUDIT-FINDINGS B). We anchor on the stable "not supported". + val ddlNegCreateNamespace: TableTest[CoreTable.type] = + coreNegative("ddl.ns.createRejected") { (spark, _) => + val e = Check.intercept[UnsupportedOperationException](spark.sql("CREATE NAMESPACE openhouse.a_new_db")) + assert(e.getMessage.contains("not supported"), s"msg: ${e.getMessage.take(160)}") + } + + val ddlNegDropNamespace: TableTest[CoreTable.type] = + coreNegative("ddl.ns.dropRejected") { (spark, _) => + val e = Check.intercept[UnsupportedOperationException](spark.sql("DROP NAMESPACE openhouse.dbMatrix")) + assert(e.getMessage.contains("not supported"), s"msg: ${e.getMessage.take(160)}") + } + + val ddlMiscOperations: List[(String, TableTest[CoreTable.type])] = List( + "ddl.sortOrder.orderedBy" -> ddlWriteOrderedBy, + "ddl.sortOrder.orderedByMulti" -> ddlWriteOrderedByMulti, + "ddl.renameTable" -> ddlRenameTable, + "ddl.renameTable.conflict" -> ddlRenameTableConflict, + "ddl.ns.createRejected" -> ddlNegCreateNamespace, + "ddl.ns.dropRejected" -> ddlNegDropNamespace + ) + + // ── DDL Phase 20: policy DDL (OpenHouse SQL extension: ALTER TABLE … SET/UNSET POLICY) ────── + private def policiesBlob(view: StepView[CoreTable.type]): String = + tableProps(view.spark, view.table).getOrElse("policies", "") + + val ddlPolicySharing: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("ddl.policy.sharing")(t => s"ALTER TABLE $t SET POLICY (SHARING=TRUE)") { view => + assert(policiesBlob(view).toLowerCase.contains("true") || policiesBlob(view).toLowerCase.contains("sharing"), + s"sharing policy not stored: ${policiesBlob(view)}") + assert(view.after.size == 3, "table not queryable after SET POLICY (SHARING)") // DML-after-DDL + } + + val ddlPolicyHistory: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("ddl.policy.history")(t => s"ALTER TABLE $t SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20)") { view => + assert(policiesBlob(view).contains("20") || policiesBlob(view).toLowerCase.contains("history"), + s"history policy not stored: ${policiesBlob(view)}") + assert(view.after.size == 3, "table not queryable after SET POLICY (HISTORY)") // DML-after-DDL + } + + val ddlPolicyReplicationRoundTrip: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("ddl.policy.replication.set")(t => s"ALTER TABLE $t SET POLICY (REPLICATION = ({destination:'WAR'}))")() + .sql("ddl.policy.replication.unset")(t => s"ALTER TABLE $t UNSET POLICY (REPLICATION)") { view => + assert(view.after.size == 3) // survives set+unset + } + + val ddlPolicyNegHistoryMaxAge: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("ddl.policy.neg.historyMaxAge") { (spark, table) => + val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table SET POLICY (HISTORY MAX_AGE=5D)")) // > 3 days + assert(e.getMessage.contains("max age must be between 1 to 3 days"), s"msg: ${e.getMessage.take(160)}") + }() + + val ddlPolicyNegHistoryVersions: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("ddl.policy.neg.historyVersions") { (spark, table) => + val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table SET POLICY (HISTORY VERSIONS=200)")) // > 100 + assert(e.getMessage.contains("must be between 2 to 100 versions"), s"msg: ${e.getMessage.take(160)}") + }() + + // Retention on a (string) time-partitioned column requires a column pattern (a valid DateTimeFormatter). + val ddlPolicyRetention: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource PARTITIONED BY (datepartition) TBLPROPERTIES ('write.format.default'='$seedFmt')")().insert(3)() + .sql("ddl.policy.retention")(t => s"ALTER TABLE $t SET POLICY (RETENTION = 30d ON COLUMN datepartition WHERE pattern = 'yyyy-MM-dd-HH')") { view => + assert(policiesBlob(view).toLowerCase.contains("retention") || policiesBlob(view).contains("30"), + s"retention policy not stored: ${policiesBlob(view)}") + assert(view.after.size == 3, "table not queryable after SET POLICY (RETENTION)") // DML-after-DDL + } + + val ddlPolicyOperations: List[(String, TableTest[CoreTable.type])] = List( + "ddl.policy.sharing" -> ddlPolicySharing, + "ddl.policy.history" -> ddlPolicyHistory, + "ddl.policy.replication" -> ddlPolicyReplicationRoundTrip, + "ddl.policy.retention" -> ddlPolicyRetention, + "ddl.policy.neg.historyMaxAge" -> ddlPolicyNegHistoryMaxAge, + "ddl.policy.neg.historyVersions" -> ddlPolicyNegHistoryVersions + ) + + // ── DDL Phase 18: CTAS / RTAS ─────────────────────────────────────────────────────────── + val ddlCtas: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("ddl.ctas") { (spark, table) => + val tgt = s"${table}_ctas" + spark.sql(s"DROP TABLE IF EXISTS $tgt") + spark.sql(s"CREATE TABLE $tgt USING $dataSource AS SELECT * FROM $table") + assert(spark.sql(s"SELECT count(*) FROM $tgt").collect()(0).getLong(0) == 3, "CTAS lost rows") + spark.sql(s"DROP TABLE IF EXISTS $tgt") + }() + + val ddlRtasEnabled: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("ddl.rtas.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('replace.enabled'='true')")() + .step("ddl.rtas.enabled") { (spark, table) => + spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "RTAS did not replace") + }() + + val ddlRtasDisabled: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("ddl.rtas.disabled") { (spark, table) => + val e = Check.intercept[BadRequestException](spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table")) + assert(e.getMessage.contains("REPLACE TABLE AS SELECT is not enabled"), s"msg: ${e.getMessage.take(160)}") + }() + + val ddlRtasReplicationConflict: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("ddl.rtas.repl.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('replace.enabled'='true')")() + .sql("ddl.rtas.repl.policy")(t => s"ALTER TABLE $t SET POLICY (REPLICATION = ({destination:'WAR'}))")() + .step("ddl.rtas.replicationConflict") { (spark, table) => + val e = Check.intercept[BadRequestException](spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table")) + assert(e.getMessage.contains("while replication is enabled"), s"msg: ${e.getMessage.take(160)}") + }() + + val ddlCtasRtasOperations: List[(String, TableTest[CoreTable.type])] = List( + "ddl.ctas" -> ddlCtas, + "ddl.rtas.enabled" -> ddlRtasEnabled, + "ddl.rtas.disabled" -> ddlRtasDisabled, + "ddl.rtas.replicationConflict" -> ddlRtasReplicationConflict + ) + + // ── DDL Phase 22: column tags + ACL (metadata/ACL-plane; tags do NOT mask query results) ──── + val ddlColumnTag: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("ddl.colTag")(t => s"ALTER TABLE $t MODIFY COLUMN ${Core.string0.columnName} SET TAG = (PII)") { view => + val vals = view.spark.sql(s"SELECT ${Core.string0.columnName} FROM ${view.table} ORDER BY ${Core.long0.columnName}").collect().toSeq.map(_.getString(0)) + assert(vals == Seq("row-1", "row-2", "row-3"), s"SET TAG changed query results (should not mask): $vals") + } + + val ddlAclGrantUnshared: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("ddl.acl.grantUnshared") { (spark, table) => + val e = Check.intercept[IllegalArgumentException](spark.sql(s"GRANT SELECT ON TABLE $table TO PUBLIC")) + assert(e.getMessage.contains("is not a shared table"), s"msg: ${e.getMessage.take(160)}") + }() + + // After SHARING=TRUE the grant is accepted (the embedded auth handler records it, no throw). + val ddlAclGrantShared: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("ddl.acl.share")(t => s"ALTER TABLE $t SET POLICY (SHARING=TRUE)")() + .sql("ddl.acl.grantShared")(t => s"GRANT SELECT ON TABLE $t TO PUBLIC") { view => + assert(view.after.size == 3, "shared/granted table not queryable") // DML-after-DDL + } + + // ── DDL Phase 15: feature-flag property (write.distribution-mode governs the write path) ─ + val ddlFeatureDistributionMode: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt', 'write.distribution-mode'='none')")() + .insert(3)() + .check("ddl.featureFlag.distributionMode") { view => + assert(tableProps(view.spark, view.table).get("write.distribution-mode").contains("none"), + s"distribution-mode not honored: ${tableProps(view.spark, view.table).get("write.distribution-mode")}") + assert(view.after.size == 3, "table not writable under distribution-mode=none") // DML-after-DDL + } + + // ── DDL Phase 23: replication / table-type contract (SQL-reachable) ───────────────────────── + val ddlReplTableTypeImmutable: TableTest[CoreTable.type] = + coreNegative("ddl.repl.tableTypeImmutable") { (spark, table) => + val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('openhouse.tableType'='REPLICA_TABLE')")) + assert(e.getMessage.contains("restriction"), s"msg: ${e.getMessage.take(160)}") + } + + val ddlTagAclFeatureOperations: List[(String, TableTest[CoreTable.type])] = List( + "ddl.colTag" -> ddlColumnTag, + "ddl.acl.grantUnshared" -> ddlAclGrantUnshared, + "ddl.acl.grantShared" -> ddlAclGrantShared, + "ddl.featureFlag.distributionMode" -> ddlFeatureDistributionMode, + "ddl.repl.tableTypeImmutable" -> ddlReplTableTypeImmutable + ) + + // ── DDL Phase 24b: encryption — asserts the INTENDED behavior, tagged SKIP in OSS ───────────── + // The KMS plugin is external/private (a repo-wide search finds no EncryptionManager / + // KeyManagementClient / crypto factory / interface / mock). This test asserts what SHOULD happen — + // with encryption configured, the data file must NOT be readable as plaintext parquet. In OSS the + // hook is un-wired so files are plaintext and this would fail; it is tagged in Plan.knownBugs and + // reports SKIP until the private plugin is present (then unskip to validate encryption-ON). + val ddlEncryptionActive: TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='parquet', 'encryption.key-id'='k1', 'write.metadata.encryption.gcm-key-id'='k1')")() + .insert(3)() + .check("ddl.encryption.active") { view => + val filePath = view.spark.sql(s"SELECT file_path FROM ${view.table}.files LIMIT 1").collect()(0).getString(0).stripPrefix("file:") + val head = new String(java.nio.file.Files.readAllBytes(java.nio.file.Paths.get(filePath)).take(4)) + assert(head != "PAR1", s"encryption not in force — data file is plaintext parquet (magic=$head); requires the private KMS plugin") + } + + val ddlEncryptionOperations: List[(String, TableTest[CoreTable.type])] = List( + "ddl.encryption.active" -> ddlEncryptionActive + ) + + // ═══ Feature-INTERACTION axis (INTERACTION-AUDIT.md) — behaviors, single layout ══════════════ + // Characterization stance: rejections are PINS of current behavior (tripwires), not contracts; + // a pin that starts failing means the product changed — update the pin and activate the dormant + // coverage it gates (see the pin inventory in INTERACTION-AUDIT.md §2b). + + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala new file mode 100644 index 000000000..e41fb5bb3 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala @@ -0,0 +1,217 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +trait NestedTypesScenarios extends ScenarioKit { + import Rows._ + + // ── nested / complex types (NestedTable) ─────────────────────────────────────────────── + val nestedLayouts: List[Layout] = + List("parquet", "orc", "avro").map(format => Layout(s"nested-unpartitioned/$format", table => + s"CREATE TABLE $table (${NestedTable.columnDefinitions}) USING $dataSource TBLPROPERTIES ('write.format.default'='$format')")) + + def createAndSeedNested(layout: Layout, numberOfRows: Int): TableTest[NestedTable.type] = + TableTest(NestedTable).sql("create")(layout.create)().insert(numberOfRows)() + + // Read every nested column back and check the seeded values roundtrip. + val nestedRoundtrip: TableTest[NestedTable.type] = + TableTest(NestedTable).check("nested.roundtrip") { view => + val got = view.spark.sql(s"SELECT id, s.x, s.y, arr, m['k'], nested.inner.z FROM ${view.table} ORDER BY id").collect().toSeq + val actual = got.map(r => (r.getLong(0), r.getInt(1), r.getString(2), r.getSeq[Int](3), r.getInt(4), r.getInt(5))) + assert(actual == (1 to 3).map(i => (i.toLong, i, s"row-$i", Seq(i, i + 1), i, i))) + } + + val nestedProjectField: TableTest[NestedTable.type] = + TableTest(NestedTable).check("nested.projectField") { view => + val xs = view.spark.sql(s"SELECT s.x FROM ${view.table} ORDER BY id").collect().map(_.getInt(0)).toSeq + assert(xs == Seq(1, 2, 3)) + } + + val nestedFilterField: TableTest[NestedTable.type] = + TableTest(NestedTable).check("nested.filterNestedField") { view => + val ids = view.spark.sql(s"SELECT id FROM ${view.table} WHERE s.x = 2 ORDER BY id").collect().map(_.getLong(0)).toSeq + assert(ids == Seq(2L)) + } + + // Update a nested struct field. + val nestedUpdateStructField: TableTest[NestedTable.type] = + TableTest(NestedTable).sql("nested.updateStructField")(table => s"UPDATE $table SET s.x = 99 WHERE id = 2") { view => + assert(view.spark.sql(s"SELECT s.x FROM ${view.table} WHERE id = 2").collect()(0).getInt(0) == 99) + assert(view.spark.sql(s"SELECT s.x FROM ${view.table} WHERE id = 1").collect()(0).getInt(0) == 1) + } + + val nestedMergeInsert: TableTest[NestedTable.type] = + TableTest(NestedTable).sql("nested.mergeInsert")(table => + s"""MERGE INTO $table tgt USING ( + SELECT * FROM VALUES + (CAST(4 AS BIGINT), named_struct('x', 4, 'y', 'row-4'), array(4, 5), map('k', 4), named_struct('inner', named_struct('z', 4))) + AS v(id, s, arr, m, nested) + ) src ON tgt.id = src.id + WHEN NOT MATCHED THEN INSERT *""") { view => + val ids = view.spark.sql(s"SELECT id FROM ${view.table} ORDER BY id").collect().map(_.getLong(0)).toSeq + assert(ids == Seq(1L, 2L, 3L, 4L)) + assert(view.spark.sql(s"SELECT s.x FROM ${view.table} WHERE id = 4").collect()(0).getInt(0) == 4) + } + + val nestedDeleteByField: TableTest[NestedTable.type] = + TableTest(NestedTable).sql("nested.deleteByNestedField")(table => s"DELETE FROM $table WHERE s.x = 2") { view => + val ids = view.spark.sql(s"SELECT id FROM ${view.table} ORDER BY id").collect().map(_.getLong(0)).toSeq + assert(ids == Seq(1L, 3L)) + } + + // Insert a row with a null struct and empty array/map. + val nestedNullValues: TableTest[NestedTable.type] = + TableTest(NestedTable).sql("nested.nullValues")(table => + s"INSERT INTO $table VALUES (CAST(4 AS BIGINT), CAST(NULL AS struct), " + + s"CAST(array() AS array), CAST(map() AS map), CAST(NULL AS struct>))") { view => + val row4 = view.spark.sql(s"SELECT id, s, arr FROM ${view.table} WHERE id = 4").collect()(0) + assert(row4.isNullAt(1)) // s is null + assert(row4.getSeq[Int](2).isEmpty) // arr is empty + } + + val nestedOperations: List[(String, TableTest[NestedTable.type])] = List( + "nested.roundtrip" -> nestedRoundtrip, + "nested.projectField" -> nestedProjectField, + "nested.filterNestedField" -> nestedFilterField, + "nested.updateStructField" -> nestedUpdateStructField, + "nested.mergeInsert" -> nestedMergeInsert, + "nested.deleteByNestedField" -> nestedDeleteByField, + "nested.nullValues" -> nestedNullValues + ) + + // ── type-edge coverage (TypesTable) ───────────────────────────────────────────────────── + val typesLayouts: List[Layout] = + List("parquet", "orc", "avro").map(format => Layout(s"types-unpartitioned/$format", table => + s"CREATE TABLE $table (${TypesTable.columnDefinitions}) USING $dataSource TBLPROPERTIES ('write.format.default'='$format')")) + + def createAndSeedTypes(layout: Layout, numberOfRows: Int): TableTest[TypesTable.type] = + TableTest(TypesTable).sql("create")(layout.create)().insert(numberOfRows)() + + // A full valued row for TypesTable with the given id; individual tests override specific columns. + private def typesRow(id: Long, n: String, x: String, dec: String, str: String): String = + s"(CAST($id AS BIGINT), $n, $x, $dec, $str, CAST('b' AS binary), DATE '2024-01-01', " + + s"TIMESTAMP '2024-01-01 00:00:00', TIMESTAMP_NTZ '2024-01-01 00:00:00')" + + val typesRoundtrip: TableTest[TypesTable.type] = + TableTest(TypesTable).check("types.roundtrip") { view => + val r = view.spark.sql(s"SELECT id, n, x, dec, str FROM ${view.table} WHERE id = 1").collect()(0) + assert(r.getLong(0) == 1L && r.getInt(1) == 1 && r.getDouble(2) == 1.5) + assert(r.getDecimal(3).compareTo(new java.math.BigDecimal("1.50")) == 0) + assert(r.getString(4) == "row-1") + } + + val typesNulls: TableTest[TypesTable.type] = + TableTest(TypesTable).sql("types.nulls")(table => + s"INSERT INTO $table VALUES (CAST(10 AS BIGINT), NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)") { view => + val r = view.spark.sql(s"SELECT n, x, str, ts, tsntz FROM ${view.table} WHERE id = 10").collect()(0) + assert((0 to 4).forall(r.isNullAt)) + } + + val typesSpecialFloats: TableTest[TypesTable.type] = + TableTest(TypesTable).sql("types.specialFloats")(table => + s"INSERT INTO $table VALUES ${typesRow(11, "0", "double('NaN')", "CAST(0 AS decimal(10,2))", "'x'")}, " + + s"${typesRow(12, "0", "double('Infinity')", "CAST(0 AS decimal(10,2))", "'y'")}") { view => + assert(view.spark.sql(s"SELECT x FROM ${view.table} WHERE id = 11").collect()(0).getDouble(0).isNaN) + assert(view.spark.sql(s"SELECT x FROM ${view.table} WHERE id = 12").collect()(0).getDouble(0).isInfinite) + } + + val typesBoundaries: TableTest[TypesTable.type] = + TableTest(TypesTable).sql("types.boundaries")(table => + s"INSERT INTO $table VALUES " + + s"${typesRow(9223372036854775807L, "2147483647", "0.0", "CAST(99999999.99 AS decimal(10,2))", "'max'")}") { view => + val r = view.spark.sql(s"SELECT id, n, dec FROM ${view.table} WHERE str = 'max'").collect()(0) + assert(r.getLong(0) == Long.MaxValue && r.getInt(1) == Int.MaxValue) + assert(r.getDecimal(2).compareTo(new java.math.BigDecimal("99999999.99")) == 0) + } + + val typesUnicodeAndEmpty: TableTest[TypesTable.type] = + TableTest(TypesTable).sql("types.unicodeAndEmpty")(table => + s"INSERT INTO $table VALUES ${typesRow(13, "0", "0.0", "CAST(0 AS decimal(10,2))", "'日本語 🎉'")}, " + + s"${typesRow(14, "0", "0.0", "CAST(0 AS decimal(10,2))", "''")}") { view => + assert(view.spark.sql(s"SELECT str FROM ${view.table} WHERE id = 13").collect()(0).getString(0) == "日本語 🎉") + assert(view.spark.sql(s"SELECT str FROM ${view.table} WHERE id = 14").collect()(0).getString(0) == "") + } + + val typesOperations: List[(String, TableTest[TypesTable.type])] = List( + "types.roundtrip" -> typesRoundtrip, + "types.nulls" -> typesNulls, + "types.specialFloats" -> typesSpecialFloats, + "types.boundaries" -> typesBoundaries, + "types.unicodeAndEmpty" -> typesUnicodeAndEmpty + ) + + // ── partition transforms + evolution ──────────────────────────────────────────────────── + // Each transform test is self-contained: create partitioned by the transform, seed, and verify + // the rows roundtrip and a partition spec is registered. + def partitionTransform(transform: String): TableTest[TypesTable.type] = + TableTest(TypesTable) + .sql("create")(table => + s"CREATE TABLE $table (${TypesTable.columnDefinitions}) USING $dataSource PARTITIONED BY ($transform) " + + s"TBLPROPERTIES ('write.format.default'='$seedFmt')")() + .insert(3)() + .check("verify") { view => + assert(view.after.size == 3) + assert(view.spark.sql(s"SELECT * FROM ${view.table}.partitions").collect().nonEmpty) + } + + // A CREATE with an unsupported partition transform is rejected. Run it on a scratch name so the + // pipeline's managed (valid) table still exists for snapshotting. + private def partitionTransformRejected(label: String, transform: String, expectMessage: String): TableTest[TypesTable.type] = + TableTest(TypesTable) + .sql("create")(table => s"CREATE TABLE $table (${TypesTable.columnDefinitions}) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt')")() + .step(label) { (spark, table) => + val scratch = table + "_x" + val error = Check.intercept[RuntimeException](spark.sql( + s"CREATE TABLE $scratch (${TypesTable.columnDefinitions}) USING $dataSource PARTITIONED BY ($transform) TBLPROPERTIES ('write.format.default'='$seedFmt')")) + spark.sql(s"DROP TABLE IF EXISTS $scratch") + assert(error.getMessage.contains(expectMessage)) + }() + + val partitionTransforms: List[(String, TableTest[TypesTable.type])] = List( + "partition.identity" -> partitionTransform("id"), + "partition.bucket" -> partitionTransform("bucket(4, id)"), + "partition.truncate" -> partitionTransform("truncate(2, str)"), + "partition.years" -> partitionTransform("years(ts)"), + "partition.months" -> partitionTransform("months(ts)"), + "partition.days" -> partitionTransform("days(ts)"), + "partition.hours" -> partitionTransform("hours(ts)"), + // OpenHouse contract: these transforms are rejected (negative tests). + "partition.void.rejected" -> partitionTransformRejected("partition.void.rejected", "void(n)", "not supported"), + "partition.dateDay.rejected" -> partitionTransformRejected("partition.dateDay.rejected", "days(dt)", "Unsupported column") + ) + + // OpenHouse contract: partition evolution is NOT supported — ALTER … ADD/DROP PARTITION FIELD is + // rejected with a 400 telling you to recreate the table. Captured as negative tests. + val partitionEvolutionAddRejected: TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt')")() + .insert(3)() + .step("partition.evolutionAdd.rejected") { (spark, table) => + val error = Check.intercept[Exception](spark.sql(s"ALTER TABLE $table ADD PARTITION FIELD datepartition")) + assert(error.getMessage.contains("Evolution of table partitioning")) + }() + + val partitionEvolutionDropRejected: TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource PARTITIONED BY (datepartition) TBLPROPERTIES ('write.format.default'='$seedFmt')")() + .insert(3)() + .step("partition.evolutionDrop.rejected") { (spark, table) => + val error = Check.intercept[Exception](spark.sql(s"ALTER TABLE $table DROP PARTITION FIELD datepartition")) + assert(error.getMessage.contains("Evolution of table partitioning")) + }() + + val partitionEvolution: List[(String, TableTest[CoreTable.type])] = List( + "partition.evolutionAdd.rejected" -> partitionEvolutionAddRejected, + "partition.evolutionDrop.rejected" -> partitionEvolutionDropRejected + ) + + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala new file mode 100644 index 000000000..ab65751b1 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala @@ -0,0 +1,23 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +/** + * The concrete tests, all on CoreTable. An operation is a HEADLESS pipeline segment (no create); + * the run crosses every operation with every `Layout` by composing `createAndSeed(layout)` before + * it via `andThen`. Every operation asserts the DELTA against the observed pre-state (rows and/or + * commit count), never an absolute row set — so a test holds under any layout. Operation sources + * are written as EXPLICIT literals. + */ +// The tests are authored across cohesive per-domain traits (see *Scenarios.scala + ScenarioKit.scala); +// this object assembles them. Trait mixin order == original top-to-bottom source order, so val +// initialization order is preserved. `object Plan` consumes the public members declared here. +object Scenarios extends MorMaintScenarios with DmlScenarios with NestedTypesScenarios with MaintControlScenarios with ForkScenarios with BranchWapScenarios with NegativeDdlScenarios with InteractionScenarios with SurfaceScenarios with HazardReaderWriterScenarios diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala new file mode 100644 index 000000000..23a5924c3 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala @@ -0,0 +1,282 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +/** Assembles the run: every operation x every layout, plus create.schema per layout. */ +object Plan { + final case class Case(id: String, run: Ctx => Unit) + + // Known PRODUCT bugs: any case whose id contains the key is reported SKIP (bug: reason) instead + // of failing the suite, and is tracked in BUGS.md. This is how we "tag a failing test and filter + // it": a genuine bug is tagged here, deferred for follow-up, and never plowed past silently. + val knownBugs: List[(String, String)] = List( + // insert.explicitColumns is NO LONGER a bug tag — reclassified to a negative PIN (engine limitation, + // not OpenHouse; code-verified). See insertExplicitColumns above and BUGS.md. + "nested.deleteByNestedField" -> + "DELETE WHERE crashes with an internal optimizer NPE (SELECT/UPDATE on the same field work). Code-verified UPSTREAM: OpenHouse contributes no code to the row-level DELETE rewrite (owned by IcebergSparkSessionExtensions + Spark optimizer); the NPE is in the nested-field DELETE-rewrite plan. Needs a full stack capture before filing — see BUGS.md", + "prep.ordered:delete.byPartitionPredicate" -> + "DELETE by a partition predicate against a table created WITH a WRITE ORDERED BY clause throws an internal analyzer NPE, while the same DELETE on an unordered table (delete.byPartitionPredicate) succeeds. Code-verified UPSTREAM and the same family as nested.deleteByNestedField: OpenHouse contributes no code to the row-level DELETE rewrite (owned by IcebergSparkSessionExtensions plus the Spark optimizer), so the NPE lives in the ORDERED-BY DELETE-rewrite plan on Spark 3.5.2 / Iceberg 1.5.2. It reproduces identically on the embedded catalog and on the remote cluster, so it is gated in both environments. Needs a full stack capture before filing — see BUGS.md", + "ddl.renameColumn" -> + "RENAME COLUMN is a silent no-op. Code-verified GENUINE OpenHouse regression from #558 (commit 0ad4914): server-side normalizeSchemaCasingToTable rewrites every field's name to the table's spelling BY FIELD ID (BaseIcebergSchemaValidator:60-73), reverting the rename, and it runs BEFORE the sameSchema gate so validateWriteSchema (which would reject loudly) never fires. Fix: guard the normalizer with equalsIgnoreCase. Silent failure worse than the pre-#558 clean rejection — see BUGS.md", + "ddl.encryption" -> + "encryption KMS plugin is external/private (no impl/interface/mock in-repo); OSS leaves the encryption() hook un-wired and writes plaintext, so the intended-behavior assertion is deferred until the plugin is present — see DDL-TEST-PLAN.md / AUDIT-FINDINGS.md", + "control.undrop" -> + "undrop is SKIP under the DEFAULT stub path (HouseTableRepository is a @Primary in-memory stub; the public Tables DELETE hard-codes purge=true). Under HARNESS_REAL_HTS=1 the real embedded HTS is booted and undrop runs for real as the undrop:* battery + undropAdmin.* lifecycle (NOT SKIP) — see HTS-EMBED-PLAN.md / HTS-EMBED-IMPL.md / REST-FIDELITY-EVAL.md" + ) + + def bugReason(id: String): Option[String] = + knownBugs.collectFirst { case (key, reason) if id.contains(key) => s"bug: $reason" } + + def cases: List[Case] = { + val dml = for { + layout <- Scenarios.layouts + (name, op) <- Scenarios.operations + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeed(layout, 3).andThen(op).run) + + val partitioned = for { + layout <- Scenarios.layouts.filter(_.label.startsWith("partitioned/")) + (name, op) <- Scenarios.partitionedOperations + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeed(layout, 3).andThen(op).run) + + // Merge-on-read: the same mutation operations, prepared on a MoR table. + val mor = for { + layout <- Scenarios.morLayouts + (name, op) <- Scenarios.mutationOperations + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeed(layout, 3).andThen(op).run) + + // MoR discriminator: prove merge-on-read wrote delete files, and copy-on-write did not. + val morVerify = Scenarios.morVerifyLayouts.map(layout => + Case(s"mor.writesDeleteFiles @ ${layout.label}", Scenarios.createAndSeedSingleFile(layout, 3).andThen(Scenarios.morWritesDeleteFiles).run)) + val cowVerify = Scenarios.cowVerifyLayouts.map(layout => + Case(s"cow.writesNoDeleteFiles @ ${layout.label}", Scenarios.createAndSeedSingleFile(layout, 3).andThen(Scenarios.cowWritesNoDeleteFiles).run)) + + // Nested / complex types, on their own schema and layouts. + val nested = for { + layout <- Scenarios.nestedLayouts + (name, op) <- Scenarios.nestedOperations + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedNested(layout, 3).andThen(op).run) + + // Type-edge coverage, on TypesTable. + val types = for { + layout <- Scenarios.typesLayouts + (name, op) <- Scenarios.typesOperations + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedTypes(layout, 3).andThen(op).run) + + // Format multiplex. Blocks whose tables are seeded via the format-aware create helpers (coreCreateParquet + // / coreCreate / propsCreate / the ddl inline creates now reading $seedFmt) run on parquet AND orc: any + // table-creating op has a real format axis, and "format-inert" is a HYPOTHESIS this harness verifies, not + // assumes. `crossFmt` sets the per-case seed format around each case (safe — cases are sequential per worker). + val dataFormats = List("parquet", "orc") + def crossFmt[S <: Schema](block: List[(String, TableTest[S])]): List[Plan.Case] = + for { f <- dataFormats; (name, t) <- block } yield Case(s"$name @ $f", ctx => Scenarios.withSeedFmt(f)(t.run(ctx))) + + // Partition transforms + evolution — multiplex (format is a hypothesis to verify, not assume). + val partitionTransforms = crossFmt(Scenarios.partitionTransforms) + val partitionEvolution = crossFmt(Scenarios.partitionEvolution) + + val timeTravel = for { f <- dataFormats; (name, t) <- Scenarios.timeTravelOps(f) } yield Case(s"$name @ $f", t.run) + val restoreRollback = for { f <- dataFormats; (name, t) <- Scenarios.restoreRollbackOps(f) } yield Case(s"$name @ $f", t.run) + val maintenance = for { f <- dataFormats; (name, t) <- Scenarios.maintenanceOps(f) } yield Case(s"$name @ $f", t.run) + val control = Scenarios.controlPlane.map { case (name, f) => Case(s"$name @ embedded", f) } + val forkColDefault = Scenarios.forkColDefaultOps.map { case (name, f) => Case(name, f) } + val forkPartitionDist = Scenarios.forkPartitionDistOps.map { case (name, f) => Case(name, f) } + val forkDeleteFileReplication = Scenarios.forkDeleteFileReplicationOps.map { case (name, f) => Case(name, f) } + val forkFileReplicationFactor = Scenarios.forkFileReplicationFactorOps.map { case (name, f) => Case(name, f) } + val forkSplitSize = Scenarios.forkSplitSizeOps.map { case (name, f) => Case(name, f) } + val forkBinPackByLength = Scenarios.forkBinPackByLengthOps.map { case (name, f) => Case(name, f) } + val forkCompactionOrder = Scenarios.forkCompactionOrderOps.map { case (name, f) => Case(name, f) } + val branching = crossFmt(Scenarios.branching) + val branchDdl = crossFmt(Scenarios.branchDdlOps) // WAP mega-axis Stage B (G8 leak, systematic) + val wapStaged = crossFmt(Scenarios.wapStagedOps) // WAP mega-axis Stage C (staged → publish) + val interactions = crossFmt(Scenarios.interactions) ++ + Scenarios.interactionCtxOps.map { case (name, f) => Case(s"$name @ embedded", f) } + val surface = crossFmt(Scenarios.surfaceOps) + val hazards = crossFmt(Scenarios.hazardOps) ++ + Scenarios.hazardCtxOps.map { case (name, f) => Case(s"$name @ embedded", f) } + val readerWriter = for { f <- dataFormats; (name, t) <- Scenarios.readerWriterOps(f) } yield Case(s"$name @ $f", t.run) + val negatives = crossFmt(Scenarios.negatives) + val ddlNegatives = crossFmt(Scenarios.ddlNegatives) + val ddlProps = crossFmt(Scenarios.ddlPropsOperations) + val ddlMisc = crossFmt(Scenarios.ddlMiscOperations) + val ddlPolicy = crossFmt(Scenarios.ddlPolicyOperations) + val ddlCtasRtas = crossFmt(Scenarios.ddlCtasRtasOperations) + val ddlTagAcl = crossFmt(Scenarios.ddlTagAclFeatureOperations) + val ddlEncryption = Scenarios.ddlEncryptionOperations.map { case (name, t) => Case(s"$name @ parquet", t.run) } + + // Phase 24 prep multipliers (full DML cross). Ordered prep × all operations; evolved prep × + // delete/update/read only (ADD COLUMN changes INSERT arity, breaking full-column inserts). + val ddlPrepOrdered = for { + layout <- Scenarios.layouts + (name, op) <- Scenarios.operations + } yield Case(s"prep.ordered:$name @ ${layout.label}", Scenarios.createAndSeedOrdered(layout, 3).andThen(op).run) + + // delete/update/read only, and excluding ops that internally INSERT a full-column row + // (delete.byNullCondition seeds a null row) — those hit the arity mismatch on the +1-column table. + val ddlPrepEvolved = for { + layout <- Scenarios.layouts + (name, op) <- Scenarios.operations.filter { case (n, _) => + (n.startsWith("delete.") || n.startsWith("update.") || n.startsWith("read.")) && !n.contains("byNullCondition") } + } yield Case(s"prep.evolved:$name @ ${layout.label}", Scenarios.createAndSeedEvolved(layout, 3).andThen(op).run) + + // T axis — the whole DML catalog routed onto a BRANCH via spark.wap.branch (SURFACE-APPRAISAL + // step 3). Format is vacuous for branches (refs never touch file encoding), so parquet only; + // both partitionings kept (partitioning changes overwrite/dynamic-overwrite semantics on the + // branch). Every op asserts its normal delta — now proving the op works branch-routed AND that + // main is untouched (isolation). ~106 cases. + // Format policy: ORC + Parquet (both), not parquet-only. Avro is intentionally NOT added to these + // ref/metadata-routed blocks (branch/undrop/DDL-consumer) — the additive ask was ORC, and the + // 3-format blocks keep Avro separately. + val branchParquetLayouts = Scenarios.layouts.filter(l => l.label.endsWith("/parquet") || l.label.endsWith("/orc")) + // WAP mega-axis Stage A — branch DML parity with the core CREATE path: all 6 layouts (incl avro) × + // operations, routed onto a branch, asserting branch delta + main isolation. + val branchWap = for { + layout <- Scenarios.layouts + (name, op) <- Scenarios.operations + } yield Case(s"branchWap:$name @ ${layout.label}", + Scenarios.createAndSeedOnBranch(layout, 3).andThen(op).andThen(Scenarios.branchMainIsolation).run) + + // Stage A — partition-only ops routed onto a branch (mirrors the core `partitioned` block). + val branchWapPartitioned = for { + layout <- Scenarios.layouts.filter(_.label.startsWith("partitioned/")) + (name, op) <- Scenarios.partitionedOperations + } yield Case(s"branchWap:$name @ ${layout.label}", + Scenarios.createAndSeedOnBranch(layout, 3).andThen(op).andThen(Scenarios.branchMainIsolation).run) + + // Branch × MoR — mutation ops routed onto a branch of a MoR table (cherry-pick rejects row-delete + // snapshots). 3-format for parity with morLayouts (Stage A). + val branchMorLayout = Scenarios.morLayouts.filter(_.label.startsWith("mor-unpartitioned/")) + val branchWapMor = for { + layout <- branchMorLayout + (name, op) <- Scenarios.mutationOperations + } yield Case(s"branchWap:$name @ ${layout.label}", + Scenarios.createAndSeedOnBranch(layout, 3).andThen(op).andThen(Scenarios.branchMainIsolation).run) + + // P axis (replace-lineage leg) — the whole DML catalog on an RTAS'd table (SURFACE-APPRAISAL + // step 2). ~106 cases. (The undrop leg is gated on the embedded-HTS restructure — see + // REST-FIDELITY-EVAL.md — so only the RTAS leg is runnable now.) + val prepRtas = for { + (label, partitionClause, fmt) <- Scenarios.rtasPrepShapes + (name, op) <- Scenarios.operations + } yield Case(s"prep.rtas:$name @ $label", Scenarios.createAndSeedRtas(partitionClause, 3, fmt).andThen(op).run) + + // RTAS full cross (Phase 28): partition-only ops on the partitioned RTAS shapes — mirrors the core + // `partitioned` block (partitionedOperations × partitioned layouts) but on a replace-lineage base. + val prepRtasPartitioned = for { + (label, partitionClause, fmt) <- Scenarios.rtasPrepShapes.filter(_._1.startsWith("partitioned/")) + (name, op) <- Scenarios.partitionedOperations + } yield Case(s"prep.rtas:$name @ $label", Scenarios.createAndSeedRtas(partitionClause, 3, fmt).andThen(op).run) + + // RTAS × MoR — mutation ops on a replace-lineage MoR table. 3-format for parity with the core MoR + // block (morLayouts = parquet/orc/avro), per the Phase-28 full cross. + val prepRtasMor = for { + fmt <- List("parquet", "orc", "avro") + (name, op) <- Scenarios.mutationOperations + } yield Case(s"prep.rtasMor:$name @ mor-unpartitioned/$fmt", + Scenarios.createAndSeedRtasMor("", 3, fmt).andThen(op).run) + + // P axis (drop→undrop leg) — the whole DML catalog on a table taken through a real HTS soft-delete + // → restore round-trip (SURFACE-APPRAISAL). Requires the embedded real HTS (HARNESS_REAL_HTS=1); + // empty otherwise. This is the surface-DOUBLING leg: every op re-verifies that the restored table + // still behaves identically, i.e. that restore's destruction set does not intersect the feature's + // state-dependency set. Undrop is metadata/ref reconstruction — file encoding is vacuous → parquet + // layouts only (as with RTAS/branch). + val undrop = + if (HtsAdmin.enabled) for { + layout <- branchParquetLayouts + (name, op) <- Scenarios.operations + } yield Case(s"undrop:$name @ ${layout.label}", + Scenarios.createAndSeedUndropped(layout, 3).andThen(op).run) + else Nil + + // Undrop admin-lifecycle block (Phase 5) — soft-delete/list/restore/purge, real HTS only. + val undropAdmin = + if (HtsAdmin.enabled) Scenarios.undropAdminOps.map { case (name, run) => Case(name, run) } + else Nil + + // Block 9 deepening: undrop 3-way compositions (branch/time-travel/schema survival), real HTS only. + val undropInteract = + if (HtsAdmin.enabled) Scenarios.undropInteractOps.map { case (name, run) => Case(name, run) } + else Nil + + // DDL × consumer battery (task #3): each state-changing DDL, then each consumer must still work. + // 4 DDL × 6 consumers × {unpartitioned, partitioned}/parquet = 48. + val ddlConsumerBattery = for { + layout <- branchParquetLayouts + (ddlName, prep) <- Scenarios.ddlPreps + (conName, con) <- Scenarios.ddlConsumers + } yield Case(s"ddlConsume:$ddlName.$conName @ ${layout.label}", prep(layout).andThen(con).run) + + // MoR reads with a live position delete (closes the scan-path gap, step 1). Read/scan ops only — + // they must apply the position delete at read time. Across formats (delete-file encoding differs). + val morReadOps = Scenarios.operations.filter { case (n, _) => n.startsWith("read.") || n == "format.materialization" } + val prepMorRead = for { + layout <- Scenarios.morVerifyLayouts // single-file-friendly MoR layouts, per format + (name, op) <- morReadOps + } yield Case(s"prep.morRead:$name @ ${layout.label}", Scenarios.createAndSeedMorDeleted(layout, 3).andThen(op).run) + + // MoR delete-file COEXISTENCE (task #5 non-vacuous core): ops on a table that already carries a + // live position delete. Format matters (delete-file encoding) → × 3 MoR formats. + val morCoexist = for { + layout <- Scenarios.morVerifyLayouts + (name, op) <- Scenarios.morCoexistOps + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedMorDeleted(layout, 3).andThen(op).run) + + // Block 8 deepening: maintenance × MoR-with-live-delete. The delete-DECODE op (rewrite_data_files + // fold) is format-relevant → × 3 MoR formats; metadata-only maintenance is format-vacuous → × 1. + val maintenanceMorFold = for { + layout <- Scenarios.morVerifyLayouts + (name, op) <- Scenarios.maintenanceMorFoldOps + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedMorDeleted(layout, 3).andThen(op).run) + val morParquetVerify = Scenarios.morVerifyLayouts.filter(l => l.label == "mor-verify/parquet" || l.label == "mor-verify/orc") + val maintenanceMorMeta = for { + layout <- morParquetVerify + (name, op) <- Scenarios.maintenanceMorMetaOps + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedMorDeleted(layout, 3).andThen(op).run) + + // Block 10 deepening: MoR delete-file modality hazards (time-travel / rollback / expire). Snapshot + // logic is format-vacuous → × 1 MoR layout. + val morHazard = for { + layout <- morParquetVerify + (name, op) <- Scenarios.morHazardOps + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedMorDeleted(layout, 3).andThen(op).run) + + // MoR × branch MERGE: position deletes carried across fast_forward / cherry_pick / REPLACE BRANCH. + // Single-file MoR seed so a branch DELETE is a real position delete; merge is format-vacuous → ×1. + val morBranchMerge = for { + layout <- morParquetVerify + (name, op) <- Scenarios.morBranchMergeOps + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedSingleFile(layout, 3).andThen(op).run) + + // Encryption capability pin (characterization): OSS writes plaintext parquet (encryption un-wired). + val encryptionPin = List(Case("surface.pin.dataPlaintext @ parquet", Scenarios.encryptionPlaintextPin.run)) + + val creates = Scenarios.layouts.map { layout => + Case(s"create.schema @ ${layout.label}", Scenarios.createSchema(layout).run) + } + + // DDL Phase 12: schema-evolution behaviors crossed with every layout. + val ddlSchema = for { + layout <- Scenarios.layouts + (name, op) <- Scenarios.ddlSchemaOperations + } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeed(layout, 3).andThen(op).run) + + dml ++ partitioned ++ mor ++ morVerify ++ cowVerify ++ nested ++ types ++ partitionTransforms ++ + partitionEvolution ++ timeTravel ++ restoreRollback ++ negatives ++ creates ++ ddlSchema ++ + ddlNegatives ++ ddlProps ++ ddlMisc ++ ddlPolicy ++ ddlCtasRtas ++ ddlTagAcl ++ ddlEncryption ++ + maintenance ++ control ++ branching ++ interactions ++ surface ++ hazards ++ branchWap ++ + branchDdl ++ wapStaged ++ branchWapPartitioned ++ branchWapMor ++ prepRtas ++ prepRtasPartitioned ++ prepRtasMor ++ prepMorRead ++ morCoexist ++ ddlConsumerBattery ++ + readerWriter ++ ddlPrepOrdered ++ ddlPrepEvolved ++ undrop ++ undropAdmin ++ + maintenanceMorFold ++ maintenanceMorMeta ++ undropInteract ++ morHazard ++ morBranchMerge ++ + encryptionPin ++ forkColDefault ++ forkPartitionDist ++ + forkDeleteFileReplication ++ forkFileReplicationFactor ++ forkSplitSize ++ + forkBinPackByLength ++ forkCompactionOrder + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala new file mode 100644 index 000000000..9e52bce4e --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala @@ -0,0 +1,275 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// Shared foundation for every Scenario trait: the table/layout/prep "kit". All domain traits +// (DmlScenarios, ForkScenarios, ...) extend this, so mixing them into `object Scenarios` puts +// ScenarioKit first in the linearization → its vals initialize before any domain's, exactly as +// in the original single object. `protected` members are the shared kit; `public` ones are also +// consumed by `object Plan`. +trait ScenarioKit { + import Rows._ + + protected val Core = CoreTable // brevity in the typed column references below + protected val cols = Core.columnNames.mkString(", ") // source column list, so renames propagate + + // Short typed views of the current rows, keyed by the long column, for incremental assertions. + protected def keyed(rows: Seq[Row]): Seq[Long] = rows.map(_.get(Core.long0)).sorted + protected def longToString(rows: Seq[Row]): Map[Long, String] = + rows.map(row => row.get(Core.long0) -> row.get(Core.string0)).toMap + + // ── the layout axis: file format x partitioning, crossed with every operation ────────── + // Each layout is a plain literal CREATE statement (no dynamic assembly): the column list is one + // shared literal `columnDefinitions`, and format/partition are literal fragments. createSchema + // cross-checks the literal against CoreTable's declared columns, so the two can't silently drift. + protected val columnDefinitions = + "foo_col_long bigint, foo_col_int int, foo_col_string string, foo_col_double double, foo_col_boolean boolean, datepartition string" + + final case class Layout(label: String, create: String => String) + + protected val partitionVariants = List("unpartitioned" -> "", "partitioned" -> "PARTITIONED BY (datepartition)") + + val layouts: List[Layout] = + for { + format <- List("parquet", "orc", "avro") + (partitionLabel, partitionClause) <- partitionVariants + } yield Layout(s"$partitionLabel/$format", table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource $partitionClause " + + s"TBLPROPERTIES ('write.format.default'='$format')") + + // Merge-on-read layouts: same shapes, but DELETE/UPDATE/MERGE write position-delete files + // (format v2) instead of rewriting data files. Crossed with the mutation operations only. + val morLayouts: List[Layout] = + for { + format <- List("parquet", "orc", "avro") + (partitionLabel, partitionClause) <- partitionVariants + } yield Layout(s"mor-$partitionLabel/$format", table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource $partitionClause " + + s"TBLPROPERTIES ('write.format.default'='$format', 'format-version'='2', " + + s"'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read', 'write.merge.mode'='merge-on-read')") + + // Dedicated layouts for the CoW/MoR *physical* discriminator (below). Both pin + // `write.distribution-mode=none` and are unpartitioned so a single seed INSERT lands all rows in + // ONE data file; deleting a strict subset is then necessarily a PARTIAL-file match, which Iceberg + // cannot satisfy by whole-file elimination. That makes the physical outcome deterministic: MoR + // must add a position-delete file, CoW must rewrite the data file and add none. (The general + // `morLayouts` seed splits across files, so a boundary-aligned delete can legitimately drop a + // whole file with no position delete — correct Iceberg behaviour, but not what we want to pin.) + val morVerifyLayouts: List[Layout] = + List("parquet", "orc", "avro").map(format => Layout(s"mor-verify/$format", table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'format-version'='2', 'write.distribution-mode'='none', " + + s"'write.delete.mode'='merge-on-read')")) + + val cowVerifyLayouts: List[Layout] = + List("parquet", "orc", "avro").map(format => Layout(s"cow-verify/$format", table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'format-version'='2', 'write.distribution-mode'='none', " + + s"'write.delete.mode'='copy-on-write')")) + + // Preparation: create under `layout` and seed `numberOfRows` deterministic rows. Interchangeable + // with RTAS / drop+undrop preparations later — same resulting state. + def createAndSeed(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(layout.create)().insert(numberOfRows)() + + // Preparation for the physical CoW/MoR discriminator: seed all rows into ONE data file. A plain + // seed INSERT fans the rows across a couple of files (writer-dependent), so a strict-subset delete + // can land on a whole file and be satisfied by file elimination rather than a position delete. The + // `COALESCE(1)` hint forces a single write task → a single data file, so deleting a strict subset + // is deterministically a PARTIAL-file match: MoR must add a position-delete file, CoW must rewrite. + def createAndSeedSingleFile(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(layout.create)() + .sql(s"seed($numberOfRows, one-file)")(table => + s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM (${RowGenerator.valuesClause(Core, numberOfRows)}) AS seed")( + view => assert(view.after.size == numberOfRows, + s"single-file seed expected $numberOfRows rows, got ${view.after.size}")) + + // Phase 24 preparation multipliers: a DDL evolves the starting state, then a DML op runs on it. + // Ordered prep (sort order) is arity-neutral → crosses ALL operations. Evolved prep adds a column + // → INSERT arity changes, so it crosses only ops that don't re-insert all columns (delete/update/read). + def createAndSeedOrdered(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = + createAndSeed(layout, numberOfRows).sql("prep.ordered")(t => s"ALTER TABLE $t WRITE ORDERED BY ${CoreTable.long0.columnName}")() + + def createAndSeedEvolved(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = + createAndSeed(layout, numberOfRows).sql("prep.evolved")(t => s"ALTER TABLE $t ADD COLUMN prep_extra int")() + + // Branch-routing prep (the T axis, wap-conf mechanism): seed on main, fork a branch, then set + // spark.wap.branch so the ENTIRE downstream operation (writes AND reads) routes to the branch — + // no per-op rewrite needed. The op's delta assertions are relative to view.before (also the + // branch), so they hold unchanged. Each case runs in its own spark.newSession() (parallel runner), + // so the conf never leaks across cases. This crosses the whole DML catalog onto a branch. + def createAndSeedOnBranch(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = + createAndSeed(layout, numberOfRows) + .sql("prep.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("prep.routeToBranch") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH b") + spark.conf.set("spark.wap.branch", "b") + }() + + // RTAS prep prefix (the P axis, replace-lineage leg — SURFACE-APPRAISAL step 2): create + seed, + // then CREATE OR REPLACE ... AS SELECT * re-specifying the SAME shape, so the table is + // functionally identical but reached via the replace path (the path G9/G10 showed misbehaves). + // Every downstream DML op then runs on a replace-lineage table. FULL CROSS (Phase 28): all 6 layouts + // ({unpartitioned,partitioned} × {parquet,orc,avro}) — mirrors the core `dml` block's layout coverage so + // the RTAS/replace-lineage substrate carries the same DML surface as the plain CREATE substrate. + // (label, partitionClause, format). + val rtasPrepShapes: List[(String, String, String)] = + for { (pl, pc) <- partitionVariants; fmt <- List("parquet", "orc", "avro") } yield (s"$pl/$fmt", pc, fmt) + + // MoR-read prep (closes the review's "reads on MoR with deletes is a distinct scan path" gap — + // SURFACE-APPRAISAL step 1). The current MoR bucket runs mutation ops (each reads back once), but + // never crosses the READ variants against a table carrying a LIVE position delete. Seed a single + // data file (COALESCE(1)) on a MoR layout, delete a strict subset → a position-delete file the + // reader must APPLY at scan time (not a whole-file elimination). Downstream read ops then assert + // the deleted row is excluded under each read shape (projection, filter-pushdown, ...). + def createAndSeedMorDeleted(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = + createAndSeedSingleFile(layout, numberOfRows) + .step("prep.morDelete") { (spark, table) => + spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") // strict subset → position delete + } { view => + assert(view.after.size == numberOfRows - 1, s"MoR prep delete failed: ${view.after.size}") + val deleteFiles = view.spark.sql(s"SELECT count(*) FROM ${view.table}.all_delete_files").collect()(0).getLong(0) + assert(deleteFiles == 1, s"MoR prep must leave a live position-delete file, got $deleteFiles") + } + + // Undrop prep (the P axis, drop→undrop leg — SURFACE-APPRAISAL, requires embedded real HTS). Seed a + // plain table, then take it through the FULL soft-delete → restore round-trip on the real HTS, and + // hand the RESTORED table to the downstream op. The point is a modality audit: every feature's state + // (rows, snapshot lineage, refs, spec, sort order, properties, MoR delete files, schema) must survive + // the round-trip, so the whole DML/DDL catalog is crossed onto the restored table. Soft-delete is + // driven directly on HTS (customer DROP hard-deletes); restore uses the customer Tables API. + def createAndSeedUndropped(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = + createAndSeed(layout, numberOfRows) + .step("prep.undrop") { (spark, table) => + val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) + val (sdCode, sdBody) = HtsAdmin.softDelete(db, tbl) + assert(sdCode >= 200 && sdCode < 300, s"HTS soft-delete failed ($sdCode): $sdBody") + val deletedAtMs = HtsAdmin.softDeletedAtMs(db, tbl) + .getOrElse(throw new AssertionError(s"soft-deleted table $db.$tbl not found in querySoftDeleted")) + val (rCode, rBody) = HtsAdmin.restore(db, tbl, deletedAtMs) + assert(rCode >= 200 && rCode < 300, s"restore failed ($rCode): $rBody") + } { view => + assert(view.after.size == numberOfRows, + s"restored table must keep its $numberOfRows rows, got ${view.after.size}") + } + + def createAndSeedRtas(partitionClause: String, numberOfRows: Int, format: String = "parquet"): TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource $partitionClause " + + s"TBLPROPERTIES ('write.format.default'='$format', 'replace.enabled'='true')")() + .insert(numberOfRows)() + .sql("prep.rtas")(t => s"CREATE OR REPLACE TABLE $t USING $dataSource $partitionClause " + + s"TBLPROPERTIES ('write.format.default'='$format') AS SELECT * FROM $t")() + // Iceberg documents CREATE OR REPLACE ... AS SELECT as ATOMIC on a SparkCatalog, so the client + // should observe a consistent table afterward with no manual refresh. On the embedded catalog it + // does; on the OpenHouse REST-backed catalog the client can retain a stale metadata pointer across + // the replace (surfacing downstream as a 400 "incorrect version" or "table not found after + // refresh"). REFRESH re-reads the committed pointer so the suite is robust to that catalog + // divergence; the divergence itself is filed as a product bug (see Remote Test Findings). + .sql("prep.rtas.refresh")(t => s"REFRESH TABLE $t")() + + // RTAS prep on a MERGE-ON-READ table (over-prune miss #1): the replace re-specifies the MoR delete/ + // update/merge modes, so downstream mutation ops exercise the MoR write path on a replace-lineage + // table. Non-vacuous per the appraisal — replace + MoR is a distinct combination. + protected def morPropsFmt(format: String) = s"'write.format.default'='$format', 'format-version'='2', " + + "'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read', 'write.merge.mode'='merge-on-read'" + protected val morProps = morPropsFmt("parquet") + + def createAndSeedRtasMor(partitionClause: String, numberOfRows: Int, format: String = "parquet"): TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource $partitionClause " + + s"TBLPROPERTIES (${morPropsFmt(format)}, 'replace.enabled'='true')")() + .insert(numberOfRows)() + .sql("prep.rtasMor")(t => s"CREATE OR REPLACE TABLE $t USING $dataSource $partitionClause " + + s"TBLPROPERTIES (${morPropsFmt(format)}) AS SELECT * FROM $t")() + // See createAndSeedRtas: REFRESH after the atomic replace guards the shared suite against the + // OpenHouse catalog's stale-pointer divergence (filed as a product bug). + .sql("prep.rtasMor.refresh")(t => s"REFRESH TABLE $t")() + + + // ── hoisted shared helpers (used across domain traits) ── + protected def coreTwoSnapshots(fmt: String): TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')")() + .insert(3)() + .sql("insertMore")(table => s"INSERT INTO $table VALUES " + + s"(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")() + // No-arg overload (parquet) keeps the many existing single-format call sites unchanged. + protected def coreTwoSnapshots: TableTest[CoreTable.type] = coreTwoSnapshots("parquet") + + // Snapshots in ancestry order (root first), following the parent_id chain — deterministic even + // if two commits happen to share a committed_at millisecond (which `ORDER BY committed_at` is not). + protected def snapshotIds(spark: SparkSession, table: String): Seq[Long] = { + val rows = spark.sql(s"SELECT snapshot_id, parent_id FROM $table.snapshots").collect().toSeq + val ids = rows.map(_.getLong(0)).toSet + val childByParent = rows.collect { case r if !r.isNullAt(1) => r.getLong(1) -> r.getLong(0) }.toMap + val root = rows.collectFirst { case r if r.isNullAt(1) || !ids.contains(r.getLong(1)) => r.getLong(0) }.get + val order = scala.collection.mutable.ListBuffer(root) + var cur = root + while (childByParent.contains(cur)) { cur = childByParent(cur); order += cur } + order.toList + } + + protected def catalogRelative(table: String): String = table.stripPrefix("openhouse.") + + protected def coreRow(long: Long, tag: String): String = + s"(CAST($long AS BIGINT), ${long.toInt}, '$tag', ${long}.5, false, '2024-01-01-00')" + + protected val L = CoreTable.long0.columnName + + // The Spark datasource short-name for `CREATE TABLE ... USING `. Defaults to "iceberg" — the + // Apache Iceberg DataSourceRegister short-name that OSS OpenHouse registers and that every OpenHouse + // itest uses. A downstream environment whose shaded runtime relocates the Iceberg datasource to a + // different short-name (for example to let multiple Iceberg libraries coexist on one classpath) would + // find that `USING iceberg` does not resolve there. This is a plain `var` — not a runtime knob — that an + // environment adapter overrides purely in code (for example `Scenarios.dataSource = "openhouse"`) once, + // before it builds `Plan.cases`. The emitted SQL is otherwise byte-identical across environments. + var dataSource: String = "iceberg" + + // "should be format-independent" is a hypothesis this harness must verify, not assume (see G8/G10, and + // the fork carries patched ORC paths). Only table-LESS ops (no CREATE) have no format axis. + protected val seedFmtTL = new ThreadLocal[String]() + def seedFmt: String = Option(seedFmtTL.get).getOrElse("parquet") + def withSeedFmt[A](fmt: String)(body: => A): A = { + seedFmtTL.set(fmt); try body finally seedFmtTL.remove() + } + protected def coreCreateParquet(table: String): String = + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt')" + + protected def undropSeed(ctx: Ctx, name: String): (String, String, String) = { + val table = s"${ctx.namespace}.$name" + val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) + ctx.spark.sql(s"DROP TABLE IF EXISTS $table") + ctx.spark.sql(coreCreateParquet(table)) + ctx.spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") + (table, db, tbl) + } + + protected def softDeleteRestore(ctx: Ctx, db: String, tbl: String): Unit = { + assert(HtsAdmin.softDelete(db, tbl)._1 / 100 == 2, s"soft-delete $db.$tbl failed") + val ms = HtsAdmin.softDeletedAtMs(db, tbl).getOrElse(throw new AssertionError(s"no deletedAtMs for $db.$tbl")) + assert(HtsAdmin.restore(db, tbl, ms)._1 / 100 == 2, s"restore $db.$tbl failed") + } + + protected def tableProps(spark: SparkSession, table: String): Map[String, String] = + spark.sql(s"SHOW TBLPROPERTIES $table").collect().toSeq.map(r => r.getString(0) -> r.getString(1)).toMap + + protected val extraColInsert9 = "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01', 42)" + protected val extraColInsert10 = "(CAST(10 AS BIGINT), 10, 'row-10', 10.5, true, '2024-01-10-01', 43)" + + protected def rtasPrep: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("enableReplace")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('replace.enabled'='true')")() + + protected def countOf(spark: SparkSession, sql: String): String = + spark.sql(sql).collect()(0).getLong(0).toString + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala new file mode 100644 index 000000000..915425f3e --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala @@ -0,0 +1,566 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +trait SurfaceScenarios extends ScenarioKit { + import Rows._ + + + // Audit-B regression guard: a rejection message shown to a SQL user must not be a raw stacktrace, + // an [INTERNAL_ERROR], or a bare NPE. (It may still be MEH — jargony — that's tracked separately.) + private def assertReadableMessage(context: String)(e: Throwable): Unit = { + val m = Option(e.getMessage).getOrElse("") + assert(m.nonEmpty, s"$context: empty error message (worst possible readability)") + assert(!m.contains("[INTERNAL_ERROR]"), s"$context: internal error surfaced to the user: ${m.take(160)}") + assert(!m.contains("\n\tat ") && !m.contains("\tat java."), s"$context: stacktrace frames in the user-facing message: ${m.take(160)}") + assert(!m.startsWith("java.lang.NullPointerException"), s"$context: bare NPE surfaced: ${m.take(160)}") + } + + val surfaceMsgReadabilityGuard: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.msg.readabilityGuard") { (spark, table) => + assertReadableMessage("dropColumn")( + Check.intercept[Exception](spark.sql(s"ALTER TABLE $table DROP COLUMN ${Core.int0.columnName}"))) + assertReadableMessage("reservedProp")( + Check.intercept[Exception](spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('openhouse.tableUUID'='x')"))) + assertReadableMessage("rtasDisabled")( + Check.intercept[Exception](spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table"))) + assertReadableMessage("createNamespace")( + Check.intercept[Exception](spark.sql("CREATE NAMESPACE openhouse.nope_ns"))) + }() + + // ── G8 legs: the other main-affecting DDLs leak from a branch to main ──────────────────────── + val surfaceBranchLeakSetProps: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("branch.leak.setProps") { (spark, table) => + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") + spark.sql(s"ALTER TABLE $table CREATE BRANCH lb2") + spark.conf.set("spark.wap.branch", "lb2") + try spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('user.leaked'='yes')") + finally spark.conf.unset("spark.wap.branch") + assert(tableProps(spark, table).get("user.leaked").contains("yes"), + "G8 appears FIXED for SET TBLPROPERTIES — props no longer leak from branch to main; update AUDIT-FINDINGS G8") + }() + + val surfaceBranchLeakWriteOrdered: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("branch.leak.writeOrderedBy") { (spark, table) => + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") + spark.sql(s"ALTER TABLE $table CREATE BRANCH lb3") + spark.conf.set("spark.wap.branch", "lb3") + try spark.sql(s"ALTER TABLE $table WRITE ORDERED BY ${Core.long0.columnName}") + finally spark.conf.unset("spark.wap.branch") + assert(tableProps(spark, table).get("write.distribution-mode").contains("range"), + "G8 appears FIXED for WRITE ORDERED BY — sort order no longer leaks from branch to main; update AUDIT-FINDINGS G8") + }() + + // ── G4 pin: toggling WAP off while staged snapshots exist is NOT guarded ───────────────────── + val surfaceWapToggleNoGuard: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("branch.wapToggle.noGuard") { (spark, table) => + spark.conf.set("spark.wap.id", "w9") + try spark.sql(s"INSERT INTO $table VALUES (CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + finally spark.conf.unset("spark.wap.id") + val staged = countOf(spark, s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'w9'") + assert(staged == "1", s"staging failed: $staged staged snapshots") + // G4 pin: the toggle is ACCEPTED with a staged snapshot outstanding (no guard exists). + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='false')") + val stagedAfter = countOf(spark, s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'w9'") + println(s"DIAG wapToggle: stagedAfterToggle=$stagedAfter") + }() + + // ── WAP negatives (B2 follow-ups) ──────────────────────────────────────────────────────────── + val surfaceWapDoubleCherrypick: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("wap.neg.doubleCherrypick") { (spark, table) => + spark.conf.set("spark.wap.id", "w1") + try spark.sql(s"INSERT INTO $table VALUES (CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + finally spark.conf.unset("spark.wap.id") + val sid = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = 'w1'").collect()(0).getLong(0) + spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${sid}L)") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "4", "first publish failed") + val e = Check.intercept[Exception]( + spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${sid}L)")) + println(s"DIAG doubleCherrypick: ${e.getClass.getName} :: ${Option(e.getMessage).getOrElse("").take(180)}") + assert(Option(e.getMessage).exists(m => m.toLowerCase.contains("duplicate") || m.toLowerCase.contains("already")), + s"double cherry-pick should be rejected as a duplicate WAP commit: ${e.getMessage.take(180)}") + }() + + val surfaceWapExpireRefTarget: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("wap.neg.expireRefTarget") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH eb2") + val headId = spark.sql(s"SELECT snapshot_id FROM $table.refs WHERE name = 'eb2'").collect()(0).getLong(0) + val e = Check.intercept[Exception](spark.sql( + s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', snapshot_ids => ARRAY(${headId}L))")) + println(s"DIAG expireRefTarget: ${e.getClass.getName} :: ${Option(e.getMessage).getOrElse("").take(180)}") + }() + + // ── Branch lifecycle tail: fast_forward IS the merge; replace branch ──────────────────────── + val surfaceBranchFastForwardMerge: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("branch.fastForward.merge") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH fb") + spark.sql(s"INSERT INTO $table.branch_fb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + spark.sql(s"INSERT INTO $table.branch_fb VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "main advanced unexpectedly") + spark.sql(s"CALL openhouse.system.fast_forward('${catalogRelative(table)}', 'main', 'fb')") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "5", + "fast_forward must merge the branch into main (main == branch head)") + }() + + val surfaceBranchFastForwardDivergent: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("branch.fastForward.divergent") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH db") + spark.sql(s"INSERT INTO $table.branch_db VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + spark.sql(s"INSERT INTO $table VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") // diverge main + val e = Check.intercept[Exception]( + spark.sql(s"CALL openhouse.system.fast_forward('${catalogRelative(table)}', 'main', 'db')")) + println(s"DIAG ffDivergent: ${e.getClass.getName} :: ${Option(e.getMessage).getOrElse("").take(180)}") + assert(Option(e.getMessage).exists(m => m.toLowerCase.contains("ancestor") || m.toLowerCase.contains("fast-forward")), + s"divergent fast_forward should be rejected with an ancestry error: ${e.getMessage.take(180)}") + }() + + val surfaceBranchReplaceBranch: TableTest[CoreTable.type] = + coreTwoSnapshots.step("branch.replaceBranch") { (spark, table) => + val snaps = snapshotIds(spark, table) + spark.sql(s"ALTER TABLE $table CREATE BRANCH rb2") + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'rb2'") == "5", "branch at head") + spark.sql(s"ALTER TABLE $table REPLACE BRANCH rb2 AS OF VERSION ${snaps.head}") + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'rb2'") == "3", + "REPLACE BRANCH must retarget the ref to the older snapshot") + }() + + // ── Streaming (structured streaming read + write) ──────────────────────────────────────────── + val surfaceStreamRead: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.stream.read") { (spark, table) => + val ckpt = java.nio.file.Files.createTempDirectory("ck-read").toString + val sink = s"memsink_${System.nanoTime}" + val q = spark.readStream.table(table) + .writeStream.format("memory").queryName(sink) + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", ckpt) + .start() + assert(q.awaitTermination(120000), "streaming read did not finish in 120s") + assert(countOf(spark, s"SELECT count(*) FROM $sink") == "3", + "streaming read must deliver the seeded rows") + }() + + val surfaceStreamWrite: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.stream.write") { (spark, table) => + import spark.implicits._ + implicit val sqlc: org.apache.spark.sql.SQLContext = spark.sqlContext + val ms = org.apache.spark.sql.execution.streaming.MemoryStream[Long] + ms.addData(100L, 101L) + val df = ms.toDF().selectExpr( + s"value AS ${Core.long0.columnName}", + s"CAST(value AS INT) AS ${Core.int0.columnName}", + s"concat('row-', value) AS ${Core.string0.columnName}", + s"CAST(value AS DOUBLE) AS ${Core.double0.columnName}", + s"true AS ${Core.boolean0.columnName}", + s"'2024-01-01-00' AS ${Core.datePartition.columnName}") + val ckpt = java.nio.file.Files.createTempDirectory("ck-write").toString + val q = df.writeStream.format("iceberg").outputMode("append") + .option("checkpointLocation", ckpt) + .toTable(table) + q.processAllAvailable() + q.stop() + assert(countOf(spark, s"SELECT count(*) FROM $table") == "5", + "streaming write must append the 2 streamed rows") + }() + + // ── CDC: changelog view procedure ───────────────────────────────────────────────────────────── + val surfaceCdcChangelogView: TableTest[CoreTable.type] = + coreTwoSnapshots.step("surface.cdc.changelogView") { (spark, table) => + val viewName = spark.sql( + s"CALL openhouse.system.create_changelog_view(table => '${catalogRelative(table)}')").collect()(0).getString(0) + val changes = spark.sql(s"SELECT count(*) FROM $viewName").collect()(0).getLong(0) + assert(changes == 5, s"changelog must contain one INSERT change per seeded row: $changes") + val types = spark.sql(s"SELECT DISTINCT _change_type FROM $viewName").collect().toSeq.map(_.getString(0)).toSet + assert(types == Set("INSERT"), s"append-only history must yield INSERT changes only: $types") + }() + + // ── Procedures not yet exercised ───────────────────────────────────────────────────────────── + // Manifest compaction must actually DO ITS JOB — reduce the manifest count — not merely preserve data. + // Five separate appends produce ~5 manifests (one per commit); rewrite_manifests must coalesce them. + val surfaceProcRewriteManifests: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)() + .step("surface.proc.rewriteManifests") { (spark, table) => + (1 to 5).foreach(i => spark.sql(s"INSERT INTO $table VALUES ${coreRow(i, s"r$i")}")) + val before = spark.sql(s"SELECT count(*) FROM $table.manifests").collect()(0).getLong(0) + spark.sql(s"CALL openhouse.system.rewrite_manifests(table => '${catalogRelative(table)}', use_caching => false)") + val after = spark.sql(s"SELECT count(*) FROM $table.manifests").collect()(0).getLong(0) + println(s"DIAG surface.proc.rewriteManifests: manifests before=$before after=$after") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "5", "rewrite_manifests changed the live row set") + assert(before >= 2 && after < before, + s"rewrite_manifests did not COMPACT the manifests (before=$before after=$after) — it should coalesce them") + }() + + val surfaceProcRewritePositionDeletes: TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$seedFmt', 'write.delete.mode'='merge-on-read')")() + .sql("seed(3, one-file)")(t => + s"INSERT INTO $t SELECT /*+ COALESCE(1) */ * FROM (${RowGenerator.valuesClause(Core, 3)}) AS seed")() + .step("surface.proc.rewritePositionDeletes") { (spark, table) => + spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") + assert(countOf(spark, s"SELECT count(*) FROM $table.all_delete_files") == "1", "MoR delete file missing") + spark.sql(s"CALL openhouse.system.rewrite_position_delete_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "2", "rewrite_position_delete_files changed data") + }() + + val surfaceProcPublishChanges: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .sql("enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("surface.proc.publishChanges") { (spark, table) => + spark.conf.set("spark.wap.id", "pw1") + try spark.sql(s"INSERT INTO $table VALUES (CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + finally spark.conf.unset("spark.wap.id") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "staged write must not be visible") + spark.sql(s"CALL openhouse.system.publish_changes(table => '${catalogRelative(table)}', wap_id => 'pw1')") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "4", + "publish_changes (the wap_id publish path beside cherrypick) must publish the staged write") + }() + + val surfaceProcAncestorsOf: TableTest[CoreTable.type] = + coreTwoSnapshots.step("surface.proc.ancestorsOf") { (spark, table) => + val n = spark.sql(s"CALL openhouse.system.ancestors_of(table => '${catalogRelative(table)}')").collect().length + assert(n == 2, s"ancestors_of must list main's full ancestry (2 snapshots): $n") + }() + + val surfaceProcRemoveOrphanReal: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.proc.removeOrphanReal") { (spark, table) => + val dataFile = spark.sql(s"SELECT file_path FROM $table.files LIMIT 1").collect()(0).getString(0).stripPrefix("file:") + val orphan = java.nio.file.Paths.get(dataFile).getParent.resolve("zz_orphan_plant.parquet") + java.nio.file.Files.write(orphan, "not-a-real-parquet".getBytes) + java.nio.file.Files.setLastModifiedTime(orphan, + java.nio.file.attribute.FileTime.fromMillis(1546300800000L)) // 2019-01-01 + spark.sql(s"CALL openhouse.system.remove_orphan_files(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2020-01-01 00:00:00')") + assert(java.nio.file.Files.notExists(orphan), "planted orphan file must be removed") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "live data must survive orphan removal") + }() + + // ── Metadata surface: hidden columns + full metadata-table sweep ───────────────────────────── + val surfaceMetaHiddenColumns: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.meta.hiddenColumns") { (spark, table) => + val rows = spark.sql(s"SELECT _file, _pos, _spec_id, _partition FROM $table").collect().toSeq + assert(rows.size == 3, s"hidden metadata columns must be selectable per row: ${rows.size}") + assert(rows.forall(r => r.getString(0) != null && r.getString(0).nonEmpty), "_file must be populated") + assert(rows.forall(r => r.getLong(1) >= 0), "_pos must be populated") + }() + + val surfaceMetaTableSweep: TableTest[CoreTable.type] = + coreTwoSnapshots.step("surface.meta.tableSweep") { (spark, table) => + val metaTables = Seq("entries", "files", "manifests", "snapshots", "history", "refs", "partitions", + "metadata_log_entries", "data_files", "all_data_files", "all_manifests", "all_entries", "all_files") + metaTables.foreach { m => + val n = spark.sql(s"SELECT count(*) FROM $table.`$m`").collect()(0).getLong(0) + assert(n >= 0, s"metadata table $m unreadable") // queryability is the assertion; count is a bonus + } + assert(countOf(spark, s"SELECT count(*) FROM $table.snapshots") == "2", "snapshots count sanity") + }() + + val surfaceMetaPositionDeletes: TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$seedFmt', 'write.delete.mode'='merge-on-read')")() + .sql("seed(3, one-file)")(t => + s"INSERT INTO $t SELECT /*+ COALESCE(1) */ * FROM (${RowGenerator.valuesClause(Core, 3)}) AS seed")() + .step("surface.meta.positionDeletes") { (spark, table) => + spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") + assert(countOf(spark, s"SELECT count(*) FROM $table.position_deletes") == "1", + "position_deletes metadata table must expose the position delete") + }() + + // ── Concurrency: invariant-based (no torn state; failures must be typed) ───────────────────── + private def runConcurrently(fs: Seq[() => Unit]): Seq[Throwable] = { + val errors = new java.util.concurrent.ConcurrentLinkedQueue[Throwable]() + val threads = fs.map(f => new Thread(() => try f() catch { case t: Throwable => errors.add(t) })) + threads.foreach(_.start()) + threads.foreach(_.join(180000)) + errors.toArray(Array.empty[Throwable]).toSeq + } + + private def isTypedCommitConflict(t: Throwable): Boolean = + Exceptions.causeChain(t).exists { c => + val n = c.getClass.getName + n.contains("CommitFailed") || n.contains("CommitStateUnknown") || n.contains("Validation") || + n.contains("BadRequest") || n.contains("WebClientResponse") + } + + val surfaceConcAppendAppend: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.conc.appendAppend") { (spark, table) => + val failures = new java.util.concurrent.atomic.AtomicInteger(0) + def writer(base: Int): () => Unit = () => (0 until 3).foreach { i => + try spark.sql(s"INSERT INTO $table VALUES (CAST(${base + i} AS BIGINT), ${base + i}, 'row-c', 1.5, true, '2024-01-09-01')") + catch { case t: Throwable => + assert(isTypedCommitConflict(t), s"concurrent append failed with an UNTYPED error: ${t.getClass.getName} ${Option(t.getMessage).getOrElse("").take(160)}") + failures.incrementAndGet() + } + } + val errs = runConcurrently(Seq(writer(100), writer(200))) + assert(errs.isEmpty, s"writer thread died outside the insert loop: ${errs.headOption.map(_.toString)}") + val expected = 3 + 6 - failures.get + assert(countOf(spark, s"SELECT count(*) FROM $table") == expected.toString, + s"row count must equal successful appends (3 seed + ${6 - failures.get} landed)") + println(s"DIAG conc.appendAppend: ${failures.get}/6 inserts hit a typed commit conflict") + }() + + val surfaceConcUpdateUpdate: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.conc.updateUpdate") { (spark, table) => + val col = Core.string0.columnName + def updater(v: String): () => Unit = () => + try spark.sql(s"UPDATE $table SET $col = '$v' WHERE ${Core.long0.columnName} = 2") + catch { case t: Throwable => + assert(isTypedCommitConflict(t), s"concurrent update failed with an UNTYPED error: ${t.getClass.getName} ${Option(t.getMessage).getOrElse("").take(160)}") } + val errs = runConcurrently(Seq(updater("AAA"), updater("BBB"))) + assert(errs.isEmpty, s"updater thread died with a non-conflict error: ${errs.headOption.map(_.toString)}") + val v = spark.sql(s"SELECT $col FROM $table WHERE ${Core.long0.columnName} = 2").collect()(0).getString(0) + assert(v == "AAA" || v == "BBB" || v == "row-2", s"row must hold one writer's value or the original, not torn state: $v") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "row count must be unchanged") + }() + + val surfaceConcRtasVsAppend: TableTest[CoreTable.type] = + rtasPrep.step("surface.conc.rtasVsAppend") { (spark, table) => + def rtas(): Unit = + try spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") + catch { case t: Throwable => assert(isTypedCommitConflict(t), s"RTAS race failed UNTYPED: ${t.getClass.getName}") } + def append(): Unit = + try spark.sql(s"INSERT INTO $table VALUES (CAST(30 AS BIGINT), 30, 'row-30', 30.5, true, '2024-01-09-01')") + catch { case t: Throwable => assert(isTypedCommitConflict(t), s"append race failed UNTYPED: ${t.getClass.getName}") } + val errs = runConcurrently(Seq(() => rtas(), () => append())) + assert(errs.isEmpty, s"racing thread died with a non-conflict error: ${errs.headOption.map(_.toString)}") + spark.sql(s"REFRESH TABLE $table") + val n = countOf(spark, s"SELECT count(*) FROM $table").toLong + assert(n == 2 || n == 3, s"RTAS-vs-append must settle to a consistent state (2 or 3 rows), got $n") + println(s"DIAG conc.rtasVsAppend: settled at $n rows") + }() + + // ── Schema-evolution edges ─────────────────────────────────────────────────────────────────── + val surfaceSchemaRelaxNotNull: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.schema.relaxNotNull") { (spark, table) => + val side = s"${table}_nn" + spark.sql(s"DROP TABLE IF EXISTS $side") + try { + spark.sql(s"CREATE TABLE $side (id BIGINT, req INT NOT NULL) USING $dataSource") + spark.sql(s"ALTER TABLE $side ALTER COLUMN req DROP NOT NULL") + spark.sql(s"INSERT INTO $side VALUES (CAST(1 AS BIGINT), NULL)") + assert(spark.sql(s"SELECT count(*) FROM $side WHERE req IS NULL").collect()(0).getLong(0) == 1, + "relaxing NOT NULL must allow null writes (the inverse of the pinned-rejected tighten)") + } finally spark.sql(s"DROP TABLE IF EXISTS $side") + }() + + val surfaceSchemaDecimalWiden: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.schema.decimalWiden") { (spark, table) => + val side = s"${table}_dec" + spark.sql(s"DROP TABLE IF EXISTS $side") + try { + spark.sql(s"CREATE TABLE $side (id BIGINT, dec DECIMAL(10,2)) USING $dataSource") + spark.sql(s"INSERT INTO $side VALUES (CAST(1 AS BIGINT), CAST(12345678.99 AS DECIMAL(10,2)))") + spark.sql(s"ALTER TABLE $side ALTER COLUMN dec TYPE DECIMAL(12,2)") + spark.sql(s"INSERT INTO $side VALUES (CAST(2 AS BIGINT), CAST(1234567890.99 AS DECIMAL(12,2)))") + assert(spark.sql(s"SELECT count(*) FROM $side").collect()(0).getLong(0) == 2, + "decimal precision widen must keep old data readable and accept wider values") + } finally spark.sql(s"DROP TABLE IF EXISTS $side") + }() + + val surfaceSchemaNestedAddField: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.schema.nestedAddField") { (spark, table) => + val side = s"${table}_nst" + spark.sql(s"DROP TABLE IF EXISTS $side") + try { + spark.sql(s"CREATE TABLE $side (id BIGINT, s STRUCT) USING $dataSource") + spark.sql(s"INSERT INTO $side VALUES (CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") + spark.sql(s"ALTER TABLE $side ADD COLUMN s.w INT") + assert(spark.sql(s"SELECT count(*) FROM $side WHERE s.w IS NULL").collect()(0).getLong(0) == 1, + "adding a nested struct field must null-fill existing rows") + spark.sql(s"INSERT INTO $side VALUES (CAST(2 AS BIGINT), named_struct('x', 2, 'y', 'b', 'w', 9))") + assert(spark.sql(s"SELECT count(*) FROM $side WHERE s.w = 9").collect()(0).getLong(0) == 1, + "the new nested field must be writable") + } finally spark.sql(s"DROP TABLE IF EXISTS $side") + }() + + val surfaceSchemaNestedDropField: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.schema.nestedDropField") { (spark, table) => + val side = s"${table}_nsd" + spark.sql(s"DROP TABLE IF EXISTS $side") + try { + spark.sql(s"CREATE TABLE $side (id BIGINT, s STRUCT) USING $dataSource") + spark.sql(s"INSERT INTO $side VALUES (CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") + val e = Check.intercept[Exception](spark.sql(s"ALTER TABLE $side DROP COLUMN s.x")) + println(s"DIAG nestedDropField: ${e.getClass.getName} :: ${Option(e.getMessage).getOrElse("").take(180)}") + assert(spark.sql(s"SELECT s.x FROM $side").collect()(0).getInt(0) == 1, + "rejected nested drop must leave the field readable") + } finally spark.sql(s"DROP TABLE IF EXISTS $side") + }() + + val surfaceSchemaReorderExisting: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.schema.reorderExisting") { (spark, table) => + spark.sql(s"ALTER TABLE $table ALTER COLUMN ${Core.string0.columnName} FIRST") + val cols = spark.sql(s"SELECT * FROM $table LIMIT 1").columns.toSeq + assert(cols.head == Core.string0.columnName, s"column reorder (FIRST) must change projection order: $cols") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "reorder must not affect data") + }() + + // ── Write-path configs ─────────────────────────────────────────────────────────────────────── + val surfaceWriteDistributionHash: TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource PARTITIONED BY (${Core.datePartition.columnName}) " + + s"TBLPROPERTIES ('write.format.default'='$seedFmt', 'write.distribution-mode'='hash')")() + .insert(3)() + .check("surface.write.distributionHash") { view => + assert(tableProps(view.spark, view.table).get("write.distribution-mode").contains("hash"), "hash mode not honored") + assert(view.after.size == 3, "hash-distributed write failed") + } + + val surfaceWriteTargetFileSize: TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$seedFmt', 'write.target-file-size-bytes'='1048576')")() + .insert(3)() + .check("surface.write.targetFileSize") { view => + assert(tableProps(view.spark, view.table).get("write.target-file-size-bytes").contains("1048576"), "target size not honored") + assert(view.after.size == 3, "write under custom target file size failed") + } + + val surfaceWriteDfToBranch: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.write.dfToBranch") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH wb") + val df = spark.sql(s"SELECT CAST(50 AS BIGINT) AS ${Core.long0.columnName}, 50 AS ${Core.int0.columnName}, " + + s"'row-50' AS ${Core.string0.columnName}, 50.5 AS ${Core.double0.columnName}, " + + s"true AS ${Core.boolean0.columnName}, '2024-01-09-01' AS ${Core.datePartition.columnName}") + df.writeTo(s"$table.branch_wb").append() + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'wb'") == "4", + "DataFrame-API write must land on the branch") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "main must be untouched by the branch DF write") + }() + + // ── Pins: import/migration procedures, views, ANALYZE (expected-unsupported tripwires) ─────── + // The bogus-input probes showed these procedures fail on INPUT (NotFound/NoSuchTable), not on an + // OpenHouse catalog block — so settle register_table with a REAL metadata file: is importing a + // table into the managed catalog (bypassing normal creation) actually possible? + val surfacePinImportProcs: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.pin.importProcs") { (spark, table) => + val metadataFile = spark.sql( + s"SELECT file FROM $table.metadata_log_entries ORDER BY timestamp DESC LIMIT 1").collect()(0).getString(0) + val regOutcome = + try { + spark.sql(s"CALL openhouse.system.register_table(table => 'dbMatrix.zz_reg', metadata_file => '$metadataFile')") + val n = countOf(spark, "SELECT count(*) FROM openhouse.dbMatrix.zz_reg") + spark.sql("DROP TABLE IF EXISTS openhouse.dbMatrix.zz_reg") + s"REGISTERED (readable, $n rows) — import into the managed catalog is NOT blocked" + } catch { case t: Throwable => + s"REJECTED ${t.getClass.getName} :: ${Option(t.getMessage).getOrElse("").take(160)}" } + println(s"DIAG pin.register_table(real): $regOutcome") + val snap = Check.intercept[Exception](spark.sql( + s"CALL openhouse.system.snapshot(source_table => '${catalogRelative(table)}', table => 'dbMatrix.zz_snap')")) + println(s"DIAG pin.snapshot: ${snap.getClass.getName} :: ${Option(snap.getMessage).getOrElse("").take(160)}") + val add = Check.intercept[Exception](spark.sql( + s"CALL openhouse.system.add_files(table => '${catalogRelative(table)}', source_table => '`parquet`.`/tmp/zz_nope_dir`')")) + println(s"DIAG pin.add_files: ${add.getClass.getName} :: ${Option(add.getMessage).getOrElse("").take(160)}") + }() + + val surfacePinViewsAnalyze: TableTest[CoreTable.type] = + TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() + .step("surface.pin.viewsAnalyze") { (spark, table) => + val view = Check.intercept[Exception](spark.sql(s"CREATE VIEW openhouse.dbMatrix.zz_v1 AS SELECT 1 AS one")) + println(s"DIAG pin.createView: ${view.getClass.getName} :: ${Option(view.getMessage).getOrElse("").take(160)}") + val analyze = Check.intercept[Exception](spark.sql(s"ANALYZE TABLE $table COMPUTE STATISTICS")) + println(s"DIAG pin.analyze: ${analyze.getClass.getName} :: ${Option(analyze.getMessage).getOrElse("").take(160)}") + }() + + // Compaction × branch: does rewrite_data_files touch/break branch state, and where does it land + // when spark.wap.branch is set? (Untested cell flagged in the surface appraisal.) + val surfaceMaintCompactWithBranch: TableTest[CoreTable.type] = + coreTwoSnapshots.step("surface.maint.compactWithBranch") { (spark, table) => + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") + spark.sql(s"ALTER TABLE $table CREATE BRANCH cb") + spark.sql(s"INSERT INTO $table.branch_cb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + spark.sql(s"INSERT INTO $table VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + val r = spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('min-input-files', '2'))").collect()(0) + println(s"DIAG compactWithBranch: mainCompaction rewritten=${r.get(0)} added=${r.get(1)}") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "6", "main data preserved by compaction") + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'cb'") == "6", + "branch data preserved and readable after main compaction") + spark.conf.set("spark.wap.branch", "cb") + val confOutcome = try { + val rc = spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}')").collect()(0) + s"RAN (rewritten=${rc.get(0)}, added=${rc.get(1)})" + } catch { case t: Throwable => s"THREW ${t.getClass.getSimpleName} :: ${Option(t.getMessage).getOrElse("").take(140)}" } + finally spark.conf.unset("spark.wap.branch") + println(s"DIAG compactUnderWapConf: $confOutcome") + spark.sql(s"REFRESH TABLE $table") + assert(countOf(spark, s"SELECT count(*) FROM $table") == "6", "main intact after conf-routed compaction attempt") + assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'cb'") == "6", "branch intact after conf-routed compaction attempt") + }() + + val surfaceOps: List[(String, TableTest[CoreTable.type])] = List( + "surface.maint.compactWithBranch" -> surfaceMaintCompactWithBranch, + "surface.msg.readabilityGuard" -> surfaceMsgReadabilityGuard, + "branch.leak.setProps" -> surfaceBranchLeakSetProps, + "branch.leak.writeOrderedBy" -> surfaceBranchLeakWriteOrdered, + "branch.wapToggle.noGuard" -> surfaceWapToggleNoGuard, + "wap.neg.doubleCherrypick" -> surfaceWapDoubleCherrypick, + "wap.neg.expireRefTarget" -> surfaceWapExpireRefTarget, + "branch.fastForward.merge" -> surfaceBranchFastForwardMerge, + "branch.fastForward.divergent" -> surfaceBranchFastForwardDivergent, + "branch.replaceBranch" -> surfaceBranchReplaceBranch, + "surface.stream.read" -> surfaceStreamRead, + "surface.stream.write" -> surfaceStreamWrite, + "surface.cdc.changelogView" -> surfaceCdcChangelogView, + "surface.proc.rewriteManifests" -> surfaceProcRewriteManifests, + "surface.proc.rewritePositionDeletes" -> surfaceProcRewritePositionDeletes, + "surface.proc.publishChanges" -> surfaceProcPublishChanges, + "surface.proc.ancestorsOf" -> surfaceProcAncestorsOf, + "surface.proc.removeOrphanReal" -> surfaceProcRemoveOrphanReal, + "surface.meta.hiddenColumns" -> surfaceMetaHiddenColumns, + "surface.meta.tableSweep" -> surfaceMetaTableSweep, + "surface.meta.positionDeletes" -> surfaceMetaPositionDeletes, + "surface.conc.appendAppend" -> surfaceConcAppendAppend, + "surface.conc.updateUpdate" -> surfaceConcUpdateUpdate, + "surface.conc.rtasVsAppend" -> surfaceConcRtasVsAppend, + "surface.schema.relaxNotNull" -> surfaceSchemaRelaxNotNull, + "surface.schema.decimalWiden" -> surfaceSchemaDecimalWiden, + "surface.schema.nestedAddField" -> surfaceSchemaNestedAddField, + "surface.schema.nestedDropField" -> surfaceSchemaNestedDropField, + "surface.schema.reorderExisting" -> surfaceSchemaReorderExisting, + "surface.write.distributionHash" -> surfaceWriteDistributionHash, + "surface.write.targetFileSize" -> surfaceWriteTargetFileSize, + "surface.write.dfToBranch" -> surfaceWriteDfToBranch, + "surface.pin.importProcs" -> surfacePinImportProcs, + "surface.pin.viewsAnalyze" -> surfacePinViewsAnalyze + ) + + // ═══ Hazard demonstrations H1-H8 (MODALITY-RECON.md; gates cleared per FEATURE-ANALYSIS-PLAN) ══ + // Each was PREDICTED by the state-flow model, verified in code/bytecode, and is demonstrated + // live here. Characterizations flip loudly if the product fixes the hazard. + + // H1 — streaming checkpoint × expiration (G11's streaming twin). Three acts: + // (1) stream + checkpoint; (2) CONTROL: plain restart picks up new rows (restart mechanics fine); + // (3) expire past the checkpointed offset → restart is BRICKED with the typed error. + +} diff --git a/settings.gradle b/settings.gradle index 9acc07d67..4f9b29e12 100644 --- a/settings.gradle +++ b/settings.gradle @@ -40,6 +40,7 @@ include ':integrations:spark:spark-3.1:openhouse-spark-runtime' include ':integrations:spark:spark-3.1:openhouse-spark-itest' include ':integrations:spark:spark-3.5:openhouse-spark-runtime' include ':integrations:spark:spark-3.5:openhouse-spark-itest' +include ':integrations:spark:delta-harness' include ':iceberg:openhouse:htscatalog' include ':iceberg:openhouse:internalcatalog' @@ -77,4 +78,5 @@ project(':integrations:java:iceberg-1.5:openhouse-java-runtime').name = 'openhou project(':integrations:java:iceberg-1.5:openhouse-java-itest').name = 'openhouse-java-iceberg-1.5-itest' project(':integrations:spark:spark-3.5:openhouse-spark-runtime').name = 'openhouse-spark-3.5-runtime_2.12' project(':integrations:spark:spark-3.5:openhouse-spark-itest').name = 'openhouse-spark-3.5-itest' +project(':integrations:spark:delta-harness').name = 'openhouse-spark-delta-harness_2.12' project(':tables-test-fixtures:tables-test-fixtures-iceberg-1.5').name = 'tables-test-fixtures-iceberg-1.5_2.12' From ce8bf891fc154412aadbfecd0215bedac50c3cba Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Thu, 13 Aug 2026 12:55:21 -0700 Subject: [PATCH 02/24] delta-harness: document the testing matrix Adds TESTING-MATRIX.md, a living reference that explains the harness as a cross product of independent axes (operation family, data file format, partitioning, write mode, schema, preparation lineage, and reference routing). Documents how a case id reads, the CoreTable/NestedTypesTable/TypesTable schemas, the table layouts, the preparation lineages, and each operation family including the DDL sub-families. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../spark/delta-harness/TESTING-MATRIX.md | 236 ++++++++++++++++++ 1 file changed, 236 insertions(+) create mode 100644 integrations/spark/delta-harness/TESTING-MATRIX.md diff --git a/integrations/spark/delta-harness/TESTING-MATRIX.md b/integrations/spark/delta-harness/TESTING-MATRIX.md new file mode 100644 index 000000000..4713c16bf --- /dev/null +++ b/integrations/spark/delta-harness/TESTING-MATRIX.md @@ -0,0 +1,236 @@ +# Delta-harness testing matrix + +This document describes how the harness is organized. Every case in the suite is one point in a +cross product of independent axes, so the suite is best understood as a matrix rather than a flat +list of tests. `Plan.cases` (in `Plan.scala`) assembles the matrix by crossing an operation list with +a layout or format axis for each family, and `Scenarios` (in `OpenHouseMatrix.scala`) supplies the +operations by mixing in the per-domain traits. + +## How to read a case id + +A case id has the shape ` @ `, and some families add a preparation prefix. + +| Part | Meaning | +|------|---------| +| `` | The behavior under test, for example `delete.byPredicate`, `ddl.addColumn.single`, or `merge.upsert`. | +| `@ ` | The table shape or environment the operation ran against, for example `partitioned/orc`, `mor-unpartitioned/avro`, `@ parquet`, or `@ embedded`. | +| `prep.rtas:`, `prep.ordered:`, `prep.evolved:`, `branchWap:`, `undrop:` | A prefix that names the preparation lineage the base table was taken through before the operation ran. | + +For example, `prep.ordered:update.byPredicate @ partitioned/parquet` is the `update.byPredicate` +operation, run on a partitioned Parquet table that was created with a `WRITE ORDERED BY` clause. + +## The axes + +The matrix is the product of the following axes. Not every family uses every axis, because some axes +are vacuous for some operations. A branch reference, for instance, never touches file encoding, so +branch-routed families do not multiply across all three file formats. + +| Axis | Values | Notes | +|------|--------|-------| +| Operation | The families listed below | The behavior being asserted. | +| Data file format | `parquet`, `orc`, `avro` | Applied through `write.format.default` and, for table-creating operations, through a per-case seed format. Format independence is treated as a hypothesis the harness verifies, not an assumption. | +| Partitioning | `unpartitioned`, `partitioned` | Partitioned tables partition by the `datepartition` string column. | +| Write mode | copy-on-write, merge-on-read | Merge-on-read tables set `format-version=2` and the merge-on-read delete, update, and merge modes, so mutations write position-delete files instead of rewriting data files. | +| Schema | `CoreTable`, `NestedTypesTable`, `TypesTable` | The column set the operation reads and writes. | +| Preparation lineage | base, ordered, evolved, replace (RTAS), branch, merge-on-read-deleted, undropped | How the base table was created and seeded before the operation ran. | +| Reference routing | main, WAP branch | Whether the operation was applied to the table directly or routed onto a write-audit-publish branch. | + +## Data file formats + +| Format | Explanation | +|--------|-------------| +| `parquet` | The default columnar format and the seed format when no other is set. | +| `orc` | Exercised because the fork carries patched ORC paths, so ORC coverage is not assumed to match Parquet. | +| `avro` | Exercised for the row-oriented write path on the create and merge-on-read families. | + +## Schemas and data types + +The harness pins one representative table per type concern. Column value generators are pure +functions of the row index, so a seed of N rows is reproducible. + +### CoreTable + +`CoreTable` carries one column per common primitive type plus a string date-partition column, and it +is the schema for the DML, DDL, maintenance, branching, and negative families. + +| Column | SQL type | +|--------|----------| +| `foo_col_long` | `bigint` | +| `foo_col_int` | `int` | +| `foo_col_string` | `string` | +| `foo_col_double` | `double` | +| `foo_col_boolean` | `boolean` | +| `datepartition` | `string` | + +### NestedTypesTable + +`NestedTypesTable` covers complex and nested types, and it is the schema for the nested family. + +| Column | SQL type | +|--------|----------| +| `id` | `bigint` | +| `s` | `struct` | +| `arr` | `array` | +| `m` | `map` | +| `nested` | `struct>` | + +### TypesTable + +`TypesTable` covers type-edge cases such as decimal and binary, and it is the schema for the type +family. + +| Column | SQL type | +|--------|----------| +| `id` | `bigint` | +| `n` | `int` | +| `x` | `double` | +| `dec` | `decimal(10,2)` | +| `str` | `string` | +| `bin` | `binary` | + +## Table layouts + +A layout is a labeled `CREATE TABLE` recipe. The label encodes the partitioning and format so it +reads directly in the case id. + +| Layout family | Labels | Explanation | +|---------------|--------|-------------| +| `layouts` | `{unpartitioned,partitioned}/{parquet,orc,avro}` | The six copy-on-write CoreTable shapes that back the DML, DDL, and negative families. | +| `morLayouts` | `mor-{unpartitioned,partitioned}/{parquet,orc,avro}` | The six merge-on-read CoreTable shapes that back the mutation families. | +| `morVerifyLayouts`, `cowVerifyLayouts` | `mor-verify/{format}`, `cow-verify/{format}` | Single-data-file shapes with `write.distribution-mode=none` so a subset delete is a partial-file match, which makes the physical outcome deterministic for the merge-on-read versus copy-on-write discriminator. | +| `nestedLayouts` | `nested-unpartitioned/{format}` | Unpartitioned shapes on `NestedTypesTable`. | +| `typesLayouts` | `types-unpartitioned/{format}` | Unpartitioned shapes on `TypesTable`. | + +## Preparation lineages + +Preparation determines what the base table has already been through when the operation runs. The +same operation list is reused across lineages so a behavior can be checked on each base. + +| Lineage | Explanation | +|---------|-------------| +| base (`createAndSeed`) | Create under the layout and seed a fixed number of deterministic rows. | +| ordered (`createAndSeedOrdered`) | The base plus `ALTER TABLE ... WRITE ORDERED BY`, so the operation runs on a table with a declared sort order. | +| evolved (`createAndSeedEvolved`) | The base plus an added column, so the operation runs against a schema-evolved table. | +| replace (`createAndSeedRtas`, `createAndSeedRtasMor`) | The base is rebuilt through `CREATE OR REPLACE TABLE ... AS SELECT`, so the operation runs on a replace-lineage table. | +| branch (`createAndSeedOnBranch`) | The seed and the operation are routed onto a write-audit-publish branch, and the case also asserts that main is untouched. | +| merge-on-read-deleted (`createAndSeedMorDeleted`) | The base carries a live position delete, so read and maintenance operations must apply the delete at read time. | +| undropped (`createAndSeedUndropped`) | The base is taken through a real House Table Service soft-delete and restore. These cases run only when the embedded real House Table Service is enabled. | +| single-file (`createAndSeedSingleFile`) | The seed lands all rows in one data file, which is required by the copy-on-write versus merge-on-read physical discriminator. | + +## Operation families + +Each family is an operation list that `Plan.cases` crosses with a layout or format axis. The tables +below name the family and describe what it exercises. Representative operation names are included so +the family is recognizable in the case ids. + +### DML + +| Family | Explanation | +|--------|-------------| +| Reads (`read.projection`, `read.filter`, `format.materialization`) | Read-path and scan behavior, including projection, predicate pushdown, and materialization. | +| Deletes (`delete.byPredicate`, `delete.byInList`, `delete.byInSubquery`, `delete.byPartitionPredicate`, `delete.all`, `delete.truncate`, and more) | Row-level deletes across the full range of predicate shapes, including in-list, correlated and scalar subqueries, null conditions, partition predicates, and whole-table truncation. | +| Updates (`update.byPredicate`, `update.multipleColumns`, `update.byExpression`, `update.movePartition`, and more) | Row-level updates across predicate shapes, multi-column assignments, expression assignments, and partition-moving updates. | +| Merges (`merge.upsert`, `merge.insertNotMatched`, `merge.deleteMatched`, `merge.multipleMatchedClauses`, `merge.resolveByName`, and more) | `MERGE INTO` across matched and not-matched clauses, conditional clauses, upserts, source common table expressions, set operations, and by-name resolution. | +| Inserts and overwrites (`insert.into`, `insert.explicitColumns`, `append.dataFrame`, `insert.overwrite`, `insert.dynamicOverwrite`, `overwrite.dataFrame`) | The append and overwrite write paths through both SQL and the DataFrame API, including dynamic partition overwrite. | + +The mutation subset (`delete.*`, `update.*`, `merge.*`) is reused on the merge-on-read, replace, and +branch lineages, because those lineages are about the mutation write path. + +### DDL + +DDL is split into sub-families so each area of the OpenHouse table surface is exercised on its own. +Every sub-family crosses its operation list with the six copy-on-write layouts unless noted. + +| Sub-family | Operations | Explanation | +|------------|-----------|-------------| +| Schema evolution (`ddlSchemaOperations`) | `ddl.addColumn.single`, `ddl.addColumn.multiple`, `ddl.addColumn.comment`, `ddl.addColumn.position`, `ddl.alterColumn.typeWiden`, `ddl.renameColumn` | Column additions in each position and with comments, safe type widening, and column rename. These exercise how the server validates and applies a schema change. | +| Table properties (`ddlPropsOperations`) | `ddl.props.userRoundTrip`, `ddl.props.reservedOpenhouse`, `ddl.props.formatVersionForced`, `ddl.props.previousVersionsHonored` | User property round-tripping, the handling of reserved OpenHouse properties, forced format version, and honoring previously set versions. | +| Miscellaneous (`ddlMiscOperations`) | `ddl.sortOrder.orderedBy`, `ddl.sortOrder.orderedByMulti`, `ddl.renameTable`, `ddl.renameTable.conflict`, `ddl.ns.createRejected`, `ddl.ns.dropRejected` | Setting a sort order, renaming a table and the name-conflict case, and the namespace create and drop rejections. | +| Policy (`ddlPolicyOperations`) | `ddl.policy.sharing`, `ddl.policy.history`, `ddl.policy.replication`, `ddl.policy.retention`, `ddl.policy.neg.historyMaxAge`, `ddl.policy.neg.historyVersions` | `SET POLICY` for sharing, history, replication, and retention, plus the negative cases where a policy bound is out of range. | +| CTAS and RTAS (`ddlCtasRtasOperations`) | `ddl.ctas`, `ddl.rtas.enabled`, `ddl.rtas.disabled`, `ddl.rtas.replicationConflict` | Create-table-as-select, replace-table-as-select with replace enabled and disabled, and the replace-under-replication conflict. | +| Tagging, ACL, and features (`ddlTagAclFeatureOperations`) | `ddl.colTag`, `ddl.acl.grantUnshared`, `ddl.acl.grantShared`, `ddl.featureFlag.distributionMode`, `ddl.repl.tableTypeImmutable`, `ddl.encryption.active` | Column tagging, ACL grants on shared and unshared tables, the distribution-mode feature flag, replica-table-type immutability, and the encryption-active property. | +| Encryption (`ddlEncryptionOperations`) | `ddl.encryption` | The encryption capability, pinned on Parquet. | + +The schema-evolution operations are also crossed with every layout as a separate `ddlSchema` block, +and there is a DDL-then-consumer battery (`ddlConsumeBattery`) that applies each state-changing DDL +and then runs each consumer to confirm the table still reads and writes. + +### Maintenance + +| Operations | Explanation | +|-----------|-------------| +| `maintenance.expireSnapshots`, `maintenance.rewriteDataFiles`, `maintenance.removeOrphanFiles` | The table-maintenance procedures, including the locked variants, crossed with both file formats. | + +### Merge-on-read verification + +| Family | Explanation | +|--------|-------------| +| `mor.writesDeleteFiles`, `cow.writesNoDeleteFiles` | The physical discriminator that proves merge-on-read wrote a position delete and copy-on-write did not. | +| merge-on-read read (`prep.morRead`), coexistence (`morCoexist`), maintenance fold and meta, hazards, and branch merge | Reads and maintenance over a table that already carries a live position delete, and the survival of position deletes across time travel, rollback, expiration, and branch merges. | + +### Branching and write-audit-publish + +| Family | Explanation | +|--------|-------------| +| `branching` | Branch creation and the basic branch operations. | +| `branchWap:` blocks | The DML catalog routed onto a branch, asserting both the branch delta and that main stays isolated. | +| `branchDdl`, `wapStaged` | The DDL-on-branch axis and the staged-then-publish write-audit-publish flow. | + +### Interactions, surface, and hazards + +| Family | Explanation | +|--------|-------------| +| `interactions` | Cross-feature cases where one feature is exercised in the presence of another. | +| `surface` | The read and write surface, including streaming read and write and the plaintext data pin. | +| `hazards`, `readerWriter` | Reader and writer hazard scenarios, such as a streaming checkpoint crossed with snapshot expiration, change-data-capture over an expired range, and replace-table-as-select wiping column tags. | + +### Nested and type-edge coverage + +| Family | Explanation | +|--------|-------------| +| nested (`nestedOperations` on `NestedTypesTable`) | Operations over struct, array, map, and doubly-nested struct columns. | +| types (`typesOperations` on `TypesTable`) | Operations over type-edge columns such as decimal and binary. | + +### Time travel, restore, and rollback + +| Family | Explanation | +|--------|-------------| +| `timeTravel` | Reads at an earlier snapshot, crossed with both file formats. | +| `restoreRollback` | `RESTORE` and rollback to an earlier snapshot, crossed with both file formats. | + +### Fork behavior pins + +These families pin behaviors specific to the `com.linkedin.iceberg` fork the harness runs against. +They characterize the fork surface at the API and table-property level. + +| Family | Explanation | +|--------|-------------| +| `forkColDefault` | Column-default serialization through `SchemaParser`. | +| `forkPartitionDist` | Partition distribution behavior. | +| `forkDeleteFileReplication`, `forkFileReplicationFactor` | Delete-file replication and the output-file replication factor. | +| `forkSplitSize`, `forkBinPackByLength`, `forkCompactionOrder` | Split size, bin-pack by length, and compaction ordering. | + +### Negatives + +| Operations | Explanation | +|-----------|-------------| +| `negative.nonExistentColumn`, `negative.nonDeterministicDelete`, `negative.nonDeterministicUpdate`, `negative.insertArity`, `negative.mergeConflictingUpdates`, `negative.mergeCardinalityViolation`, `negative.partitionByNonExistent` | Cases that must be rejected. Each asserts that the operation fails, so a silent acceptance is itself a failure. | + +### Control plane + +| Family | Explanation | +|--------|-------------| +| `control`, `undropAdmin`, `undropInteract` | Control-plane cases such as lock and unlock, and the soft-delete, list, restore, and purge lifecycle. The undrop lifecycle cases run only when the embedded real House Table Service is enabled, and are otherwise empty. | + +## How assertions are framed + +Every case asserts a delta against the pre-state it observed, meaning the change in rows or in the +commit count, rather than an absolute row set. Framing assertions as deltas is what lets one +operation hold under any layout, format, and lineage, which is what makes the cross product +meaningful. + +## Known bugs + +A genuine product or upstream bug is tagged in `Plan.knownBugs` by a substring of the case id, along +with a prose explanation. A tagged case is reported as skipped with its reason rather than failing +the suite, which keeps the suite green while keeping the defect visible and documented. From d18c90eb8e9e86222239939190dedf5170fadd6d Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Mon, 24 Aug 2026 16:32:13 -0700 Subject: [PATCH 03/24] test(delta-harness): localize test cases Move each test's preparation, action, and assertions into its scenario file so the complete behavior is readable in one place. Keep reusable preparation recipes while creating a fresh table for every case. Preserve the exact 2,574-case catalog, ordering, and known-bug behavior with regression tests for the catalog fingerprint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- integrations/spark/delta-harness/build.gradle | 10 + .../spark/delta-harness/run-openhouse.sh | 3 +- .../scripts/print-cp.init.gradle | 1 + .../openhouse/BranchWapScenarios.scala | 971 ++++++--- .../harness/openhouse/DmlScenarios.scala | 1859 ++++++++++------- .../main/scala/harness/openhouse/Env.scala | 2 +- .../harness/openhouse/ForkScenarios.scala | 78 +- .../scala/harness/openhouse/Framework.scala | 133 +- .../HazardReaderWriterScenarios.scala | 1023 ++++++--- .../openhouse/InteractionScenarios.scala | 1314 ++++++++---- .../openhouse/MaintControlScenarios.scala | 237 ++- .../harness/openhouse/MorMaintScenarios.scala | 720 +++++-- .../openhouse/NegativeDdlScenarios.scala | 852 ++++---- .../openhouse/NestedTypesScenarios.scala | 458 ++-- .../harness/openhouse/OpenHouseMatrix.scala | 33 +- .../main/scala/harness/openhouse/Plan.scala | 313 +-- .../scala/harness/openhouse/ScenarioKit.scala | 112 +- .../harness/openhouse/SurfaceScenarios.scala | 1518 +++++++++----- .../test/scala/harness/CaseCatalogTest.scala | 41 + .../scala/harness/TablePreparationTest.scala | 20 + 20 files changed, 6274 insertions(+), 3424 deletions(-) create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala diff --git a/integrations/spark/delta-harness/build.gradle b/integrations/spark/delta-harness/build.gradle index 628d62056..cf1da2f16 100644 --- a/integrations/spark/delta-harness/build.gradle +++ b/integrations/spark/delta-harness/build.gradle @@ -43,6 +43,16 @@ dependencies { } // Provides com.linkedin.openhouse.javaclient.* (WebClientResponseWithMessageException, etc.). compileOnly(project(path: ':integrations:spark:spark-3.5:openhouse-spark-3.5-runtime_2.12', configuration: 'shadow')) + + // The catalog regression test builds Plan.cases without starting Spark. Its JVM still loads the + // harness signatures, so the compile-only harness dependencies must be present on the test classpath. + testImplementation("org.apache.spark:spark-sql_2.12:${sparkVersion}") { + exclude group: 'io.netty' + } + testImplementation("com.linkedin.iceberg:iceberg-spark-runtime-3.5_2.12:${icebergVersion}") { + exclude group: 'io.netty' + } + testImplementation(project(path: ':integrations:spark:spark-3.5:openhouse-spark-3.5-runtime_2.12', configuration: 'shadow')) } jar.enabled = true diff --git a/integrations/spark/delta-harness/run-openhouse.sh b/integrations/spark/delta-harness/run-openhouse.sh index 047c9d086..6432e98a5 100755 --- a/integrations/spark/delta-harness/run-openhouse.sh +++ b/integrations/spark/delta-harness/run-openhouse.sh @@ -35,7 +35,8 @@ else echo ">> resolving OpenHouse itest runtime classpath (builds the runtime uber jar + fixtures)" ( cd "$REPO_ROOT" && "$GRADLE" -Dorg.gradle.java.home="$JDK17" -DcpOut="$WORK/oh-cp.txt" \ --init-script "$HERE/scripts/print-cp.init.gradle" \ - :integrations:spark:spark-3.5:openhouse-spark-3.5-itest:printHarnessCp --console=plain ) + :integrations:spark:spark-3.5:openhouse-spark-3.5-itest:printHarnessCp \ + -x CopyGitHooksTask --console=plain ) fi OHCP="$(cat "$WORK/oh-cp.txt")" diff --git a/integrations/spark/delta-harness/scripts/print-cp.init.gradle b/integrations/spark/delta-harness/scripts/print-cp.init.gradle index 5b5bee049..c876c2dc4 100644 --- a/integrations/spark/delta-harness/scripts/print-cp.init.gradle +++ b/integrations/spark/delta-harness/scripts/print-cp.init.gradle @@ -26,6 +26,7 @@ allprojects { dependencies.add('testImplementation', project(':services:housetables')) } tasks.register('printHarnessCp') { + dependsOn configurations.testRuntimeClasspath doLast { def cp = configurations.testRuntimeClasspath.resolve().collect { it.absolutePath } new File(System.getProperty('cpOut')).text = cp.join(':') diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala index 86df19c88..b79d9bd4e 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala @@ -56,293 +56,696 @@ trait BranchWapScenarios extends ScenarioKit { ctx.spark.sql(s"DROP TABLE IF EXISTS $table") } - val undropInteractOps: List[(String, Ctx => Unit)] = List( - "interact.undrop.branchSurvives" -> interactUndropBranchSurvives, - "interact.undrop.timeTravelSurvives" -> interactUndropTimeTravelSurvives, - "interact.undrop.schemaSurvives" -> interactUndropSchemaSurvives - ) - - // ── Branching / WAP (format-agnostic → parquet only; behavior-focused, not matrixed) ───────── - // A CoreTable row literal for branch writes (long,int,string,double,boolean,datepartition). - - // B1(a) direct branch ops (no WAP needed): write to t.branch_b, read it via VERSION AS OF 'b'; - // main stays isolated. - val branchDirectIsolation: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("branch.direct.create")(t => s"ALTER TABLE $t CREATE BRANCH b")() - .step("branch.direct.isolation") { (spark, table) => - spark.sql(s"INSERT INTO $table.branch_b VALUES ${coreRow(99, "branch")}") - val onBranch = spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'b'").collect()(0).getLong(0) - val onMain = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) - assert(onBranch == 4, s"branch b should have 4 rows, got $onBranch") - assert(onMain == 3, s"main should be unchanged at 3, got $onMain") // isolation - }() - - // B1(b) spark.wap.branch conf: with write.wap.enabled, the conf routes BOTH reads and writes to the - // branch transparently; unsetting reverts to main. - val branchWapConfRouting: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("branch.wapconf.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .sql("branch.wapconf.create")(t => s"ALTER TABLE $t CREATE BRANCH wapbr")() - .step("branch.wapConf.routing") { (spark, table) => - spark.conf.set("spark.wap.branch", "wapbr") - val onBranch = + def undropInteractionCases: List[Plan.Case] = + if (HtsAdmin.enabled) { + List( + Plan.Case( + "interact.undrop.branchSurvives", + interactUndropBranchSurvives), + Plan.Case( + "interact.undrop.timeTravelSurvives", + interactUndropTimeTravelSurvives), + Plan.Case( + "interact.undrop.schemaSurvives", + interactUndropSchemaSurvives)) + } else { + Nil + } + + val wapStagedCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + val preparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableWap")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")()) + + List( + preparation.test("wapStaged.insert") { table => + table.spark.conf.set("spark.wap.id", "wS") + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES ${coreRow(99, "staged")}") + } finally { + table.spark.conf.unset("spark.wap.id") + } + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val stagedSnapshotCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'wS'") + .collect()(0) + .getLong(0) + + println( + "DIAG wapStaged.insert: " + + s"mainPreCount=$mainRowCount stagedSnapshots=$stagedSnapshotCount") + assert(mainRowCount == 3, "staged insert changed main before publish") + + val stagedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'wS'") + .collect()(0) + .getLong(0) + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', $stagedSnapshotId)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "publishing the staged insert did not advance main") + }, + preparation.test("wapStaged.overwrite") { table => + table.spark.conf.set("spark.wap.id", "wS") + try { + table.spark.sql( + s"INSERT OVERWRITE ${table.name} VALUES ${coreRow(7, "ow")}") + } finally { + table.spark.conf.unset("spark.wap.id") + } + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val stagedSnapshotCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'wS'") + .collect()(0) + .getLong(0) + + println( + "DIAG wapStaged.overwrite: " + + s"mainPreCount=$mainRowCount stagedSnapshots=$stagedSnapshotCount") + assert(mainRowCount == 3, "staged overwrite changed main before publish") + + val stagedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'wS'") + .collect()(0) + .getLong(0) + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', $stagedSnapshotId)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 1, + "publishing the staged overwrite did not replace main") + }, + preparation.test("wapStaged.delete.bypassesWap") { table => + table.spark.conf.set("spark.wap.id", "wD") + try { + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + } finally { + table.spark.conf.unset("spark.wap.id") + } + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val stagedSnapshotCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'wD'") + .collect()(0) + .getLong(0) + + println( + "DIAG wapStaged.delete.bypassesWap: " + + s"mainAfterStagedDelete=$mainRowCount " + + s"stagedSnapshots=$stagedSnapshotCount") + assert( + mainRowCount == 2 && stagedSnapshotCount == 0, + "staged DELETE should commit directly to main without a WAP snapshot") + }, + preparation.test("wapStaged.merge") { table => + table.spark.conf.set("spark.wap.id", "wS") + try { + table.spark.sql( + s"MERGE INTO ${table.name} " + + "USING (SELECT CAST(99 AS BIGINT) AS key) source " + + s"ON ${table.name}.${Core.long0.columnName} = source.key " + + "WHEN NOT MATCHED THEN INSERT " + + s"(${Core.columnNames.mkString(", ")}) " + + "VALUES (source.key, 9, 'm', 9.5, true, '2024-01-09-01')") + } finally { + table.spark.conf.unset("spark.wap.id") + } + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val stagedSnapshotCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'wS'") + .collect()(0) + .getLong(0) + + println( + "DIAG wapStaged.merge: " + + s"mainPreCount=$mainRowCount stagedSnapshots=$stagedSnapshotCount") + assert(mainRowCount == 3, "staged merge changed main before publish") + + val stagedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'wS'") + .collect()(0) + .getLong(0) + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', $stagedSnapshotId)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "publishing the staged merge did not advance main") + }, + preparation.test("wapStaged.update.valueVisibleOnlyAfterPublish") { table => + table.spark.conf.set("spark.wap.id", "wU") + try { + table.spark.sql( + s"UPDATE ${table.name} " + + s"SET ${Core.string0.columnName} = 'staged-upd' " + + s"WHERE ${Core.long0.columnName} = 1") + } finally { + table.spark.conf.unset("spark.wap.id") + } + val valueBeforePublish = table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + .collect()(0) + .getString(0) + + assert( + valueBeforePublish != "staged-upd", + s"staged update changed main before publish: $valueBeforePublish") + + val stagedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'wU'") + .collect()(0) + .getLong(0) + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', $stagedSnapshotId)") + val valueAfterPublish = table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + .collect()(0) + .getString(0) + + assert( + valueAfterPublish == "staged-upd", + s"published update returned $valueAfterPublish") + }, + preparation.test("wapStaged.twoIdsIndependent") { table => + def stageInsert(wapId: String, key: Int): Unit = { + table.spark.conf.set("spark.wap.id", wapId) + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + coreRow(key, s"s-$wapId")) + } finally { + table.spark.conf.unset("spark.wap.id") + } + } + def snapshotId(wapId: String): Long = + table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + s"WHERE summary['wap.id'] = '$wapId'") + .collect()(0) + .getLong(0) + + stageInsert("wa", 101) + stageInsert("wb", 102) + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 3, + "a staged ID changed main before publish") + + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', ${snapshotId("wa")})") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "publishing wa did not advance main") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 102") + .collect()(0) + .getLong(0) == 0, + "wb published before its cherry-pick") + + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', ${snapshotId("wb")})") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 5, + "publishing wb did not advance main") + }, + preparation.test("wapStaged.expireVsStaged") { table => + table.spark.conf.set("spark.wap.id", "wE") + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES ${coreRow(200, "stg")}") + } finally { + table.spark.conf.unset("spark.wap.id") + } + val stagedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'wE'") + .collect()(0) + .getLong(0) + + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + val survivedExpiration = table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.snapshots " + + s"WHERE snapshot_id = $stagedSnapshotId") + .collect()(0) + .getLong(0) + val publishOutcome = + try { + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', $stagedSnapshotId)") + "published" + } catch { + case NonFatal(exception) => + s"stranded:${Exceptions.root(exception).getClass.getSimpleName}" + } + + println( + "DIAG wapStaged.expireVsStaged: " + + s"stagedSurvivedExpire=$survivedExpiration " + + s"cherrypickAfterExpire=$publishOutcome") + assert( + survivedExpiration == 0 && publishOutcome.startsWith("stranded"), + "expiration should remove and strand the unreferenced staged snapshot") + }) + } + + val branchDdlCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + val preparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableWap")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .sql("createBranch")(table => + s"ALTER TABLE $table CREATE BRANCH bddl")()) + + List( + preparation.test("branchDdl.addColumn.leaksToMain") { table => + table.spark.conf.set("spark.wap.branch", "bddl") + val outcome = + try { + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN br_added int") + "accepted" + } catch { + case NonFatal(exception) => + s"rejected:${Exceptions.root(exception).getClass.getSimpleName}" + } finally { + table.spark.conf.unset("spark.wap.branch") + } + val columnNames = table.spark + .sql(s"DESCRIBE TABLE ${table.name}") + .collect() + .map(_.getString(0).trim) + .toSet + + println( + "DIAG branchDdl.addColumn.leaksToMain: " + + s"branch-routed DDL $outcome") + assert( + columnNames.contains("br_added"), + "ADD COLUMN on a branch should change the table-global schema") + }, + preparation.test("branchDdl.setTblProp.leaksToMain") { table => + table.spark.conf.set("spark.wap.branch", "bddl") + val outcome = + try { + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('user.branchkey'='v1')") + "accepted" + } catch { + case NonFatal(exception) => + s"rejected:${Exceptions.root(exception).getClass.getSimpleName}" + } finally { + table.spark.conf.unset("spark.wap.branch") + } + val properties = table.spark + .sql(s"SHOW TBLPROPERTIES ${table.name}") + .collect() + .map(row => row.getString(0) -> row.getString(1)) + .toMap + + println( + "DIAG branchDdl.setTblProp.leaksToMain: " + + s"branch-routed DDL $outcome") + assert( + properties.get("user.branchkey").contains("v1"), + "SET TBLPROPERTIES on a branch should change table-global properties") + }, + preparation.test("branchDdl.alterColumnComment.leaksToMain") { table => + table.spark.conf.set("spark.wap.branch", "bddl") + val outcome = + try { + table.spark.sql( + s"ALTER TABLE ${table.name} " + + s"ALTER COLUMN ${Core.string0.columnName} COMMENT 'br-comment'") + "accepted" + } catch { + case NonFatal(exception) => + s"rejected:${Exceptions.root(exception).getClass.getSimpleName}" + } finally { + table.spark.conf.unset("spark.wap.branch") + } + val comment = table.spark + .sql(s"DESCRIBE TABLE ${table.name}") + .collect() + .find(_.getString(0).trim == Core.string0.columnName) + .map(_.getString(2)) + .getOrElse("") + + println( + "DIAG branchDdl.alterColumnComment.leaksToMain: " + + s"branch-routed DDL $outcome") + assert( + Option(comment).getOrElse("").contains("br-comment"), + "ALTER COLUMN COMMENT on a branch should change table-global metadata") + }, + preparation.test("branchDdl.dropColumn.rejected") { table => + table.spark.conf.set("spark.wap.branch", "bddl") + val outcome = + try { + table.spark.sql( + s"ALTER TABLE ${table.name} " + + s"DROP COLUMN ${Core.string0.columnName}") + "accepted" + } catch { + case NonFatal(exception) => + s"rejected:${Exceptions.root(exception).getClass.getSimpleName}" + } finally { + table.spark.conf.unset("spark.wap.branch") + } + val columnNames = table.spark + .sql(s"DESCRIBE TABLE ${table.name}") + .collect() + .map(_.getString(0).trim) + .toSet + + println( + "DIAG branchDdl.dropColumn.rejected: " + + s"branch-routed DDL $outcome") + assert( + columnNames.contains(Core.string0.columnName), + "DROP COLUMN should remain rejected while a branch is selected") + }) + } + + val branchingCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + val preparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + + List( + preparation.test("branch.direct.isolation") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH b") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_b VALUES " + + coreRow(99, "branch")) + val branchRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'b'") + .collect()(0) + .getLong(0) + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + branchRowCount == 4, + s"branch b should have 4 rows, got $branchRowCount") + assert( + mainRowCount == 3, + s"main should be unchanged at 3 rows, got $mainRowCount") + }, + preparation.test("branch.wapConf.routing") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH wapbr") + table.spark.conf.set("spark.wap.branch", "wapbr") + val branchRowCount = + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES ${coreRow(99, "wap")}") + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + } finally { + table.spark.conf.unset("spark.wap.branch") + } + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + branchRowCount == 4, + s"branch-routed read should see 4 rows, got $branchRowCount") + assert( + mainRowCount == 3, + s"branch-routed write changed main to $mainRowCount rows") + }, + preparation.test("wap.stagePublish") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.conf.set("spark.wap.id", "w1") try { - spark.sql(s"INSERT INTO $table VALUES ${coreRow(99, "wap")}") // routed to branch - spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) // reads branch - } finally spark.conf.unset("spark.wap.branch") - assert(onBranch == 4, s"on-branch read should see 4, got $onBranch") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "main leaked") - }() - - // B2 WAP stage → publish: a staged write (spark.wap.id) does NOT advance main; cherrypick publishes it. - val wapStagePublish: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("wap.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("wap.stagePublish") { (spark, table) => - spark.conf.set("spark.wap.id", "w1") - try spark.sql(s"INSERT INTO $table VALUES ${coreRow(99, "staged")}") - finally spark.conf.unset("spark.wap.id") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "staged write leaked to main") - val stagedId = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = 'w1'").collect()(0).getLong(0) - spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', $stagedId)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "publish did not advance main") - }() - - // ── WAP mega-axis Stage C — staged-WAP write surface (stage → publish visibility) ──────────── - // The op is written as a STAGED snapshot (spark.wap.id): it must NOT advance main; assert main is - // unchanged pre-publish, then cherrypick_snapshot PUBLISHES it and main reflects it. This is the - // Phase-29 "T2 staged" target. Format-multiplexed by crossFmt (seedFmt-aware create). - private def wapStagedWrite(label: String)(write: String => String)(preRows: Long, postRows: Long): TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql(s"$label.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step(label) { (spark, table) => - spark.conf.set("spark.wap.id", "wS") - try spark.sql(write(table)) finally spark.conf.unset("spark.wap.id") - val mainPre = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) - val stagedCount = spark.sql(s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'wS'").collect()(0).getLong(0) - println(s"DIAG $label: mainPreCount=$mainPre (expected $preRows) stagedSnapshots=$stagedCount") - assert(mainPre == preRows, - s"$label: staged write LEAKED to main pre-publish (main=$mainPre, expected $preRows)") - val stagedId = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = 'wS'").collect()(0).getLong(0) - spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', $stagedId)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == postRows, - s"$label: publish did not reflect the staged write (expected $postRows)") - }() - - val wapStagedOps: List[(String, TableTest[CoreTable.type])] = List( - "wapStaged.insert" -> wapStagedWrite("wapStaged.insert")(t => s"INSERT INTO $t VALUES ${coreRow(99, "staged")}")(3, 4), - "wapStaged.overwrite" -> wapStagedWrite("wapStaged.overwrite")(t => s"INSERT OVERWRITE $t VALUES ${coreRow(7, "ow")}")(3, 1), - // FINDING (WAP1): a staged DELETE is NOT honored by WAP — it commits to MAIN immediately and creates - // NO staged snapshot (main 3→2, zero snapshots tagged wap.id), unlike staged INSERT/OVERWRITE/UPDATE/ - // MERGE which all stage. Observed on parquet+orc; whether this is stock Iceberg or OpenHouse-specific is - // not determined here. A "staged" DELETE therefore silently publishes to main. Pins the observed behavior. - "wapStaged.delete.bypassesWap" -> { - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("wapStaged.delete.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("wapStaged.delete.bypassesWap") { (spark, table) => - spark.conf.set("spark.wap.id", "wD") - try spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") finally spark.conf.unset("spark.wap.id") - val mainPre = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) - val staged = spark.sql(s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'wD'").collect()(0).getLong(0) - println(s"DIAG wapStaged.delete.bypassesWap: mainAfterStagedDelete=$mainPre stagedSnapshots=$staged") - assert(mainPre == 2 && staged == 0, - s"FINDING WAP1: expected staged DELETE to BYPASS WAP (commit to main=2, no staged snapshot); got main=$mainPre staged=$staged — behavior changed, re-audit AUDIT-FINDINGS WAP1") - }() - }, - "wapStaged.merge" -> wapStagedWrite("wapStaged.merge")(t => - s"MERGE INTO $t USING (SELECT CAST(99 AS BIGINT) AS k) s ON $t.${Core.long0.columnName} = s.k " + - s"WHEN NOT MATCHED THEN INSERT (${Core.columnNames.mkString(", ")}) VALUES (s.k, 9, 'm', 9.5, true, '2024-01-09-01')")(3, 4), - // Staged UPDATE: main's value is unchanged pre-publish, changed after publish (count stays 3). - "wapStaged.update.valueVisibleOnlyAfterPublish" -> { - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("wapStaged.update.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("wapStaged.update.valueVisibleOnlyAfterPublish") { (spark, table) => - spark.conf.set("spark.wap.id", "wU") - try spark.sql(s"UPDATE $table SET ${Core.string0.columnName} = 'staged-upd' WHERE ${Core.long0.columnName} = 1") - finally spark.conf.unset("spark.wap.id") - val pre = spark.sql(s"SELECT ${Core.string0.columnName} FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getString(0) - assert(pre != "staged-upd", s"staged UPDATE leaked to main pre-publish: $pre") - val stagedId = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = 'wU'").collect()(0).getLong(0) - spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', $stagedId)") - val post = spark.sql(s"SELECT ${Core.string0.columnName} FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getString(0) - assert(post == "staged-upd", s"publish did not reflect the staged UPDATE: $post") - }() - }, - // C3(a): two concurrent staged ids publish INDEPENDENTLY and in the chosen order. - "wapStaged.twoIdsIndependent" -> { - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("wapStaged.two.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("wapStaged.twoIdsIndependent") { (spark, table) => - def staged(id: String, k: Int): Unit = { - spark.conf.set("spark.wap.id", id) - try spark.sql(s"INSERT INTO $table VALUES ${coreRow(k, s"s-$id")}") finally spark.conf.unset("spark.wap.id") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES ${coreRow(99, "staged")}") + } finally { + table.spark.conf.unset("spark.wap.id") } - staged("wa", 101); staged("wb", 102) - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "either staged id leaked to main") - def idOf(w: String): Long = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = '$w'").collect()(0).getLong(0) - spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${idOf("wa")})") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "publishing wa did not advance main by 1") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 102").collect()(0).getLong(0) == 0, "wb published without being cherrypicked") - spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${idOf("wb")})") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 5, "publishing wb did not advance main to 5") - }() - }, - // C3(b): a staged (unpublished) snapshot is UNREFERENCED — assert expire_snapshots behaviour toward it - // (G11(d): age-based expiration can delete staged WAP snapshots pre-publish). Characterize: after a - // far-future expire, can the staged id still be cherrypicked, or is it stranded? - "wapStaged.expireVsStaged" -> { - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("wapStaged.exp.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("wapStaged.expireVsStaged") { (spark, table) => - spark.conf.set("spark.wap.id", "wE") - try spark.sql(s"INSERT INTO $table VALUES ${coreRow(200, "stg")}") finally spark.conf.unset("spark.wap.id") - val stagedId = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = 'wE'").collect()(0).getLong(0) - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - val survived = spark.sql(s"SELECT count(*) FROM $table.snapshots WHERE snapshot_id = $stagedId").collect()(0).getLong(0) - val pub = try { spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', $stagedId)"); "published" } - catch { case NonFatal(e) => s"stranded:${Exceptions.root(e).getClass.getSimpleName}" } - println(s"DIAG wapStaged.expireVsStaged: stagedSurvivedExpire=$survived cherrypickAfterExpire=$pub") - // Pin the audited hazard (G11 d): unreferenced staged snapshot is expirable -> stranded pre-publish. - assert(survived == 0 && pub.startsWith("stranded"), - s"G11(d): expected the unreferenced staged snapshot to be expired then un-cherrypickable; survived=$survived pub=$pub — re-audit") - }() + val mainBeforePublish = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + assert( + mainBeforePublish == 3, + s"staged write changed main to $mainBeforePublish rows") + + val stagedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w1'") + .collect()(0) + .getLong(0) + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', $stagedSnapshotId)") + val mainAfterPublish = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + mainAfterPublish == 4, + s"publishing the staged write left main at $mainAfterPublish rows") + }, + preparation.test("branch.ddlLeak.addColumn") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH leakbr") + table.spark.conf.set("spark.wap.branch", "leakbr") + try { + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN leaked_col int") + } finally { + table.spark.conf.unset("spark.wap.branch") + } + val mainColumnNames = + table.spark.table(table.name).schema.fields.map(_.name).toSeq + + assert( + mainColumnNames.contains("leaked_col"), + "ADD COLUMN on a branch should change the table-global schema") + }, + preparation.test("branch.dml.updateDelete") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH dmlbr") + table.spark.conf.set("spark.wap.branch", "dmlbr") + try { + table.spark.sql( + s"UPDATE ${table.name} " + + s"SET ${Core.string0.columnName} = 'br-upd' " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + s"DELETE FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 2") + } finally { + table.spark.conf.unset("spark.wap.branch") + } + val branchRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'dmlbr'") + .collect()(0) + .getLong(0) + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val branchValue = table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + "VERSION AS OF 'dmlbr' " + + s"WHERE ${Core.long0.columnName} = 1") + .collect()(0) + .getString(0) + + assert( + branchRowCount == 2, + s"branch should have 2 rows after delete, got $branchRowCount") + assert( + mainRowCount == 3, + s"branch DML changed main to $mainRowCount rows") + assert( + branchValue == "br-upd", + s"branch update returned $branchValue") + }, + preparation.test("branch.lifecycle.tag") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE TAG mytag") + val tagCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.refs " + + "WHERE name = 'mytag' AND type = 'TAG'") + .collect()(0) + .getLong(0) + + assert(tagCount == 1, "CREATE TAG did not create the tag ref") + }, + preparation.test("branch.lifecycle.dropBranch") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH tmpbr") + val branchCountBeforeDrop = table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.refs " + + "WHERE name = 'tmpbr'") + .collect()(0) + .getLong(0) + assert( + branchCountBeforeDrop == 1, + "CREATE BRANCH did not create the branch ref") + + table.spark.sql( + s"ALTER TABLE ${table.name} DROP BRANCH tmpbr") + val branchCountAfterDrop = table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.refs " + + "WHERE name = 'tmpbr'") + .collect()(0) + .getLong(0) + + assert( + branchCountAfterDrop == 0, + "DROP BRANCH did not remove the branch ref") + }, + preparation.test("branch.neg.wapIdAndBranch") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH nb") + table.spark.conf.set("spark.wap.id", "w1") + table.spark.conf.set("spark.wap.branch", "nb") + try { + val exception = Check.intercept[ValidationException]( + table.spark.sql( + s"INSERT INTO ${table.name} VALUES ${coreRow(99, "x")}")) + assert( + exception.getMessage.contains("Cannot set both WAP ID and branch"), + s"unexpected validation message: ${exception.getMessage.take(140)}") + } finally { + table.spark.conf.unset("spark.wap.id") + table.spark.conf.unset("spark.wap.branch") + } + }, + preparation.test("branch.neg.insertNonexistentBranch") { table => + val exception = Check.intercept[ValidationException]( + table.spark.sql( + s"INSERT INTO ${table.name}.branch_nope VALUES " + + coreRow(99, "x"))) + + assert( + exception.getMessage.contains("does not exist"), + s"unexpected validation message: ${exception.getMessage.take(140)}") + }) } - ) - - // B3 DDL-on-branch is NOT isolated — characterizes the leak (finding): schema/props/sortOrder are - // table-global; ADD COLUMN while "on branch" mutates MAIN's schema, with no guard. - val branchDdlLeakAddColumn: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("branch.leak.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .sql("branch.leak.create")(t => s"ALTER TABLE $t CREATE BRANCH leakbr")() - .step("branch.ddlLeak.addColumn") { (spark, table) => - spark.conf.set("spark.wap.branch", "leakbr") - try spark.sql(s"ALTER TABLE $table ADD COLUMN leaked_col int") - finally spark.conf.unset("spark.wap.branch") - val mainCols = spark.table(table).schema.fields.map(_.name).toSeq - assert(mainCols.contains("leaked_col"), - s"characterizing the leak: ADD COLUMN on a branch mutated MAIN's schema — expected leaked_col in $mainCols") - }() - - // B4 representative branch DML (update + delete on a branch), isolated from main. - val branchDmlUpdateDelete: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("branch.dml.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .sql("branch.dml.create")(t => s"ALTER TABLE $t CREATE BRANCH dmlbr")() - .step("branch.dml.updateDelete") { (spark, table) => - spark.conf.set("spark.wap.branch", "dmlbr") - try { - spark.sql(s"UPDATE $table SET ${Core.string0.columnName} = 'br-upd' WHERE ${Core.long0.columnName} = 1") - spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 2") - } finally spark.conf.unset("spark.wap.branch") - val onBranch = spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'dmlbr'").collect()(0).getLong(0) - assert(onBranch == 2, s"branch should have 2 rows after delete, got $onBranch") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "main unchanged by branch DML") - val br1 = spark.sql(s"SELECT ${Core.string0.columnName} FROM $table VERSION AS OF 'dmlbr' WHERE ${Core.long0.columnName} = 1").collect()(0).getString(0) - assert(br1 == "br-upd", s"branch update not applied: $br1") - }() - - // B5 lifecycle (CREATE TAG / DROP BRANCH — both supported, verified) + WAP mixing negatives. - val branchCreateTag: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("branch.lifecycle.tag") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE TAG mytag") - assert(spark.sql(s"SELECT count(*) FROM $table.refs WHERE name = 'mytag' AND type = 'TAG'").collect()(0).getLong(0) == 1, - "CREATE TAG did not create the tag ref") - }() - - val branchDropBranch: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("branch.drop.create")(t => s"ALTER TABLE $t CREATE BRANCH tmpbr")() - .step("branch.lifecycle.dropBranch") { (spark, table) => - assert(spark.sql(s"SELECT count(*) FROM $table.refs WHERE name = 'tmpbr'").collect()(0).getLong(0) == 1, "branch not created") - spark.sql(s"ALTER TABLE $table DROP BRANCH tmpbr") - assert(spark.sql(s"SELECT count(*) FROM $table.refs WHERE name = 'tmpbr'").collect()(0).getLong(0) == 0, "DROP BRANCH did not remove the ref") - }() - - val branchNegWapIdAndBranch: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("branch.neg.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .sql("branch.neg.create")(t => s"ALTER TABLE $t CREATE BRANCH nb")() - .step("branch.neg.wapIdAndBranch") { (spark, table) => - spark.conf.set("spark.wap.id", "w1") - spark.conf.set("spark.wap.branch", "nb") - try { - val e = Check.intercept[ValidationException](spark.sql(s"INSERT INTO $table VALUES ${coreRow(99, "x")}")) - assert(e.getMessage.contains("Cannot set both WAP ID and branch"), s"msg: ${e.getMessage.take(140)}") - } finally { spark.conf.unset("spark.wap.id"); spark.conf.unset("spark.wap.branch") } - }() - - val branchNegInsertNonexistent: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("branch.neg.insertNonexistentBranch") { (spark, table) => - val e = Check.intercept[ValidationException](spark.sql(s"INSERT INTO $table.branch_nope VALUES ${coreRow(99, "x")}")) - assert(e.getMessage.contains("does not exist"), s"msg: ${e.getMessage.take(140)}") - }() - - // ── WAP mega-axis Stage B — systematic branch-DDL leak (G8) ────────────────────────────────── - // Table-global DDL (schema / props / sortOrder / policy) run WHILE `spark.wap.branch` is set: per G8 - // these apply table-globally at every layer, so they LEAK to MAIN rather than staying branch-scoped. - // Each pins the ACTUAL outcome on MAIN (wap.branch unset after the DDL) — leak / silent-no-op / rejected. - // If OpenHouse later scopes branch DDL, these flip. Format-multiplexed by crossFmt (seedFmt-aware create). - private def branchDdlOnBranch(label: String)(ddl: String => String)(assertMain: (SparkSession, String) => Unit): TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql(s"$label.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .sql(s"$label.createBranch")(t => s"ALTER TABLE $t CREATE BRANCH bddl")() - .step(label) { (spark, table) => - spark.conf.set("spark.wap.branch", "bddl") - val outcome = try { spark.sql(ddl(table)); "accepted" } - catch { case NonFatal(e) => s"rejected:${Exceptions.root(e).getClass.getSimpleName}" } - finally spark.conf.unset("spark.wap.branch") - println(s"DIAG $label: branch-routed DDL $outcome") - assertMain(spark, table) - }() - - val branchDdlOps: List[(String, TableTest[CoreTable.type])] = List( - // ADD COLUMN on a branch → main's schema gains the column (schema is table-global → leak). - "branchDdl.addColumn.leaksToMain" -> branchDdlOnBranch("branchDdl.addColumn.leaksToMain")( - t => s"ALTER TABLE $t ADD COLUMN br_added int") { (spark, table) => - val cols = spark.sql(s"DESCRIBE TABLE $table").collect().map(_.getString(0).trim).toSet - assert(cols.contains("br_added"), - "G8: ADD COLUMN on a branch should LEAK to main's schema (table-global); main did not gain the column — re-audit G8") - }, - // SET TBLPROPERTIES on a branch → main gets the property (props are table-global → leak). - "branchDdl.setTblProp.leaksToMain" -> branchDdlOnBranch("branchDdl.setTblProp.leaksToMain")( - t => s"ALTER TABLE $t SET TBLPROPERTIES ('user.branchkey'='v1')") { (spark, table) => - val props = spark.sql(s"SHOW TBLPROPERTIES $table").collect().map(r => r.getString(0) -> r.getString(1)).toMap - assert(props.get("user.branchkey").contains("v1"), - s"G8: SET TBLPROPERTIES on a branch should LEAK to main; got ${props.get("user.branchkey")} — re-audit G8") - }, - // ALTER COLUMN comment on a branch → main's schema metadata changes (leak). - "branchDdl.alterColumnComment.leaksToMain" -> branchDdlOnBranch("branchDdl.alterColumnComment.leaksToMain")( - t => s"ALTER TABLE $t ALTER COLUMN ${Core.string0.columnName} COMMENT 'br-comment'") { (spark, table) => - val c = spark.sql(s"DESCRIBE TABLE $table").collect() - .find(_.getString(0).trim == Core.string0.columnName).map(_.getString(2)).getOrElse("") - assert(Option(c).getOrElse("").contains("br-comment"), - s"G8: ALTER COLUMN COMMENT on a branch should LEAK to main; main comment='$c' — re-audit G8") - }, - // DROP COLUMN is rejected on main (unsupported) — assert it is ALSO rejected via a branch (the guard - // is schema-global, not branch-aware): pin the rejection is unchanged under wap.branch. - "branchDdl.dropColumn.rejected" -> branchDdlOnBranch("branchDdl.dropColumn.rejected")( - t => s"ALTER TABLE $t DROP COLUMN ${Core.string0.columnName}") { (spark, table) => - val cols = spark.sql(s"DESCRIBE TABLE $table").collect().map(_.getString(0).trim).toSet - assert(cols.contains(Core.string0.columnName), - "DROP COLUMN must remain rejected (main keeps the column) whether or not spark.wap.branch is set") - } - ) - - val branching: List[(String, TableTest[CoreTable.type])] = List( - "branch.direct.isolation" -> branchDirectIsolation, - "branch.wapConf.routing" -> branchWapConfRouting, - "wap.stagePublish" -> wapStagePublish, - "branch.ddlLeak.addColumn" -> branchDdlLeakAddColumn, - "branch.dml.updateDelete" -> branchDmlUpdateDelete, - "branch.lifecycle.tag" -> branchCreateTag, - "branch.lifecycle.dropBranch" -> branchDropBranch, - "branch.neg.wapIdAndBranch" -> branchNegWapIdAndBranch, - "branch.neg.insertNonexistentBranch" -> branchNegInsertNonexistent - ) + + } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala index 6ddb15a52..4f64ecf4e 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala @@ -13,745 +13,1113 @@ import scala.util.control.NonFatal trait DmlScenarios extends ScenarioKit { import Rows._ - // ── DDL × consumer battery (BUILD-STATUS task #3) ──────────────────────────────────────────── - // A DDL op is a STATE CHANGE; the battery asserts every consumer still works after it (the - // modality thesis at the DDL level). DDL preps leave a distinct post-state; consumers are - // arity-safe (they use SELECT * / metadata tables, never a fixed column list) so they compose - // over ANY post-DDL schema. NOTE: this is the NON-VACUOUS core — the appraisal's 420 assumed - // 35 DDL (incl. negatives/one-shots) × 6, but a rejected DDL or a rename has no post-state for a - // consumer to exercise. State-changing DDL × real consumers is ~54, and that's what's built. - val ddlPreps: List[(String, Layout => TableTest[CoreTable.type])] = List( - "addColumn" -> (l => createAndSeed(l, 3).sql("ddl")(t => s"ALTER TABLE $t ADD COLUMN cc int")()), - "typeWiden" -> (l => createAndSeed(l, 3).sql("ddl")(t => s"ALTER TABLE $t ALTER COLUMN ${Core.int0.columnName} TYPE bigint")()), - "writeOrder" -> (l => createAndSeed(l, 3).sql("ddl")(t => s"ALTER TABLE $t WRITE ORDERED BY ${Core.long0.columnName}")()), - "distMode" -> (l => createAndSeed(l, 3).sql("ddl")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.distribution-mode'='range')")()) - ) - - private def dupRow(key: Long) = s"SELECT * FROM %s WHERE ${Core.long0.columnName} = $key" // arity-safe append source - - val ddlConsumers: List[(String, TableTest[CoreTable.type])] = List( - // C1 the table stays WRITABLE (append) after the DDL — arity-safe self-select append. - "dmlWrite" -> TableTest(Core).step("consume.dmlWrite") { (spark, table) => - spark.sql(s"INSERT INTO $table ${dupRow(1).format(table)}") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "not writable post-DDL") - }(), - // C2 the MUTATION path still works after the DDL. - "dmlMutate" -> TableTest(Core).step("consume.dmlMutate") { (spark, table) => - spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 2") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "mutation broken post-DDL") - }(), - // C3 TIME TRAVEL to the pre-DDL/seed snapshot still resolves. - "timeTravel" -> TableTest(Core).step("consume.timeTravel") { (spark, table) => - val s0 = snapshotIds(spark, table).head - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF $s0").collect()(0).getLong(0) == 3, - "pre-DDL snapshot not travelable") - }(), - // C4 RESTORE across the DDL: write post-DDL, then roll back to the seed snapshot. - "restore" -> TableTest(Core).step("consume.restore") { (spark, table) => - val s0 = snapshotIds(spark, table).head - spark.sql(s"INSERT INTO $table ${dupRow(1).format(table)}") - spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', $s0)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "restore across DDL failed") - }(), - // C5 EXPIRE after the DDL: history trims, current data survives and reads. - "expire" -> TableTest(Core).step("consume.expire") { (spark, table) => - spark.sql(s"INSERT INTO $table ${dupRow(1).format(table)}") - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "unreadable after expire post-DDL") - }(), - // C6 BRANCH after the DDL: branchable, write on branch, main isolated. - "branch" -> TableTest(Core).step("consume.branch") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH cb") - spark.sql(s"INSERT INTO $table.branch_cb ${dupRow(1).format(table)}") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'cb'").collect()(0).getLong(0) == 4, "branch write failed post-DDL") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "branch leaked to main post-DDL") - }(), - // C7 COMPACTION after the DDL: a second data file, then rewrite_data_files preserves the rows. - "compact" -> TableTest(Core).step("consume.compact") { (spark, table) => - spark.sql(s"INSERT INTO $table ${dupRow(1).format(table)}") // second data file - spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('min-input-files', '2'))") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "compaction changed rows post-DDL") - }() - ) - - // Closing assertion for the branch axis: after the branch-routed op, MAIN must be untouched - // (still the 3-row seed) — the isolation half of the branch contract. Uniform across all ops - // because with spark.wap.branch set every write routes to the branch, never to main. - val branchMainIsolation: TableTest[CoreTable.type] = - TableTest(Core).step("branch.mainIsolated") { (spark, table) => - spark.conf.unset("spark.wap.branch") - val mainCount = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) - assert(mainCount == 3, s"branch op leaked to MAIN — expected 3 rows, got $mainCount (isolation broken)") - }() - - // ── reads ──────────────────────────────────────────────────────────────────────────── - val readProjection: TableTest[CoreTable.type] = - TableTest(Core).check("read.projection") { view => - val expected = view.before.sortBy(_.get(Core.long0)).map(_.get(Core.string0)) - val actual = view.spark - .sql(s"SELECT ${Core.string0.columnName} FROM ${view.table} ORDER BY ${Core.long0.columnName}") - .collect().toSeq.map(_.get(Core.string0)) - assert(actual == expected) - } - - val readFilter: TableTest[CoreTable.type] = - TableTest(Core).check("read.filter") { view => - val expected = view.before.map(_.get(Core.long0)).filter(_ >= 2).sorted - val actual = view.spark - .sql(s"SELECT ${Core.long0.columnName} FROM ${view.table} WHERE ${Core.long0.columnName} >= 2 ORDER BY ${Core.long0.columnName}") - .collect().toSeq.map(_.get(Core.long0)) - assert(actual == expected) - } - - // The declared write format actually materializes: every data file carries that extension. - val formatMaterialization: TableTest[CoreTable.type] = - TableTest(Core).check("format.materialization") { view => - val format = view.spark.sql(s"SHOW TBLPROPERTIES ${view.table} ('write.format.default')").collect()(0).getString(1) - val paths = view.spark.sql(s"SELECT file_path FROM ${view.table}.files").collect().toSeq.map(_.getString(0)) - assert(paths.nonEmpty && paths.forall(_.toLowerCase.endsWith(s".$format")), s"data files are not all .$format: $paths") - } - - // ── delete ─────────────────────────────────────────────────────────────────────────── - val deleteByPredicate: TableTest[CoreTable.type] = - TableTest(Core).delete(core => s"${core.long0.columnName} < 2") { view => - assert(view.after == view.before.filterNot(_.get(Core.long0) < 2)) - } - - val deleteWhereFalseKeepsSnapshot: TableTest[CoreTable.type] = - TableTest(Core).delete(_ => "false") { view => - assert(view.after == view.before) - assert(view.snapshotsAfter == view.snapshotsBefore, "DELETE WHERE false must not commit a snapshot") - } - - val truncate: TableTest[CoreTable.type] = - TableTest(Core).sql("delete.truncate")(table => s"TRUNCATE TABLE $table") { view => - assert(view.after.isEmpty) - } - - val deleteAtSnapshotRejected: TableTest[CoreTable.type] = - TableTest(Core).step("delete.atSnapshot.rejected") { (spark, table) => - val snapshotId = spark - .sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at DESC LIMIT 1") - .collect()(0).getLong(0) - val error = Check.intercept[IllegalArgumentException]( - spark.sql(s"DELETE FROM $table.snapshot_id_$snapshotId WHERE ${Core.long0.columnName} < 4")) - assert(error.getMessage == s"Cannot delete from table at a specific snapshot: $snapshotId") - } { view => - assert(view.after == view.before) // a rejected delete leaves the table unchanged - } - - // Removes exactly the keys in the list. - val deleteByInList: TableTest[CoreTable.type] = - TableTest(Core).delete(core => s"${core.long0.columnName} IN (1, 3)") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(Set(1L, 3L)).sorted) - } - - // Predicate is an IN-subquery over an explicit source. - val deleteByInSubquery: TableTest[CoreTable.type] = - TableTest(Core).delete(core => - s"${core.long0.columnName} IN (SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(_ == 2L).sorted) - } - - val deleteByNotInSubquery: TableTest[CoreTable.type] = - TableTest(Core).delete(core => - s"${core.long0.columnName} NOT IN (SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filter(_ == 2L).sorted) - } - - val deleteByExistsSubquery: TableTest[CoreTable.type] = - TableTest(Core).delete(core => - s"EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) WHERE s.x = ${core.long0.columnName})") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(_ == 2L).sorted) - } - - val deleteByNotExistsSubquery: TableTest[CoreTable.type] = - TableTest(Core).delete(core => - s"NOT EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) WHERE s.x = ${core.long0.columnName})") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filter(_ == 2L).sorted) - } - - val deleteByScalarSubquery: TableTest[CoreTable.type] = - TableTest(Core).delete(core => - s"${core.long0.columnName} = (SELECT max(col1) FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(_ == 2L).sorted) - } - - // Seed a null-string row, then DELETE WHERE string IS NULL must remove exactly it (and nothing - // else) — a real IS-NULL match, not a vacuous no-op. - val deleteByNullCondition: TableTest[CoreTable.type] = - TableTest(Core) - .sql("delete.byNullCondition.seed")(table => - s"INSERT INTO $table VALUES (CAST(99 AS BIGINT), 99, NULL, 99.5, false, '2024-01-01-00')")() - .delete(core => s"${core.string0.columnName} IS NULL") { view => - assert(view.before.exists(_.get(Core.string0) == null), "precondition: a null-string row was seeded") - val expected = view.before.filterNot(_.get(Core.string0) == null).map(_.get(Core.long0)).sorted - assert(keyed(view.after) == expected) // exactly the non-null rows remain - assert(!keyed(view.after).contains(99L)) // the null-string row was removed + val ddlConsumerCases: List[Plan.Case] = + layouts + .filter(layout => + layout.label.endsWith("/parquet") || + layout.label.endsWith("/orc")) + .flatMap { layout => + val preparations = List( + TablePreparation( + layout.label, + createAndSeed(layout, 3) + .sql("ddl")(table => s"ALTER TABLE $table ADD COLUMN cc int")(), + "ddlConsume:addColumn."), + TablePreparation( + layout.label, + createAndSeed(layout, 3) + .sql("ddl")(table => + s"ALTER TABLE $table ALTER COLUMN ${Core.int0.columnName} TYPE bigint")(), + "ddlConsume:typeWiden."), + TablePreparation( + layout.label, + createAndSeed(layout, 3) + .sql("ddl")(table => + s"ALTER TABLE $table WRITE ORDERED BY ${Core.long0.columnName}")(), + "ddlConsume:writeOrder."), + TablePreparation( + layout.label, + createAndSeed(layout, 3) + .sql("ddl")(table => + s"ALTER TABLE $table SET TBLPROPERTIES " + + "('write.distribution-mode'='range')")(), + "ddlConsume:distMode.")) + + preparations.flatMap { preparation => + List( + preparation.test("dmlWrite") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "table is not writable after DDL") + }, + preparation.test("dmlMutate") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "mutation failed after DDL") + }, + preparation.test("timeTravel") { table => + val seedSnapshotId = + snapshotIds(table.spark, table.name).head + + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF $seedSnapshotId") + .collect()(0) + .getLong(0) == 3, + "seed snapshot is not readable after DDL") + }, + preparation.test("restore") { table => + val seedSnapshotId = + snapshotIds(table.spark, table.name).head + + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $seedSnapshotId)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 3, + "restore across DDL failed") + }, + preparation.test("expire") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "table is unreadable after snapshot expiration") + }, + preparation.test("branch") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH cb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_cb " + + s"SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") + .collect()(0) + .getLong(0) == 4, + "branch write failed after DDL") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 3, + "branch write changed the main table") + }, + preparation.test("compact") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('min-input-files', '2'))") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "compaction changed rows after DDL") + }) + } } - // DELETE with no WHERE clause empties the table. - val deleteAll: TableTest[CoreTable.type] = - TableTest(Core).sql("delete.all")(table => s"DELETE FROM $table") { view => - assert(view.after.isEmpty) - } - - // A real predicate that matches nothing: rows unchanged, but one (empty) snapshot is still - // committed — a scanned no-match, unlike the constant-folded `DELETE WHERE false` no-op above. - val deleteNone: TableTest[CoreTable.type] = - TableTest(Core).delete(core => s"${core.long0.columnName} = 999") { view => - assert(view.after == view.before) - assert(view.snapshotsAfter == view.snapshotsBefore + 1, "no-match DELETE with a real predicate still commits one snapshot") - } - - // A partition-column predicate (a metadata-only delete on a partitioned layout). - val deleteByPartitionPredicate: TableTest[CoreTable.type] = - TableTest(Core).delete(core => s"${core.datePartition.columnName} = '2024-01-01-00'") { view => - val expected = view.before.filterNot(_.get(Core.datePartition) == "2024-01-01-00").map(_.get(Core.long0)).sorted - assert(keyed(view.after) == expected) - } - - val deleteWithAlias: TableTest[CoreTable.type] = - TableTest(Core).sql("delete.withAlias")(table => - s"DELETE FROM $table AS x WHERE x.${Core.long0.columnName} < 2") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(_ < 2L).sorted) - } - - // ── update ─────────────────────────────────────────────────────────────────────────── - val updateByPredicate: TableTest[CoreTable.type] = - TableTest(Core).sql("update.byPredicate")(table => - s"UPDATE $table SET ${Core.string0.columnName} = 'X' WHERE ${Core.long0.columnName} = 2") { view => - val expected = longToString(view.before).map { case (id, s) => id -> (if (id == 2) "X" else s) } - assert(longToString(view.after) == expected) - } - - val updateWithoutCondition: TableTest[CoreTable.type] = - TableTest(Core).sql("update.withoutCondition")(table => - s"UPDATE $table SET ${Core.string0.columnName} = 'Z'") { view => - assert(longToString(view.after) == longToString(view.before).map { case (id, _) => id -> "Z" }) - } - - // A real predicate matching nothing still commits an (empty) snapshot — unlike the - // constant-folded `DELETE WHERE false` no-op (confirmed vs OSS TestUpdate.testUpdateNonExistingRecords). - val updateNoMatch: TableTest[CoreTable.type] = - TableTest(Core).sql("update.noMatch")(table => - s"UPDATE $table SET ${Core.string0.columnName} = 'Y' WHERE ${Core.long0.columnName} = 99") { view => - assert(longToString(view.after) == longToString(view.before)) - assert(view.snapshotsAfter == view.snapshotsBefore + 1, "no-match UPDATE still commits one snapshot") - } - - private def stringUpdatedWhere(view: StepView[CoreTable.type], matches: Long => Boolean, to: String): Boolean = - longToString(view.after) == longToString(view.before).map { case (id, s) => id -> (if (matches(id)) to else s) } - - val updateByInSubquery: TableTest[CoreTable.type] = - TableTest(Core).sql("update.byInSubquery")(table => - s"UPDATE $table SET ${Core.string0.columnName} = 'X' " + - s"WHERE ${Core.long0.columnName} IN (SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => - assert(stringUpdatedWhere(view, _ == 2, "X")) - } - - val updateByNotInSubquery: TableTest[CoreTable.type] = - TableTest(Core).sql("update.byNotInSubquery")(table => - s"UPDATE $table SET ${Core.string0.columnName} = 'X' " + - s"WHERE ${Core.long0.columnName} NOT IN (SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => - assert(stringUpdatedWhere(view, _ != 2, "X")) - } - - val updateByExistsSubquery: TableTest[CoreTable.type] = - TableTest(Core).sql("update.byExistsSubquery")(table => - s"UPDATE $table SET ${Core.string0.columnName} = 'X' " + - s"WHERE EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) WHERE s.x = ${Core.long0.columnName})") { view => - assert(stringUpdatedWhere(view, _ == 2, "X")) - } - - val updateByNotExistsSubquery: TableTest[CoreTable.type] = - TableTest(Core).sql("update.byNotExistsSubquery")(table => - s"UPDATE $table SET ${Core.string0.columnName} = 'X' " + - s"WHERE NOT EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) WHERE s.x = ${Core.long0.columnName})") { view => - assert(stringUpdatedWhere(view, _ != 2, "X")) - } - - val updateByScalarSubquery: TableTest[CoreTable.type] = - TableTest(Core).sql("update.byScalarSubquery")(table => - s"UPDATE $table SET ${Core.string0.columnName} = 'X' " + - s"WHERE ${Core.long0.columnName} = (SELECT max(col1) FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") { view => - assert(stringUpdatedWhere(view, _ == 2, "X")) - } - - val updateWithAlias: TableTest[CoreTable.type] = - TableTest(Core).sql("update.withAlias")(table => - s"UPDATE $table AS x SET x.${Core.string0.columnName} = 'X' WHERE x.${Core.long0.columnName} = 2") { view => - assert(stringUpdatedWhere(view, _ == 2, "X")) - } - - // Sets two columns in one statement; assert both landed on the matched row. - val updateMultipleColumns: TableTest[CoreTable.type] = - TableTest(Core).sql("update.multipleColumns")(table => - s"UPDATE $table SET ${Core.string0.columnName} = 'X', ${Core.int0.columnName} = 99 WHERE ${Core.long0.columnName} = 2") { view => - assert(stringUpdatedWhere(view, _ == 2, "X")) - assert(view.after.find(_.get(Core.long0) == 2L).map(_.get(Core.int0)).contains(99)) - } - - // Assign a column by an expression over itself (updates the key column). - val updateByExpression: TableTest[CoreTable.type] = - TableTest(Core).sql("update.byExpression")(table => - s"UPDATE $table SET ${Core.long0.columnName} = ${Core.long0.columnName} + 10 WHERE ${Core.long0.columnName} = 2") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).map(l => if (l == 2L) 12L else l).sorted) - } - - // Update the partition column so the row moves partitions. - val updateMovePartition: TableTest[CoreTable.type] = - TableTest(Core).sql("update.movePartition")(table => - s"UPDATE $table SET ${Core.datePartition.columnName} = '2099-12-31-23' WHERE ${Core.long0.columnName} = 2") { view => - val part = (rows: Seq[Row]) => rows.map(r => r.get(Core.long0) -> r.get(Core.datePartition)).toMap - assert(part(view.after) == part(view.before).map { case (id, d) => id -> (if (id == 2) "2099-12-31-23" else d) }) - } - - val updateNullAssignment: TableTest[CoreTable.type] = - TableTest(Core).sql("update.nullAssignment")(table => - s"UPDATE $table SET ${Core.string0.columnName} = NULL WHERE ${Core.long0.columnName} = 2") { view => - assert(longToString(view.after) == longToString(view.before).map { case (id, s) => id -> (if (id == 2) null else s) }) - } - - // ── merge ──────────────────────────────────────────────────────────────────────────── - // Source rows are written as EXPLICIT literals. The generator-sourced alternative for this - // test would be: - // USING (${RowGenerator.valuesClause(Core, ...)} for indices 4,5) ... WHEN NOT MATCHED THEN INSERT * - // i.e. name the row *indices* and let the column generators fill every column. We prefer the - // explicit form so the source values are visible in the test. - val mergeInsertNotMatched: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.insertNotMatched")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES - (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), - (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') - AS s($cols) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN NOT MATCHED THEN INSERT *""") { view => - assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) ++ Seq(4L, 5L)).sorted) - // INSERT * must map the columns correctly, not just land the join key. - assert(view.after.find(_.get(Core.long0) == 4L).map(_.get(Core.string0)).contains("row-4")) - assert(view.after.find(_.get(Core.long0) == 5L).map(_.get(Core.string0)).contains("row-5")) - } - - val mergeUpdateMatched: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.updateMatched")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES (CAST(2 AS BIGINT), 'M') AS s(${Core.long0.columnName}, ${Core.string0.columnName}) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") { view => - val expected = longToString(view.before).map { case (id, s) => id -> (if (id == 2) "M" else s) } - assert(longToString(view.after) == expected) - } - - val mergeDeleteMatched: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.deleteMatched")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES (CAST(1 AS BIGINT)), (CAST(3 AS BIGINT)) AS s(${Core.long0.columnName}) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN MATCHED THEN DELETE""") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(Set(1L, 3L)).sorted) - } - - val mergeUpsert: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.upsert")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES - (CAST(2 AS BIGINT), 2, 'U', 2.5, true, '2024-01-02-01'), - (CAST(7 AS BIGINT), 7, 'g', 7.5, false, '2024-01-07-06') - AS s($cols) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName} - WHEN NOT MATCHED THEN INSERT *""") { view => - val updated = longToString(view.before).map { case (id, s) => id -> (if (id == 2) "U" else s) } - val withInsert = if (view.before.exists(_.get(Core.long0) == 7L)) updated else updated + (7L -> "g") - assert(longToString(view.after) == withInsert) - } - - // Keep only rows the source knows about: delete every row NOT matched by a source row. - val mergeDeleteNotMatchedBySource: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.deleteNotMatchedBySource")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES (CAST(2 AS BIGINT)) AS s(${Core.long0.columnName}) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN NOT MATCHED BY SOURCE THEN DELETE""") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filter(_ == 2L).sorted) - } - - // Both keys 2 and 3 match, but the per-clause condition only fires for key 2. - val mergeConditionalUpdate: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.conditionalUpdate")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES (CAST(2 AS BIGINT), 'U2'), (CAST(3 AS BIGINT), 'U3') - AS s(${Core.long0.columnName}, ${Core.string0.columnName}) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN MATCHED AND s.${Core.long0.columnName} = 2 THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") { view => - assert(longToString(view.after) == longToString(view.before).map { case (id, s) => id -> (if (id == 2) "U2" else s) }) - } - - // First matched clause wins: key 2 updates (conditional), key 3 falls through to DELETE. - val mergeMultipleMatchedClauses: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.multipleMatchedClauses")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES (CAST(2 AS BIGINT), 'U'), (CAST(3 AS BIGINT), 'x') - AS s(${Core.long0.columnName}, ${Core.string0.columnName}) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN MATCHED AND s.${Core.long0.columnName} = 2 THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName} - WHEN MATCHED THEN DELETE""") { view => - assert(keyed(view.after) == view.before.map(_.get(Core.long0)).filterNot(_ == 3L).sorted) - assert(view.after.find(_.get(Core.long0) == 2L).map(_.get(Core.string0)).contains("U")) - } - - // Conditional NOT MATCHED: source keys 4 and 5, but only 4 satisfies the insert condition. - val mergeConditionalInsert: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.conditionalInsert")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES - (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), - (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') - AS s($cols) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN NOT MATCHED AND s.${Core.long0.columnName} = 4 THEN INSERT *""") { view => - assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 4L).sorted) - } - - // All three clause kinds in one statement: update key 2, insert key 4, delete-by-source rows 1 & 3. - val mergeAllClauses: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.allClauses")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES - (CAST(2 AS BIGINT), 2, 'M2', 2.5, true, '2024-01-02-01'), - (CAST(4 AS BIGINT), 4, 'row-4', 4.5, false, '2024-01-04-03') - AS s($cols) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName} - WHEN NOT MATCHED THEN INSERT * - WHEN NOT MATCHED BY SOURCE THEN DELETE""") { view => - assert(keyed(view.after) == Seq(2L, 4L)) - assert(view.after.find(_.get(Core.long0) == 2L).map(_.get(Core.string0)).contains("M2")) - } - - // UPDATE SET * replaces every column of the matched row from the source. - val mergeUpdateStar: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.updateStar")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES (CAST(2 AS BIGINT), 22, 'S2', 22.5, true, '2024-06-06-06') AS s($cols) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN MATCHED THEN UPDATE SET *""") { view => - val row2 = view.after.find(_.get(Core.long0) == 2L) - assert(row2.map(_.get(Core.string0)).contains("S2")) - assert(row2.map(_.get(Core.int0)).contains(22)) - } - - // Explicit column-specification INSERT (other columns null-filled). - val mergeInsertExplicitColumns: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.insertExplicitColumns")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES (CAST(7 AS BIGINT), 'g') AS s(${Core.long0.columnName}, ${Core.string0.columnName}) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}, ${Core.string0.columnName}) VALUES (s.${Core.long0.columnName}, s.${Core.string0.columnName})""") { view => - assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 7L).sorted) - assert(view.after.find(_.get(Core.long0) == 7L).map(_.get(Core.string0)).contains("g")) - } - - // Source is a CTE. - val mergeSourceCTE: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.sourceCTE")(table => - s"""MERGE INTO $table t USING ( - WITH src AS (SELECT CAST(8 AS BIGINT) AS ${Core.long0.columnName}) SELECT * FROM src - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}) VALUES (s.${Core.long0.columnName})""") { view => - assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 8L).sorted) - } + val createSchemaCases: List[Plan.Case] = preparedEmptyCoreTables.map { preparation => + preparation.test("create.schema") { table => + val actual = table.spark + .table(table.name) + .schema + .fields + .toList + .map(field => field.name -> field.dataType.simpleString) + val expected = Core.tableColumns.toList.map(column => (column.columnName, column.sqlType)) - // Source is a set operation (UNION ALL). - val mergeSourceSetOp: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.sourceSetOp")(table => - s"""MERGE INTO $table t USING ( - SELECT CAST(8 AS BIGINT) AS ${Core.long0.columnName} UNION ALL SELECT CAST(9 AS BIGINT) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}) VALUES (s.${Core.long0.columnName})""") { view => - assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) ++ Seq(8L, 9L)).sorted) + assert(actual == expected) + assert(table.rows.isEmpty) } - - // Merge into an empty target inserts all non-matching source rows (empties the seed first). - val mergeIntoEmptyTarget: TableTest[CoreTable.type] = - TableTest(Core) - .sql("merge.intoEmptyTarget.empty")(table => s"DELETE FROM $table")() - .sql("merge.intoEmptyTarget")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES + } + + val ddlSchemaCases: List[Plan.Case] = preparedCoreTables.flatMap { preparation => + List( + preparation.test("ddl.addColumn.single") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_int int") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + val nullCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name} WHERE added_int IS NULL") + .collect()(0) + .getLong(0) + + assert(columnNames.contains("added_int"), s"added_int missing: $columnNames") + assert( + nullCount == table.preparedRows.size, + s"existing rows should read null for added_int: $nullCount != ${table.preparedRows.size}") + assert(table.rows.size == table.preparedRows.size) + }, + preparation.test("ddl.addColumn.multiple") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMNS (added_a int, added_b string)") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + + assert( + columnNames.contains("added_a") && columnNames.contains("added_b"), + s"added columns missing: $columnNames") + assert(table.rows.size == table.preparedRows.size) + }, + preparation.test("ddl.addColumn.comment") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_c int COMMENT 'a note'") + + val addedColumn = table.spark + .table(table.name) + .schema + .fields + .find(_.name == "added_c") + .getOrElse(throw new AssertionError("added_c missing")) + + assert( + addedColumn.getComment().contains("a note"), + s"comment not stored: ${addedColumn.getComment()}") + }, + preparation.test("ddl.addColumn.position") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN added_after int AFTER ${Core.long0.columnName}") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + + assert( + columnNames.indexOf("added_after") == columnNames.indexOf(Core.long0.columnName) + 1, + s"added_after not after long0: $columnNames") + }, + preparation.test("ddl.alterColumn.typeWiden") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.int0.columnName} TYPE bigint") + + val liveColumns = table.spark.table(table.name).schema.fields.toSeq + .map(field => field.name -> field.dataType.simpleString) + .toMap + val values = table.spark + .sql( + s"SELECT ${Core.int0.columnName} FROM ${table.name} ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) + + assert( + liveColumns.get(Core.int0.columnName).contains("bigint"), + s"int0 not widened: ${liveColumns.get(Core.int0.columnName)}") + assert(values == Seq(1L, 2L, 3L), s"values not preserved after widening: $values") + }, + preparation.test("ddl.renameColumn") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN to_rename int") + table.spark.sql(s"ALTER TABLE ${table.name} RENAME COLUMN to_rename TO renamed_col") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + + assert( + columnNames.contains("renamed_col") && !columnNames.contains("to_rename"), + s"RENAME COLUMN silently no-oped: $columnNames") + assert(table.rows.size == table.preparedRows.size) + }) + } + + private def localizedDmlCases( + preparation: TablePreparation[CoreTable.type] + ): List[Plan.Case] = + List( + preparation.test("read.projection") { table => + val expected = table.preparedRows + .sortBy(_.get(Core.long0)) + .map(_.get(Core.string0)) + val actual = table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.get(Core.string0)) + + assert(actual == expected) + }, + preparation.test("read.filter") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filter(_ >= 2) + .sorted + val actual = table.spark + .sql( + s"SELECT ${Core.long0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} >= 2 ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.get(Core.long0)) + + assert(actual == expected) + }, + preparation.test("format.materialization") { table => + val format = table.spark + .sql( + s"SHOW TBLPROPERTIES ${table.name} ('write.format.default')") + .collect()(0) + .getString(1) + val filePaths = table.spark + .sql(s"SELECT file_path FROM ${table.name}.files") + .collect() + .toSeq + .map(_.getString(0)) + + assert( + filePaths.nonEmpty && + filePaths.forall(_.toLowerCase.endsWith(s".$format")), + s"data files are not all .$format: $filePaths") + }, + preparation.test("delete.byPredicate") { table => + val expected = table.preparedRows.filterNot(_.get(Core.long0) < 2) + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") + + assert(table.rows == expected) + }, + preparation.test("delete.byInList") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filterNot(Set(1L, 3L)) + .sorted + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} IN (1, 3)") + + assert(keyed(table.rows) == expected) + }, + preparation.test("delete.byInSubquery") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filterNot(_ == 2L) + .sorted + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} IN (" + + "SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + + assert(keyed(table.rows) == expected) + }, + preparation.test("delete.byNotInSubquery") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filter(_ == 2L) + .sorted + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} NOT IN (" + + "SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + + assert(keyed(table.rows) == expected) + }, + preparation.test("delete.byExistsSubquery") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filterNot(_ == 2L) + .sorted + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE EXISTS (" + + "SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) " + + s"WHERE s.x = ${Core.long0.columnName})") + + assert(keyed(table.rows) == expected) + }, + preparation.test("delete.byNotExistsSubquery") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filter(_ == 2L) + .sorted + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE NOT EXISTS (" + + "SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) " + + s"WHERE s.x = ${Core.long0.columnName})") + + assert(keyed(table.rows) == expected) + }, + preparation.test("delete.byScalarSubquery") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filterNot(_ == 2L) + .sorted + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = (" + + "SELECT max(col1) FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + + assert(keyed(table.rows) == expected) + }, + preparation.test("delete.byNullCondition") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES (" + + "CAST(99 AS BIGINT), 99, NULL, 99.5, false, '2024-01-01-00')") + val rowsBeforeDelete = table.rows + val expected = rowsBeforeDelete + .filter(row => Option(row.get(Core.string0)).nonEmpty) + .map(_.get(Core.long0)) + .sorted + + assert( + rowsBeforeDelete.exists(row => Option(row.get(Core.string0)).isEmpty), + "precondition: a null-string row was seeded") + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.string0.columnName} IS NULL") + + assert(keyed(table.rows) == expected) + assert(!keyed(table.rows).contains(99L)) + }, + preparation.test("delete.all") { table => + table.spark.sql(s"DELETE FROM ${table.name}") + + assert(table.rows.isEmpty) + }, + preparation.test("delete.none") { table => + val snapshotsBefore = table.snapshotCount + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 999") + + assert(table.rows == table.preparedRows) + assert( + table.snapshotCount == snapshotsBefore + 1, + "no-match DELETE with a real predicate still commits one snapshot") + }, + preparation.test("delete.byPartitionPredicate") { table => + val expected = table.preparedRows + .filterNot(_.get(Core.datePartition) == "2024-01-01-00") + .map(_.get(Core.long0)) + .sorted + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE " + + s"${Core.datePartition.columnName} = '2024-01-01-00'") + + assert(keyed(table.rows) == expected) + }, + preparation.test("delete.withAlias") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filterNot(_ < 2L) + .sorted + + table.spark.sql( + s"DELETE FROM ${table.name} AS x WHERE x.${Core.long0.columnName} < 2") + + assert(keyed(table.rows) == expected) + }, + preparation.test("delete.whereFalse.noSnapshot") { table => + val snapshotsBefore = table.snapshotCount + + table.spark.sql(s"DELETE FROM ${table.name} WHERE false") + + assert(table.rows == table.preparedRows) + assert( + table.snapshotCount == snapshotsBefore, + "DELETE WHERE false must not commit a snapshot") + }, + preparation.test("delete.truncate") { table => + table.spark.sql(s"TRUNCATE TABLE ${table.name}") + + assert(table.rows.isEmpty) + }, + preparation.test("delete.atSnapshot.rejected") { table => + val snapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "ORDER BY committed_at DESC LIMIT 1") + .collect()(0) + .getLong(0) + val exception = Check.intercept[IllegalArgumentException]( + table.spark.sql( + s"DELETE FROM ${table.name}.snapshot_id_$snapshotId " + + s"WHERE ${Core.long0.columnName} < 4")) + + assert( + exception.getMessage == + s"Cannot delete from table at a specific snapshot: $snapshotId") + assert(table.rows == table.preparedRows) + }, + preparation.test("update.byPredicate") { table => + val expected = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id == 2) "X" else value) + } + + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + + s"WHERE ${Core.long0.columnName} = 2") + + assert(longToString(table.rows) == expected) + }, + preparation.test("update.withoutCondition") { table => + val expected = longToString(table.preparedRows).map { + case (id, _) => id -> "Z" + } + + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'Z'") + + assert(longToString(table.rows) == expected) + }, + preparation.test("update.noMatch") { table => + val snapshotsBefore = table.snapshotCount + + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'Y' " + + s"WHERE ${Core.long0.columnName} = 99") + + assert(longToString(table.rows) == longToString(table.preparedRows)) + assert( + table.snapshotCount == snapshotsBefore + 1, + "no-match UPDATE still commits one snapshot") + }, + preparation.test("update.byInSubquery") { table => + val expected = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id == 2) "X" else value) + } + + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + + s"WHERE ${Core.long0.columnName} IN (" + + "SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + + assert(longToString(table.rows) == expected) + }, + preparation.test("update.byNotInSubquery") { table => + val expected = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id != 2) "X" else value) + } + + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + + s"WHERE ${Core.long0.columnName} NOT IN (" + + "SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + + assert(longToString(table.rows) == expected) + }, + preparation.test("update.byExistsSubquery") { table => + val expected = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id == 2) "X" else value) + } + + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + + "WHERE EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) " + + s"WHERE s.x = ${Core.long0.columnName})") + + assert(longToString(table.rows) == expected) + }, + preparation.test("update.byNotExistsSubquery") { table => + val expected = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id != 2) "X" else value) + } + + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + + "WHERE NOT EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) " + + s"WHERE s.x = ${Core.long0.columnName})") + + assert(longToString(table.rows) == expected) + }, + preparation.test("update.byScalarSubquery") { table => + val expected = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id == 2) "X" else value) + } + + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + + s"WHERE ${Core.long0.columnName} = (" + + "SELECT max(col1) FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + + assert(longToString(table.rows) == expected) + }, + preparation.test("update.withAlias") { table => + val expected = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id == 2) "X" else value) + } + + table.spark.sql( + s"UPDATE ${table.name} AS x SET x.${Core.string0.columnName} = 'X' " + + s"WHERE x.${Core.long0.columnName} = 2") + + assert(longToString(table.rows) == expected) + }, + preparation.test("update.multipleColumns") { table => + val expectedStrings = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id == 2) "X" else value) + } + + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X', " + + s"${Core.int0.columnName} = 99 WHERE ${Core.long0.columnName} = 2") + + assert(longToString(table.rows) == expectedStrings) + assert( + table.rows + .find(_.get(Core.long0) == 2L) + .map(_.get(Core.int0)) + .contains(99)) + }, + preparation.test("update.byExpression") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .map(value => if (value == 2L) 12L else value) + .sorted + + table.spark.sql( + s"UPDATE ${table.name} SET " + + s"${Core.long0.columnName} = ${Core.long0.columnName} + 10 " + + s"WHERE ${Core.long0.columnName} = 2") + + assert(keyed(table.rows) == expected) + }, + preparation.test("update.movePartition") { table => + val expected = table.preparedRows.map { row => + val id = row.get(Core.long0) + id -> (if (id == 2) "2099-12-31-23" else row.get(Core.datePartition)) + }.toMap + + table.spark.sql( + s"UPDATE ${table.name} SET " + + s"${Core.datePartition.columnName} = '2099-12-31-23' " + + s"WHERE ${Core.long0.columnName} = 2") + + val actual = table.rows.map(row => + row.get(Core.long0) -> row.get(Core.datePartition)).toMap + + assert(actual == expected) + }, + preparation.test("update.nullAssignment") { table => + val expected = table.preparedRows.map { row => + val id = row.get(Core.long0) + id -> (if (id == 2) None else Option(row.get(Core.string0))) + }.toMap + + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = NULL " + + s"WHERE ${Core.long0.columnName} = 2") + + val actual = table.rows.map(row => + row.get(Core.long0) -> Option(row.get(Core.string0))).toMap + + assert(actual == expected) + }, + preparation.test("merge.insertNotMatched") { table => + val expectedKeys = + (table.preparedRows.map(_.get(Core.long0)) ++ Seq(4L, 5L)).sorted + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES + (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), + (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN INSERT *""") + + assert(keyed(table.rows) == expectedKeys) + assert( + table.rows + .find(_.get(Core.long0) == 4L) + .map(_.get(Core.string0)) + .contains("row-4")) + assert( + table.rows + .find(_.get(Core.long0) == 5L) + .map(_.get(Core.string0)) + .contains("row-5")) + }, + preparation.test("merge.updateMatched") { table => + val expected = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id == 2) "M" else value) + } + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT), 'M') + AS s(${Core.long0.columnName}, ${Core.string0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN UPDATE + SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") + + assert(longToString(table.rows) == expected) + }, + preparation.test("merge.deleteMatched") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filterNot(Set(1L, 3L)) + .sorted + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES (CAST(1 AS BIGINT)), (CAST(3 AS BIGINT)) + AS s(${Core.long0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN DELETE""") + + assert(keyed(table.rows) == expected) + }, + preparation.test("merge.upsert") { table => + val updated = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id == 2) "U" else value) + } + val expected = + if (table.preparedRows.exists(_.get(Core.long0) == 7L)) updated + else updated + (7L -> "g") + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES + (CAST(2 AS BIGINT), 2, 'U', 2.5, true, '2024-01-02-01'), + (CAST(7 AS BIGINT), 7, 'g', 7.5, false, '2024-01-07-06') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN UPDATE + SET t.${Core.string0.columnName} = s.${Core.string0.columnName} + WHEN NOT MATCHED THEN INSERT *""") + + assert(longToString(table.rows) == expected) + }, + preparation.test("merge.deleteNotMatchedBySource") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filter(_ == 2L) + .sorted + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT)) + AS s(${Core.long0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED BY SOURCE THEN DELETE""") + + assert(keyed(table.rows) == expected) + }, + preparation.test("merge.conditionalUpdate") { table => + val expected = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id == 2) "U2" else value) + } + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT), 'U2'), + (CAST(3 AS BIGINT), 'U3') + AS s(${Core.long0.columnName}, ${Core.string0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED AND s.${Core.long0.columnName} = 2 THEN UPDATE + SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") + + assert(longToString(table.rows) == expected) + }, + preparation.test("merge.multipleMatchedClauses") { table => + val expected = table.preparedRows + .map(_.get(Core.long0)) + .filterNot(_ == 3L) + .sorted + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT), 'U'), + (CAST(3 AS BIGINT), 'x') + AS s(${Core.long0.columnName}, ${Core.string0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED AND s.${Core.long0.columnName} = 2 THEN UPDATE + SET t.${Core.string0.columnName} = s.${Core.string0.columnName} + WHEN MATCHED THEN DELETE""") + + assert(keyed(table.rows) == expected) + assert( + table.rows + .find(_.get(Core.long0) == 2L) + .map(_.get(Core.string0)) + .contains("U")) + }, + preparation.test("merge.conditionalInsert") { table => + val expected = + (table.preparedRows.map(_.get(Core.long0)) :+ 4L).sorted + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES + (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), + (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED AND s.${Core.long0.columnName} = 4 THEN INSERT *""") + + assert(keyed(table.rows) == expected) + }, + preparation.test("merge.allClauses") { table => + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES + (CAST(2 AS BIGINT), 2, 'M2', 2.5, true, '2024-01-02-01'), + (CAST(4 AS BIGINT), 4, 'row-4', 4.5, false, '2024-01-04-03') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN UPDATE + SET t.${Core.string0.columnName} = s.${Core.string0.columnName} + WHEN NOT MATCHED THEN INSERT * + WHEN NOT MATCHED BY SOURCE THEN DELETE""") + + assert(keyed(table.rows) == Seq(2L, 4L)) + assert( + table.rows + .find(_.get(Core.long0) == 2L) + .map(_.get(Core.string0)) + .contains("M2")) + }, + preparation.test("merge.updateStar") { table => + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES + (CAST(2 AS BIGINT), 22, 'S2', 22.5, true, '2024-06-06-06') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN UPDATE SET *""") + + val updatedRow = table.rows.find(_.get(Core.long0) == 2L) + + assert(updatedRow.map(_.get(Core.string0)).contains("S2")) + assert(updatedRow.map(_.get(Core.int0)).contains(22)) + }, + preparation.test("merge.insertExplicitColumns") { table => + val expected = + (table.preparedRows.map(_.get(Core.long0)) :+ 7L).sorted + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES (CAST(7 AS BIGINT), 'g') + AS s(${Core.long0.columnName}, ${Core.string0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN + INSERT (${Core.long0.columnName}, ${Core.string0.columnName}) + VALUES (s.${Core.long0.columnName}, s.${Core.string0.columnName})""") + + assert(keyed(table.rows) == expected) + assert( + table.rows + .find(_.get(Core.long0) == 7L) + .map(_.get(Core.string0)) + .contains("g")) + }, + preparation.test("merge.sourceCTE") { table => + val expected = + (table.preparedRows.map(_.get(Core.long0)) :+ 8L).sorted + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + WITH src AS ( + SELECT CAST(8 AS BIGINT) AS ${Core.long0.columnName} + ) + SELECT * FROM src + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN + INSERT (${Core.long0.columnName}) VALUES (s.${Core.long0.columnName})""") + + assert(keyed(table.rows) == expected) + }, + preparation.test("merge.sourceSetOp") { table => + val expected = + (table.preparedRows.map(_.get(Core.long0)) ++ Seq(8L, 9L)).sorted + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT CAST(8 AS BIGINT) AS ${Core.long0.columnName} + UNION ALL + SELECT CAST(9 AS BIGINT) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN + INSERT (${Core.long0.columnName}) VALUES (s.${Core.long0.columnName})""") + + assert(keyed(table.rows) == expected) + }, + preparation.test("merge.intoEmptyTarget") { table => + table.spark.sql(s"DELETE FROM ${table.name}") + assert(table.rows.isEmpty) + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES + (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), + (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') + AS s($cols) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN INSERT *""") + + assert(keyed(table.rows) == Seq(4L, 5L)) + }, + preparation.test("merge.nullJoinKey") { table => + val expectedStrings = longToString(table.preparedRows).map { + case (id, value) => id -> (if (id == 2) "M" else value) + } + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES (CAST(NULL AS BIGINT), 'n'), + (CAST(2 AS BIGINT), 'M') + AS s(${Core.long0.columnName}, ${Core.string0.columnName}) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN MATCHED THEN UPDATE + SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") + + assert( + keyed(table.rows) == + table.preparedRows.map(_.get(Core.long0)).sorted) + assert(longToString(table.rows) == expectedStrings) + }, + preparation.test("merge.resolveByName") { table => + val expected = + (table.preparedRows.map(_.get(Core.long0)) :+ 7L).sorted + + table.spark.sql( + s"""MERGE INTO ${table.name} t USING ( + SELECT * FROM VALUES + ('g', CAST(7 AS BIGINT), 7, 7.5, false, '2024-07-07-07') + AS s( + ${Core.string0.columnName}, + ${Core.long0.columnName}, + ${Core.int0.columnName}, + ${Core.double0.columnName}, + ${Core.boolean0.columnName}, + datepartition) + ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} + WHEN NOT MATCHED THEN INSERT *""") + + assert(keyed(table.rows) == expected) + assert( + table.rows + .find(_.get(Core.long0) == 7L) + .map(_.get(Core.string0)) + .contains("g")) + }, + preparation.test("insert.into") { table => + val expected = + (table.preparedRows.map(_.get(Core.long0)) ++ Seq(4L, 5L)).sorted + + table.spark.sql( + s"""INSERT INTO ${table.name} VALUES (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), - (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') - AS s($cols) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN NOT MATCHED THEN INSERT *""") { view => - assert(view.before.isEmpty) - assert(keyed(view.after) == Seq(4L, 5L)) - } - - // A null join key never matches, so it neither updates nor errors. - val mergeNullJoinKey: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.nullJoinKey")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES (CAST(NULL AS BIGINT), 'n'), (CAST(2 AS BIGINT), 'M') - AS s(${Core.long0.columnName}, ${Core.string0.columnName}) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") { view => - assert(keyed(view.after) == keyed(view.before)) - assert(longToString(view.after) == longToString(view.before).map { case (id, s) => id -> (if (id == 2) "M" else s) }) - } - - // INSERT * resolves columns by name even when the source lists them in a different order. - val mergeResolveByName: TableTest[CoreTable.type] = - TableTest(Core).sql("merge.resolveByName")(table => - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES ('g', CAST(7 AS BIGINT), 7, 7.5, false, '2024-07-07-07') - AS s(${Core.string0.columnName}, ${Core.long0.columnName}, ${Core.int0.columnName}, ${Core.double0.columnName}, ${Core.boolean0.columnName}, datepartition) - ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} - WHEN NOT MATCHED THEN INSERT *""") { view => - assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 7L).sorted) - assert(view.after.find(_.get(Core.long0) == 7L).map(_.get(Core.string0)).contains("g")) + (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')""") + + assert(keyed(table.rows) == expected) + }, + preparation.test("insert.explicitColumns") { table => + val exception = Check.intercept[Exception]( + table.spark.sql( + s"INSERT INTO ${table.name} " + + s"(${Core.long0.columnName}, ${Core.string0.columnName}) " + + "VALUES (CAST(4 AS BIGINT), 'd'), (CAST(5 AS BIGINT), 'e')")) + val exceptionMessage = + Option(exception.getMessage).getOrElse("") + + assert( + exceptionMessage.toUpperCase.contains("CANNOT_FIND_DATA") || + exceptionMessage.toUpperCase.contains("CANNOT FIND DATA") || + exceptionMessage.toUpperCase.contains("INCOMPATIBLE_DATA"), + "expected a partial-INSERT rejection naming the omitted column " + + s"(engine limitation), got: ${exceptionMessage.take(200)}") + }, + preparation.test("insert.intoSelect") { table => + val expected = + (table.preparedRows.map(_.get(Core.long0)) :+ 6L).sorted + + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM VALUES " + + s"(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') " + + s"AS s($cols)") + + assert(keyed(table.rows) == expected) + }, + preparation.test("append.dataFrame") { table => + val expected = + (table.preparedRows.map(_.get(Core.long0)) :+ 6L).sorted + val frame = table.spark.sql( + s"SELECT * FROM VALUES " + + s"(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') " + + s"AS s($cols)") + + frame.writeTo(table.name).append() + + assert(keyed(table.rows) == expected) + }, + preparation.test("insert.overwrite") { table => + table.spark.sql( + s"""INSERT OVERWRITE ${table.name} VALUES + (CAST(1 AS BIGINT), 1, 'p', 1.5, false, '2024-01-01-00'), + (CAST(2 AS BIGINT), 2, 'q', 2.5, true, '2024-01-02-01')""") + + assert(keyed(table.rows) == Seq(1L, 2L)) + }, + preparation.test("overwrite.dataFrame") { table => + val frame = table.spark.sql( + s"SELECT * FROM VALUES " + + s"(CAST(8 AS BIGINT), 8, 'h', 8.5, false, '2024-01-08-07') " + + s"AS s($cols)") + + frame.writeTo(table.name).overwrite( + org.apache.spark.sql.functions.lit(true)) + assert(keyed(table.rows) == Seq(8L)) + }) + + private def operationName( + testCase: Plan.Case, + preparation: TablePreparation[CoreTable.type] + ): String = + testCase.id + .split(" @ ", 2) + .head + .stripPrefix(preparation.casePrefix) + + private def localizedMutationDmlCases( + preparation: TablePreparation[CoreTable.type] + ): List[Plan.Case] = + localizedDmlCases(preparation).filter { testCase => + val caseName = operationName(testCase, preparation) + caseName.startsWith("delete.") || + caseName.startsWith("update.") || + caseName.startsWith("merge.") } - // ── insert / append / overwrite ──────────────────────────────────────────────────────── - val insertInto: TableTest[CoreTable.type] = - TableTest(Core).sql("insert.into")(table => - s"""INSERT INTO $table VALUES - (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), - (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')""") { view => - assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) ++ Seq(4L, 5L)).sorted) - } - - val appendDataFrame: TableTest[CoreTable.type] = - TableTest(Core).step("append.dataFrame") { (spark, table) => - val frame = spark.sql( - s"SELECT * FROM VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') AS s($cols)") - frame.writeTo(table).append() - } { view => - assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 6L).sorted) - } + val coreDmlCases: List[Plan.Case] = + preparedCoreTables.flatMap(localizedDmlCases) - // INSERT OVERWRITE (static mode, the Spark default) replaces the whole table regardless of state. - val insertOverwrite: TableTest[CoreTable.type] = - TableTest(Core).sql("insert.overwrite")(table => - s"""INSERT OVERWRITE $table VALUES - (CAST(1 AS BIGINT), 1, 'p', 1.5, false, '2024-01-01-00'), - (CAST(2 AS BIGINT), 2, 'q', 2.5, true, '2024-01-02-01')""") { view => - assert(keyed(view.after) == Seq(1L, 2L)) - } + val morDmlCases: List[Plan.Case] = + preparedMorCoreTables.flatMap(localizedMutationDmlCases) - val overwriteDataFrame: TableTest[CoreTable.type] = - TableTest(Core).step("overwrite.dataFrame") { (spark, table) => - val frame = spark.sql( - s"SELECT * FROM VALUES (CAST(8 AS BIGINT), 8, 'h', 8.5, false, '2024-01-08-07') AS s($cols)") - frame.writeTo(table).overwrite(org.apache.spark.sql.functions.lit(true)) - } { view => - assert(keyed(view.after) == Seq(8L)) - } - - // INSERT INTO with an explicit column list; the unlisted columns are null-filled. - // NEGATIVE PIN (was SKIP-as-bug; reclassified after code-verified investigation). A partial/named- - // column INSERT that omits other columns is REJECTED with INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA. - // This is an ENGINE limitation, not an OpenHouse policy: OpenHouse creates columns nullable-by-default - // and the server round-trips the schema verbatim (verified) — but Iceberg 1.5's SparkTable does not - // advertise column defaults (no SupportsColumnDefaultValue), so Spark's byName output resolution never - // inserts the NULL-fill projection for the omitted (nullable) columns. Pin the rejection; it flips - // only when the read+write APPLICATION of column defaults is wired (SparkTable implements - // SupportsColumnDefaultValue + the reader injects initial-default for missing columns). NOTE (fork - // audit): the com.linkedin.iceberg 1.5.2 fork #251 backported the NestedField initial/write-default - // APIs + SchemaParser serialization ONLY — no SparkTable, no reader wiring — so the fork does NOT - // satisfy the flip condition (and persists v3-style defaults on a v2 table with no gate). See - // ICEBERG-FORK-AUDIT.md. - val insertExplicitColumns: TableTest[CoreTable.type] = - TableTest(Core).step("insert.explicitColumns") { (spark, table) => - val e = Check.intercept[Exception]( - spark.sql(s"INSERT INTO $table (${Core.long0.columnName}, ${Core.string0.columnName}) " + - s"VALUES (CAST(4 AS BIGINT), 'd'), (CAST(5 AS BIGINT), 'e')")) - val msg = Option(e.getMessage).getOrElse("").toUpperCase - assert(msg.contains("CANNOT_FIND_DATA") || msg.contains("CANNOT FIND DATA") || msg.contains("INCOMPATIBLE_DATA"), - s"expected a partial-INSERT rejection naming the omitted column (engine limitation), got: ${Option(e.getMessage).getOrElse("").take(200)}") - }() - - // INSERT INTO … SELECT appends the selected rows. - val insertIntoSelect: TableTest[CoreTable.type] = - TableTest(Core).sql("insert.intoSelect")(table => - s"INSERT INTO $table SELECT * FROM VALUES " + - s"(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') AS s($cols)") { view => - assert(keyed(view.after) == (view.before.map(_.get(Core.long0)) :+ 6L).sorted) - } - - // ── partitioned-only: selective-partition replacement (meaningful only when partitioned) ── - // Seed rows 1/2/3 live in partitions '2024-01-01-00'/'01'/'02'. Writing one row into partition - // '…-00' must replace only that partition, leaving rows 2 and 3. - // Delta-sound: writing row 10 into partition '…-00' replaces ONLY that partition's rows (the - // seeded row 1), leaving every other partition's rows and adding 10. - private def onlyFirstPartitionReplaced(view: StepView[CoreTable.type]): Seq[Long] = - (view.before.filterNot(_.get(Core.datePartition) == "2024-01-01-00").map(_.get(Core.long0)) :+ 10L).sorted - - val insertDynamicOverwrite: TableTest[CoreTable.type] = - TableTest(Core).step("insert.dynamicOverwrite") { (spark, table) => - spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic") - try spark.sql(s"INSERT OVERWRITE $table VALUES (CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00')") - finally spark.conf.set("spark.sql.sources.partitionOverwriteMode", "static") - } { view => - assert(keyed(view.after) == onlyFirstPartitionReplaced(view)) - } - - val overwritePartitions: TableTest[CoreTable.type] = - TableTest(Core).step("overwrite.partitions") { (spark, table) => - val frame = spark.sql( - s"SELECT * FROM VALUES (CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00') AS s($cols)") - frame.writeTo(table).overwritePartitions() - } { view => - assert(keyed(view.after) == onlyFirstPartitionReplaced(view)) - } + val orderedDmlCases: List[Plan.Case] = + preparedOrderedCoreTables.flatMap(localizedDmlCases) - // ── create (a preparation-only test: create under the layout, assert schema + emptiness) ─ - // Also the guard that the literal `columnDefinitions` matches CoreTable's declared columns. - def createSchema(layout: Layout): TableTest[CoreTable.type] = - TableTest(Core).sql("create")(layout.create) { view => - val actual = view.spark.table(view.table).schema.fields.toList.map(field => (field.name, field.dataType.simpleString)) - val expected = Core.tableColumns.toList.map(column => (column.columnName, column.sqlType)) - assert(actual == expected) - assert(view.after.isEmpty) - } - - // ── DDL Phase 12: schema evolution — ADD COLUMN family (❓ probes settle B-vs-N) ─────────── - // The added column is not one of CoreTable's typed handles, so these assert on the LIVE schema - // (name / type / comment / order) and raw SQL, not on typed row handles. Row snapshots - // (view.before/after) still read only CoreTable's columns, so they stay valid across the ALTER. - private def liveColumns(view: StepView[CoreTable.type]): Seq[(String, String)] = - view.spark.table(view.table).schema.fields.toSeq.map(field => (field.name, field.dataType.simpleString)) - - val ddlAddColumnSingle: TableTest[CoreTable.type] = - TableTest(Core).sql("ddl.addColumn.single")(t => s"ALTER TABLE $t ADD COLUMN added_int int") { view => - assert(liveColumns(view).map(_._1).contains("added_int"), s"added_int missing: ${liveColumns(view).map(_._1)}") - val nullCount = view.spark.sql(s"SELECT count(*) FROM ${view.table} WHERE added_int IS NULL").collect()(0).getLong(0) - assert(nullCount == view.before.size, s"existing rows should read null for added_int: $nullCount != ${view.before.size}") - assert(view.after.size == view.before.size) // ADD COLUMN keeps rows + val evolvedDmlCases: List[Plan.Case] = + preparedEvolvedCoreTables.flatMap { preparation => + localizedDmlCases(preparation).filter { testCase => + val caseName = operationName(testCase, preparation) + (caseName.startsWith("delete.") || + caseName.startsWith("update.") || + caseName.startsWith("read.")) && + !caseName.contains("byNullCondition") + } } - val ddlAddColumnMultiple: TableTest[CoreTable.type] = - TableTest(Core).sql("ddl.addColumn.multiple")(t => s"ALTER TABLE $t ADD COLUMNS (added_a int, added_b string)") { view => - val names = liveColumns(view).map(_._1) - assert(names.contains("added_a") && names.contains("added_b"), s"added columns missing: $names") - assert(view.after.size == view.before.size) - } + val rtasDmlCases: List[Plan.Case] = + preparedRtasCoreTables.flatMap(localizedDmlCases) - val ddlAddColumnComment: TableTest[CoreTable.type] = - TableTest(Core).sql("ddl.addColumn.comment")(t => s"ALTER TABLE $t ADD COLUMN added_c int COMMENT 'a note'") { view => - val field = view.spark.table(view.table).schema.fields.find(_.name == "added_c") - assert(field.isDefined, "added_c missing") - assert(field.get.getComment().contains("a note"), s"comment not stored: ${field.flatMap(_.getComment())}") - } + val rtasMorDmlCases: List[Plan.Case] = + preparedRtasMorCoreTables.flatMap(localizedMutationDmlCases) - val ddlAddColumnPosition: TableTest[CoreTable.type] = - TableTest(Core).sql("ddl.addColumn.position")(t => s"ALTER TABLE $t ADD COLUMN added_after int AFTER ${Core.long0.columnName}") { view => - val names = liveColumns(view).map(_._1) - assert(names.indexOf("added_after") == names.indexOf(Core.long0.columnName) + 1, s"added_after not after long0: $names") - } + val branchDmlCases: List[Plan.Case] = + preparedBranchCoreTables.flatMap(localizedDmlCases) - val ddlAlterColumnTypeWiden: TableTest[CoreTable.type] = - TableTest(Core).sql("ddl.alterColumn.typeWiden")(t => s"ALTER TABLE $t ALTER COLUMN ${Core.int0.columnName} TYPE bigint") { view => - assert(liveColumns(view).toMap.get(Core.int0.columnName).contains("bigint"), s"int0 not widened: ${liveColumns(view).toMap.get(Core.int0.columnName)}") - val vals = view.spark.sql(s"SELECT ${Core.int0.columnName} FROM ${view.table} ORDER BY ${Core.long0.columnName}").collect().toSeq.map(_.getLong(0)) - assert(vals == Seq(1L, 2L, 3L), s"values not preserved after widening: $vals") - } + val branchMorDmlCases: List[Plan.Case] = + preparedBranchMorCoreTables.flatMap(localizedMutationDmlCases) - // RENAME COLUMN is a SILENT NO-OP on OpenHouse (tagged bug): the statement neither errors nor renames - // — verified via REFRESH TABLE + fresh DESCRIBE, the column keeps its old name. The recon predicted a - // server rejection ("not found in newSchema"), but the client drops the rename before it reaches the - // server, so nothing happens. This test asserts the CORRECT behavior (rename applies) and is tagged in - // Plan.knownBugs, so it reports SKIP until fixed. A silent no-op is worse than a clean rejection. - val ddlRenameColumn: TableTest[CoreTable.type] = - TableTest(Core) - .sql("ddl.renameColumn.seed")(t => s"ALTER TABLE $t ADD COLUMN to_rename int")() - .sql("ddl.renameColumn")(t => s"ALTER TABLE $t RENAME COLUMN to_rename TO renamed_col") { view => - val names = liveColumns(view).map(_._1) - assert(names.contains("renamed_col") && !names.contains("to_rename"), s"RENAME COLUMN silently no-oped: $names") - assert(view.after.size == view.before.size) + val morReadDmlCases: List[Plan.Case] = + preparedMorReadCoreTables.flatMap { preparation => + localizedDmlCases(preparation).filter { testCase => + val caseName = operationName(testCase, preparation) + caseName.startsWith("read.") || + caseName == "format.materialization" } - - /** Phase 12 DDL schema-evolution behaviors, crossed with every layout. */ - val ddlSchemaOperations: List[(String, TableTest[CoreTable.type])] = List( - "ddl.addColumn.single" -> ddlAddColumnSingle, - "ddl.addColumn.multiple" -> ddlAddColumnMultiple, - "ddl.addColumn.comment" -> ddlAddColumnComment, - "ddl.addColumn.position" -> ddlAddColumnPosition, - "ddl.alterColumn.typeWiden" -> ddlAlterColumnTypeWiden, - "ddl.renameColumn" -> ddlRenameColumn - ) - - /** The operations crossed with every layout, each a headless segment, in report order. */ - val operations: List[(String, TableTest[CoreTable.type])] = List( - "read.projection" -> readProjection, - "read.filter" -> readFilter, - "format.materialization" -> formatMaterialization, - "delete.byPredicate" -> deleteByPredicate, - "delete.byInList" -> deleteByInList, - "delete.byInSubquery" -> deleteByInSubquery, - "delete.byNotInSubquery" -> deleteByNotInSubquery, - "delete.byExistsSubquery" -> deleteByExistsSubquery, - "delete.byNotExistsSubquery" -> deleteByNotExistsSubquery, - "delete.byScalarSubquery" -> deleteByScalarSubquery, - "delete.byNullCondition" -> deleteByNullCondition, - "delete.all" -> deleteAll, - "delete.none" -> deleteNone, - "delete.byPartitionPredicate" -> deleteByPartitionPredicate, - "delete.withAlias" -> deleteWithAlias, - "delete.whereFalse.noSnapshot" -> deleteWhereFalseKeepsSnapshot, - "delete.truncate" -> truncate, - "delete.atSnapshot.rejected" -> deleteAtSnapshotRejected, - "update.byPredicate" -> updateByPredicate, - "update.withoutCondition" -> updateWithoutCondition, - "update.noMatch" -> updateNoMatch, - "update.byInSubquery" -> updateByInSubquery, - "update.byNotInSubquery" -> updateByNotInSubquery, - "update.byExistsSubquery" -> updateByExistsSubquery, - "update.byNotExistsSubquery" -> updateByNotExistsSubquery, - "update.byScalarSubquery" -> updateByScalarSubquery, - "update.withAlias" -> updateWithAlias, - "update.multipleColumns" -> updateMultipleColumns, - "update.byExpression" -> updateByExpression, - "update.movePartition" -> updateMovePartition, - "update.nullAssignment" -> updateNullAssignment, - "merge.insertNotMatched" -> mergeInsertNotMatched, - "merge.updateMatched" -> mergeUpdateMatched, - "merge.deleteMatched" -> mergeDeleteMatched, - "merge.upsert" -> mergeUpsert, - "merge.deleteNotMatchedBySource" -> mergeDeleteNotMatchedBySource, - "merge.conditionalUpdate" -> mergeConditionalUpdate, - "merge.multipleMatchedClauses" -> mergeMultipleMatchedClauses, - "merge.conditionalInsert" -> mergeConditionalInsert, - "merge.allClauses" -> mergeAllClauses, - "merge.updateStar" -> mergeUpdateStar, - "merge.insertExplicitColumns" -> mergeInsertExplicitColumns, - "merge.sourceCTE" -> mergeSourceCTE, - "merge.sourceSetOp" -> mergeSourceSetOp, - "merge.intoEmptyTarget" -> mergeIntoEmptyTarget, - "merge.nullJoinKey" -> mergeNullJoinKey, - "merge.resolveByName" -> mergeResolveByName, - "insert.into" -> insertInto, - "insert.explicitColumns" -> insertExplicitColumns, - "insert.intoSelect" -> insertIntoSelect, - "append.dataFrame" -> appendDataFrame, - "insert.overwrite" -> insertOverwrite, - "overwrite.dataFrame" -> overwriteDataFrame - ) - - /** Operations meaningful only on a partitioned table; crossed with the partitioned layouts only. */ - val partitionedOperations: List[(String, TableTest[CoreTable.type])] = List( - "insert.dynamicOverwrite" -> insertDynamicOverwrite, - "overwrite.partitions" -> overwritePartitions - ) - - /** The DELETE/UPDATE/MERGE subset — the operations affected by the CoW-vs-MoR mode. */ - val mutationOperations: List[(String, TableTest[CoreTable.type])] = - operations.filter { case (name, _) => - name.startsWith("delete.") || name.startsWith("update.") || name.startsWith("merge.") } + def undroppedDmlCases: List[Plan.Case] = + if (HtsAdmin.enabled) preparedUndroppedCoreTables.flatMap(localizedDmlCases) + else Nil + + private def localizedPartitionedDmlCases( + preparation: TablePreparation[CoreTable.type] + ): List[Plan.Case] = + List( + preparation.test("insert.dynamicOverwrite") { table => + val expected = + (table.preparedRows + .filterNot(_.get(Core.datePartition) == "2024-01-01-00") + .map(_.get(Core.long0)) :+ 10L).sorted + + table.spark.conf.set( + "spark.sql.sources.partitionOverwriteMode", + "dynamic") + try { + table.spark.sql( + s"INSERT OVERWRITE ${table.name} VALUES " + + "(CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00')") + } finally { + table.spark.conf.set( + "spark.sql.sources.partitionOverwriteMode", + "static") + } + + assert(keyed(table.rows) == expected) + }, + preparation.test("overwrite.partitions") { table => + val expected = + (table.preparedRows + .filterNot(_.get(Core.datePartition) == "2024-01-01-00") + .map(_.get(Core.long0)) :+ 10L).sorted + val frame = table.spark.sql( + s"SELECT * FROM VALUES " + + "(CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00') " + + s"AS s($cols)") + + frame.writeTo(table.name).overwritePartitions() + + assert(keyed(table.rows) == expected) + }) + + val partitionedDmlCases: List[Plan.Case] = + preparedCoreTables + .filter(_.label.startsWith("partitioned/")) + .flatMap(localizedPartitionedDmlCases) + + val rtasPartitionedDmlCases: List[Plan.Case] = + preparedRtasCoreTables + .filter(_.label.startsWith("partitioned/")) + .flatMap(localizedPartitionedDmlCases) + + val branchPartitionedDmlCases: List[Plan.Case] = + preparedBranchCoreTables + .filter(_.label.startsWith("partitioned/")) + .flatMap(localizedPartitionedDmlCases) + // ── MoR discriminator: prove merge-on-read actually wrote position-delete files ────────── // The rest of the MoR axis reuses CoW's row-delta assertions, which pass identically whether the // write was copy-on-write or merge-on-read. These two pin the PHYSICAL difference: a MoR delete @@ -762,18 +1130,47 @@ trait DmlScenarios extends ScenarioKit { private def deleteFileCount(spark: SparkSession, table: String): Long = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) - val morWritesDeleteFiles: TableTest[CoreTable.type] = - TableTest(Core).delete(core => s"${core.long0.columnName} < 2") { view => - assert(view.after == view.before.filterNot(_.get(Core.long0) < 2)) // rows correct - assert(deleteFileCount(view.spark, view.table) >= 1, - "merge-on-read DELETE of a strict subset of a data file must write a position-delete file") - } + val deleteFileModeCases: List[Plan.Case] = { + def cases( + layouts: List[Layout], + caseName: String, + expectDeleteFiles: Boolean): List[Plan.Case] = + layouts.map { layout => + val preparation = TablePreparation( + layout.label, + createAndSeedSingleFile(layout, 3)) + preparation.test(caseName) { table => + val rowsBefore = table.rows + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") + val rowsAfter = table.rows + val deleteFileCountAfter = + deleteFileCount(table.spark, table.name) + + assert( + rowsAfter == rowsBefore.filterNot(_.get(Core.long0) < 2), + "strict-subset DELETE returned an unexpected row set") + if (expectDeleteFiles) { + assert( + deleteFileCountAfter >= 1, + "merge-on-read DELETE should write a position-delete file") + } else { + assert( + deleteFileCountAfter == 0, + "copy-on-write DELETE should not write delete files") + } + } + } - val cowWritesNoDeleteFiles: TableTest[CoreTable.type] = - TableTest(Core).delete(core => s"${core.long0.columnName} < 2") { view => - assert(view.after == view.before.filterNot(_.get(Core.long0) < 2)) - assert(deleteFileCount(view.spark, view.table) == 0, "copy-on-write DELETE must not write delete files") - } + cases( + morVerifyLayouts, + "mor.writesDeleteFiles", + expectDeleteFiles = true) ++ + cases( + cowVerifyLayouts, + "cow.writesNoDeleteFiles", + expectDeleteFiles = false) + } } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala index 967abb7c0..afc6e56e7 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala @@ -160,7 +160,7 @@ object Main { val cases = Plan.cases.filter(c => selected(c.id)) val header = if (filters.isEmpty) "all cases" else s"filter ${filters.mkString(", ")} -> ${cases.size} cases" - println(s"\n=== delta-harness :: typed pipelines @ OpenHouse catalog ($header) ===\n") + println(s"\n=== delta-harness :: localized cases @ OpenHouse catalog ($header) ===\n") // Known-bug cases are tagged (Plan.knownBugs) and reported SKIP rather than run — deferred, // not passing. Everything else executes. diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala index d3217d8cf..2b6331554 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala @@ -189,13 +189,6 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $t") } - val forkColDefaultOps: List[(String, Ctx => Unit)] = List( - "fork.colDefault.addColumnInert @ parquet" -> forkColDefaultAddColumn("parquet"), - "fork.colDefault.addColumnInert @ orc" -> forkColDefaultAddColumn("orc"), - "fork.colDefault.apiSerialization @ core" -> forkColDefaultApiSerialization, - "fork.colDefault.readApplyProbe @ core" -> forkColDefaultReadApplyProbe - ) - // ── #249 (d69c1fd91) — partitioned write distribution default ───────────────────────────────────── // The fork changes the DEFAULT write.distribution-mode for PARTITIONED writes from Apache's HASH to // NONE (Spark 3.5). With HASH, the writer shuffles rows so each partition is written by one task -> @@ -234,11 +227,6 @@ trait ForkScenarios extends ScenarioKit { s"[$fmt] default partitioned distribution produced FEWER files than HASH (default=$nDefault hash=$nHash) — unexpected; re-audit #249") } - val forkPartitionDistOps: List[(String, Ctx => Unit)] = List( - "fork.partitionDist.default @ parquet" -> forkPartitionDistDefault("parquet"), - "fork.partitionDist.default @ orc" -> forkPartitionDistDefault("orc") - ) - // (count, sumBytes) of the CURRENT data files — used by the compaction fork probes below. private def dataFileStats(spark: SparkSession, table: String): (Long, Long) = { val r = spark.sql(s"SELECT count(*), coalesce(sum(file_size_in_bytes), 0) FROM $table.data_files").collect()(0) @@ -294,10 +282,6 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $table") } - val forkDeleteFileReplicationOps: List[(String, Ctx => Unit)] = List( - "fork.deleteFileReplication @ mor" -> forkDeleteFileReplication - ) - // ── #219 (OutputFileFactory.FILE_REPLICATION_FACTOR) — output-file replication factor ───────────────── // KEY CORRECTION: the constant is FILE_REPLICATION_FACTOR = "file-replication-factor" — NOT the guessed // "write.file-replication-factor", and it is NOT a settable table property at all. It is the per-output- @@ -354,10 +338,6 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $table") } - val forkFileReplicationFactorOps: List[(String, Ctx => Unit)] = List( - "fork.fileReplicationFactor @ core" -> forkFileReplicationFactor - ) - // ── #228 (spark.sql.iceberg.split-size) — Spark read split size ─────────────────────────────────────── // SparkSQLProperties.SPLIT_SIZE = "spark.sql.iceberg.split-size". Set via spark.conf.set; SparkReadConf // uses it to combine/split data files into read tasks. This one IS observable: with several small files, @@ -419,11 +399,6 @@ trait ForkScenarios extends ScenarioKit { } } - val forkSplitSizeOps: List[(String, Ctx => Unit)] = List( - "fork.splitSize @ parquet" -> forkSplitSize("parquet"), - "fork.splitSize @ orc" -> forkSplitSize("orc") - ) - // ── #233 (bin-pack by data-file length) — rewrite_data_files compaction ────────────────────────────── // The fork's bin-pack rewrite weights data files by their LENGTH (file_size_in_bytes) when packing them // into rewrite groups. That weighting is an internal planner detail — not locally observable via SQL — so @@ -458,11 +433,6 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $table") } - val forkBinPackByLengthOps: List[(String, Ctx => Unit)] = List( - "fork.binPackByLength @ parquet" -> forkBinPackByLength("parquet"), - "fork.binPackByLength @ orc" -> forkBinPackByLength("orc") - ) - // ── #189 (budgeted rewrite ordering by file-sequence-number) — rewrite_data_files ───────────────────── // The fork's budgeted rewrite ORDERS candidate files by their file-sequence-number when spending a rewrite // budget. The ordering decision is metadata-level and NOT locally observable via SQL, and it shares the @@ -475,7 +445,7 @@ trait ForkScenarios extends ScenarioKit { val table = s"${ctx.namespace}.t_compord" spark.sql(s"DROP TABLE IF EXISTS $table") spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$seedFmt', 'write.distribution-mode'='none')") + "TBLPROPERTIES ('write.format.default'='parquet', 'write.distribution-mode'='none')") // Several commits => several data files with DISTINCT, increasing file-sequence-numbers (the ordering key). val nCommits = 4 for (i <- 0 until nCommits) spark.sql(s"INSERT INTO $table VALUES (${i}L, 'c$i')") @@ -502,10 +472,46 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $table") } - val forkCompactionOrderOps: List[(String, Ctx => Unit)] = List( - "fork.compactionOrder @ parquet" -> forkCompactionOrder - ) - - + val forkCases: List[Plan.Case] = + List( + Plan.Case( + "fork.colDefault.addColumnInert @ parquet", + forkColDefaultAddColumn("parquet")), + Plan.Case( + "fork.colDefault.addColumnInert @ orc", + forkColDefaultAddColumn("orc")), + Plan.Case( + "fork.colDefault.apiSerialization @ core", + forkColDefaultApiSerialization), + Plan.Case( + "fork.colDefault.readApplyProbe @ core", + forkColDefaultReadApplyProbe), + Plan.Case( + "fork.partitionDist.default @ parquet", + forkPartitionDistDefault("parquet")), + Plan.Case( + "fork.partitionDist.default @ orc", + forkPartitionDistDefault("orc")), + Plan.Case( + "fork.deleteFileReplication @ mor", + forkDeleteFileReplication), + Plan.Case( + "fork.fileReplicationFactor @ core", + forkFileReplicationFactor), + Plan.Case( + "fork.splitSize @ parquet", + forkSplitSize("parquet")), + Plan.Case( + "fork.splitSize @ orc", + forkSplitSize("orc")), + Plan.Case( + "fork.binPackByLength @ parquet", + forkBinPackByLength("parquet")), + Plan.Case( + "fork.binPackByLength @ orc", + forkBinPackByLength("orc")), + Plan.Case( + "fork.compactionOrder @ parquet", + forkCompactionOrder)) } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala index caa8dab5d..82cdc2f26 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala @@ -1,33 +1,14 @@ package harness -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import org.apache.spark.sql.{Row, SparkSession} import java.time.LocalDateTime import java.time.format.DateTimeFormatter -import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -// ===================================================================================== -// Delta-test harness against the real OpenHouse catalog. -// -// A test is a TYPED PIPELINE: `TableTest[S <: Schema]`. The type parameter declares which -// table implementation the test depends on, and every step references that schema's columns -// through typed handles — so the compiler forbids mixing schemas or naming a column the -// schema doesn't declare. -// -// Preparations and operations are BOTH pipeline segments of the same schema, composed with -// `andThen`: -// * a preparation prefix (create+seed, and later RTAS / drop+undrop) yields a known state, -// * an operation suffix (delete / update / merge / insert ...) runs on that state. -// The test set is `preparations x operations`. RTAS wires into every DML test by joining the -// preparations list; no operation changes. (RTAS is not built yet — only the seam is.) -// -// Catalog wiring is copied from OpenHouseLocalServer + TestSparkSessionUtil (composed, not -// extended); no OpenHouse test is altered. -// ===================================================================================== +// The harness defines typed, reusable table preparations and localized Plan.Case bodies. +// Each case gets a fresh table, executes its preparation, runs its action and assertions, +// and drops the table during teardown. final case class Ctx(spark: SparkSession, namespace: String, restUri: String = "", restToken: String = "") @@ -246,25 +227,41 @@ final case class StepView[S <: Schema]( snapshotsAfter: Long ) -/** One pipeline step: mutate the live table, then validate it against before/after. */ +/** A fresh table after its reusable preparation has completed. */ +final case class PreparedTable[S <: Schema]( + spark: SparkSession, + name: String, + schema: S, + preparedRows: Seq[Row], + preparedSnapshotCount: Long +) { + def rows: Seq[Row] = PreparedTable.currentRows(spark, name, schema) + def snapshotCount: Long = PreparedTable.snapshotCount(spark, name) +} + +object PreparedTable { + private[harness] def currentRows[S <: Schema](spark: SparkSession, table: String, schema: S): Seq[Row] = { + val columns = schema.columnNames.mkString(", ") + spark.sql(s"SELECT $columns FROM $table ORDER BY ${schema.columnNames.head}").collect().toSeq + } + + private[harness] def snapshotCount(spark: SparkSession, table: String): Long = + spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) +} + +/** One preparation step and its validation. */ final case class Step[S <: Schema]( label: String, execute: (SparkSession, String, S) => Unit, validate: StepView[S] => Unit ) -/** - * An immutable, typed pipeline. Build a preparation prefix and an operation suffix, then - * `run` executes the steps in order on one fresh, always-dropped table, validating each step. - */ +/** An immutable, typed sequence of table-preparation steps. */ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Step[S]]) { private def add(step: Step[S]): TableTest[S] = new TableTest(schema, steps :+ step) - /** Append another same-schema pipeline (this is how prep prefixes join operation suffixes). */ - def andThen(next: TableTest[S]): TableTest[S] = new TableTest(schema, steps ++ next.steps) - // The default validator asserts the seed actually appended `numberOfRows` rows. This defends the - // relative-delta operation assertions from a vacuous pass on an empty/short baseline. + // localized assertions from a vacuous pass on an empty or short baseline. def insert(numberOfRows: Int)( validate: StepView[S] => Unit = view => assert( view.after.size == view.before.size + numberOfRows, @@ -273,36 +270,38 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste add(Step(s"insert($numberOfRows)", (spark, table, schema) => spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(schema, numberOfRows)}"), validate)) - def delete(predicate: S => String)(validate: StepView[S] => Unit = _ => ()): TableTest[S] = - add(Step("delete", (spark, table, schema) => - spark.sql(s"DELETE FROM $table WHERE ${predicate(schema)}"), validate)) - - /** General operation step: run an arbitrary mutation on the table, then validate the delta. */ + /** Run an arbitrary preparation step, then validate its result. */ def step(label: String)(mutate: (SparkSession, String) => Unit) (validate: StepView[S] => Unit = _ => ()): TableTest[S] = add(Step(label, (spark, table, _) => mutate(spark, table), validate)) - /** Operation step whose mutation is a single SQL statement (the table name is supplied). */ + /** Run one preparation SQL statement, then validate its result. */ def sql(label: String)(statement: String => String) (validate: StepView[S] => Unit = _ => ()): TableTest[S] = step(label)((spark, table) => spark.sql(statement(table)))(validate) - /** Read/assert-only step: no mutation, so before == after; used for the read paths. */ - def check(label: String)(validate: StepView[S] => Unit): TableTest[S] = - step(label)((_, _) => ())(validate) - - // Execute the pipeline on a fresh, always-dropped table. Each step's `before` is the previous - // step's `after` (an empty/zero baseline for the first step), so rows and commits are only ever - // read AFTER a step has run — on a table a prior step created. There is no existence guard: a - // query against a missing table loudly fails, which is the correct behavior. - def run(ctx: Ctx): Unit = withTable(ctx) { table => - steps.foldLeft((Seq.empty[Row], 0L)) { case ((beforeRows, beforeSnapshots), step) => - step.execute(ctx.spark, table, schema) - val afterRows = currentRows(ctx.spark, table) - val afterSnapshots = snapshotCount(ctx.spark, table) - step.validate(StepView(ctx.spark, table, schema, beforeRows, afterRows, beforeSnapshots, afterSnapshots)) - (afterRows, afterSnapshots) - } + /** + * Execute these steps as a reusable preparation, then hand the prepared table to one localized + * test body. The fresh-table lifecycle covers both the preparation and the test body. + */ + def prepare(ctx: Ctx)(use: PreparedTable[S] => Unit): Unit = withTable(ctx) { table => + val (preparedRows, preparedSnapshotCount) = + steps.foldLeft((Seq.empty[Row], 0L)) { case ((beforeRows, beforeSnapshots), step) => + step.execute(ctx.spark, table, schema) + val afterRows = PreparedTable.currentRows(ctx.spark, table, schema) + val afterSnapshots = PreparedTable.snapshotCount(ctx.spark, table) + step.validate( + StepView( + ctx.spark, + table, + schema, + beforeRows, + afterRows, + beforeSnapshots, + afterSnapshots)) + (afterRows, afterSnapshots) + } + use(PreparedTable(ctx.spark, table, schema, preparedRows, preparedSnapshotCount)) } // The one table-lifecycle primitive: hand `use` a fresh table name and always drop it afterward. @@ -314,19 +313,25 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste finally try ctx.spark.sql(s"DROP TABLE IF EXISTS $table") catch { case NonFatal(_) => () } } - // Rows selected by the schema's columns, ordered by the key (first) column for deterministic - // comparison. Ordering by the key (not all columns) keeps this valid for schemas with columns - // that aren't orderable, e.g. a map. - private def currentRows(spark: SparkSession, table: String): Seq[Row] = { - val columns = schema.columnNames.mkString(", ") - spark.sql(s"SELECT $columns FROM $table ORDER BY ${schema.columnNames.head}").collect().toSeq - } - - private def snapshotCount(spark: SparkSession, table: String): Long = - spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) } object TableTest { private val counter = new java.util.concurrent.atomic.AtomicInteger(0) def apply[S <: Schema](schema: S): TableTest[S] = new TableTest(schema, Vector.empty) } + +/** An immutable recipe that prepares one fresh table for each localized test case. */ +final case class TablePreparation[S <: Schema]( + label: String, + preparation: TableTest[S], + casePrefix: String = "", + afterTest: PreparedTable[S] => Unit = (_: PreparedTable[S]) => () +) { + def test(caseName: String)(body: PreparedTable[S] => Unit): Plan.Case = + Plan.Case( + s"$casePrefix$caseName @ $label", + context => preparation.prepare(context) { table => + body(table) + afterTest(table) + }) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala index ccfbd13d3..5a434f1e8 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala @@ -13,292 +13,757 @@ import scala.util.control.NonFatal trait HazardReaderWriterScenarios extends ScenarioKit { import Rows._ - val hazardStreamExpiredCheckpoint: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("hazard.stream.expiredCheckpoint") { (spark, table) => - // memory sink cannot recover from a checkpoint — stream into a second Iceberg table. - val dst = s"${table}_sink" - spark.sql(s"DROP TABLE IF EXISTS $dst") - spark.sql(coreCreateParquet(dst)) - val ckpt = java.nio.file.Files.createTempDirectory("ck-hazard").toString - def runStream(): Unit = { - val q = spark.readStream.table(table) - .writeStream.format("iceberg").outputMode("append") - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", ckpt).toTable(dst) - assert(q.awaitTermination(120000), "stream did not finish"); q.stop() - } - try { - runStream() // act 1: offset -> s1 - assert(countOf(spark, s"SELECT count(*) FROM $dst") == "3", "initial stream delivered the seed") - spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") // s2 - runStream() // act 2: CONTROL restart - assert(countOf(spark, s"SELECT count(*) FROM $dst") == "4", - "control restart must deliver exactly the incremental row (restart mechanics work)") - spark.sql(s"INSERT INTO $table VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") // s3 - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - // act 3: the checkpointed offset (s2) is expired -> restart bricked, typed. - val e = Check.intercept[Exception](runStream()) - assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(m => - m.contains("expired or removed") || m.contains("Cannot load current offset") || m.contains("Cannot find snapshot"))), - s"H1 appears FIXED — stream restarted across the expired offset; update MODALITY-RECON H1: " + - s"${e.getClass.getName} ${Option(e.getMessage).getOrElse("").take(200)}") - } finally spark.sql(s"DROP TABLE IF EXISTS $dst") - }() - - // H2 — CDC/changelog over expired lineage: expired explicit bound → hard typed error; - // timestamp bound → SILENT under-report (the truth was 5 changes; the view shows fewer). - val hazardCdcExpiredRange: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() // s1: 3 rows - .step("hazard.cdc.expiredRange") { (spark, table) => - spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") // s2 - spark.sql(s"INSERT INTO $table VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") // s3 - val snaps = snapshotIds(spark, table) - val ts0 = spark.sql(s"SELECT committed_at FROM $table.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getTimestamp(0) - val tsMid = spark.sql(s"SELECT committed_at FROM $table.snapshots WHERE snapshot_id = ${snaps(1)}").collect()(0).getTimestamp(0) - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - // Characterize each bound placement over the punctured lineage. FULL truth would mean fixed. - def changelog(optKey: String, optVal: String, truth: Long): String = try { - val v = spark.sql( - s"CALL openhouse.system.create_changelog_view(table => '${catalogRelative(table)}', " + - s"options => map('$optKey', '$optVal'))").collect()(0).getString(0) - val n = spark.sql(s"SELECT count(*) FROM $v").collect()(0).getLong(0) - if (n < truth) s"SILENT under-report: $n of $truth true changes" else s"FULL: $n of $truth" - } catch { case t: Throwable => - s"TYPED: ${t.getClass.getSimpleName} :: ${Option(t.getMessage).getOrElse("").take(140)}" } - val a = changelog("start-snapshot-id", snaps.head.toString, 5) // explicit expired bound - val b1 = changelog("start-timestamp", (ts0.getTime - 1000).toString, 5) // before all history - val b2 = changelog("start-timestamp", (tsMid.getTime - 1).toString, 2) // mid-history, expired region - println(s"DIAG cdc.explicitExpiredId: $a") - println(s"DIAG cdc.tsBeforeHistory: $b1") - println(s"DIAG cdc.tsMidExpired: $b2") - Seq("explicitId" -> a, "tsBeforeHistory" -> b1, "tsMidExpired" -> b2).foreach { case (k, o) => - assert(!o.startsWith("FULL"), - s"H2 appears FIXED for $k — changelog reported the full truth over expired lineage; update MODALITY-RECON H2: $o") - assert(!o.toLowerCase.contains("expir"), - s"H2 error now NAMES expiration for $k (readability improved) — update MODALITY-RECON H2/Audit B: $o") - } - }() - - // H3 — RTAS wipes column tags (same policies plane as G10) and column comments (new schema from SELECT). - val hazardRtasWipesColumnTags: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("enableReplace")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('replace.enabled'='true')")() - .sql("tagPii")(t => s"ALTER TABLE $t MODIFY COLUMN ${Core.string0.columnName} SET TAG = (PII)")() - .step("hazard.rtas.wipesColumnTags") { (spark, table) => - spark.sql(s"ALTER TABLE $table ALTER COLUMN ${Core.string0.columnName} COMMENT 'contains-pii'") - val before = tableProps(spark, table).getOrElse("policies", "") - assert(before.toLowerCase.contains("pii") || before.toLowerCase.contains("columntags"), - s"PII tag not stored in policies before replace: '$before'") - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - val after = tableProps(spark, table).getOrElse("policies", "") - assert(!(after.toLowerCase.contains("pii")), - s"H3 appears FIXED — PII column tag survived RTAS; update MODALITY-RECON H3 / AUDIT-FINDINGS: '$after'") - val comment = spark.sql(s"DESCRIBE TABLE $table").collect().toSeq - .find(_.getString(0) == Core.string0.columnName).map(_.getString(2)).getOrElse("") - println(s"DIAG rtas.columnComment after replace: '${comment}' (was 'contains-pii')") - }() - - // H5 — retention × branches: the DEFENDED path (positive invariant): main-side TTL delete + - // expiration + orphan removal leave a live branch fully readable. - val hazardRetentionBranchDefended: TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource PARTITIONED BY (${Core.datePartition.columnName}) TBLPROPERTIES ('write.format.default'='$seedFmt')")() - .insert(3)() - .step("hazard.retentionBranch.defended") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH rbb") - spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} <= 2") // retention-shaped main delete - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - spark.sql(s"CALL openhouse.system.remove_orphan_files(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2020-01-01 00:00:00')") - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'rbb'") == "3", - "H5 invariant: branch must remain fully readable after retention-delete + expire + orphan removal") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "1", "main reflects the TTL delete") - }() - - // H6 — rename × consumers: metadata continuity (branch refs, history, writability survive rename). - val hazardRenameConsumers: TableTest[CoreTable.type] = - coreTwoSnapshots.step("hazard.rename.consumers") { (spark, table) => - val snaps = snapshotIds(spark, table) - spark.sql(s"ALTER TABLE $table CREATE BRANCH rnb") - spark.sql(s"INSERT INTO $table.branch_rnb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val renamed = s"${table}_rn" - spark.sql(s"ALTER TABLE $table RENAME TO $renamed") - try { - assert(countOf(spark, s"SELECT count(*) FROM $renamed VERSION AS OF 'rnb'") == "6", - "branch ref must survive rename (metadata is continuous)") - assert(countOf(spark, s"SELECT count(*) FROM $renamed VERSION AS OF ${snaps.head}") == "3", - "time travel must survive rename (same snapshot log)") - spark.sql(s"INSERT INTO $renamed VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert(countOf(spark, s"SELECT count(*) FROM $renamed") == "6", "renamed table writable") - } finally spark.sql(s"ALTER TABLE $renamed RENAME TO $table") // restore for teardown - }() - - // H7 — wap.enabled=false does NOT strand named branches (only staged wap.id snapshots — G4). - val hazardWapToggleBranchesSurvive: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("hazard.wapToggle.branchesSurvive") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH wtb") - spark.sql(s"INSERT INTO $table.branch_wtb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='false')") - spark.sql(s"INSERT INTO $table.branch_wtb VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'wtb'") == "5", - "named branches must survive the WAP toggle (branch surface is not wap-gated)") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "main untouched") - }() - - // H8 — ADD COLUMN breaks every existing explicit-column writer (composition with the - // partial-INSERT rejection): schema evolution is NOT writer-backward-compatible here, - // contrary to ANSI SQL (omitted columns default to NULL). - val hazardAddColumnBreaksWriters: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("hazard.addColumn.breaksWriters") { (spark, table) => - val allCols = Core.tableColumns.map(_.columnName).mkString(", ") - val writerStatement = s"INSERT INTO $table ($allCols) VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')" - spark.sql(writerStatement) // the fleet's writer: green today - assert(countOf(spark, s"SELECT count(*) FROM $table") == "4", "writer works pre-evolution") - spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") - val e = Check.intercept[AnalysisException](spark.sql(writerStatement)) // IDENTICAL statement - assert(e.getMessage.contains("extra_col") && - (e.getMessage.contains("CANNOT_FIND_DATA") || e.getMessage.toLowerCase.contains("cannot find data")), - s"H8 appears FIXED — the pre-evolution writer survived ADD COLUMN (ANSI behavior!); update MODALITY-RECON H8 and BUGS.md: ${e.getMessage.take(200)}") - }() - - // ── Reader × writer-class battery (BUILD-STATUS task #4) ───────────────────────────────────── - // A reader (CDC changelog / incremental read / streaming) must correctly REPRESENT each writer - // class (append / overwrite / delete / update / merge), and the physical mode (CoW vs MoR) must - // not change what the reader reports. Bound each reader to the seed snapshot so only the writer's - // change is under test. Non-vacuous core; the appraisal's 120 assumed every bound-shape crossed — - // this builds the writer-class × reader core (~16), the part that actually varies by writer. - // Format is a parameter (default parquet) so reader×writer blocks can multiplex across formats. private def cowCreate(t: String, fmt: String): String = s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')" private def cowCreate(t: String): String = cowCreate(t, "parquet") private def morCreate(t: String, fmt: String): String = s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (${morPropsFmt(fmt)})" - private def morCreate(t: String): String = morCreate(t, "parquet") - - private val writerClasses: List[(String, String => String)] = List( - "append" -> (t => s"INSERT INTO $t VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')"), - "overwrite" -> (t => s"INSERT OVERWRITE $t SELECT * FROM $t WHERE ${Core.long0.columnName} <= 2"), - "delete" -> (t => s"DELETE FROM $t WHERE ${Core.long0.columnName} = 1"), - "update" -> (t => s"UPDATE $t SET ${Core.string0.columnName} = 'upd' WHERE ${Core.long0.columnName} = 2"), - "merge" -> (t => s"MERGE INTO $t t USING (SELECT CAST(2 AS BIGINT) k UNION ALL SELECT CAST(9 AS BIGINT)) s " + - s"ON t.${Core.long0.columnName} = s.k WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + - s"WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}, ${Core.int0.columnName}, ${Core.string0.columnName}, " + - s"${Core.double0.columnName}, ${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + - s"VALUES (s.k, 9, 'row-9', 9.5, true, '2024-01-09-01')") - ) - - // CDC changelog must represent each writer class; assert the defining change-type + print the map. - private def changelogWriterTest(cls: String, mor: Boolean, fmt: String): TableTest[CoreTable.type] = - TableTest(Core).sql("create")(t => if (mor) morCreate(t, fmt) else cowCreate(t, fmt))().insert(3)() - .step(s"readerWriter.changelog.$cls${if (mor) ".mor" else ""}") { (spark, table) => - val s0 = snapshotIds(spark, table).head - spark.sql(writerClasses.toMap.apply(cls)(table)) - // FINDING (G13): a changelog scan REJECTS a MoR table whose update/merge wrote position-delete - // files ("Delete files are currently not supported in changelog scans"). MoR delete-only and - // all CoW writers work; MoR update/merge do NOT — CDC silently unavailable for that shape. - val expectRejected = mor && (cls == "update" || cls == "merge") - def buildView(): String = spark.sql( - s"CALL openhouse.system.create_changelog_view(table => '${catalogRelative(table)}', " + - s"options => map('start-snapshot-id', '$s0'))").collect()(0).getString(0) - if (expectRejected) { - val e = Check.intercept[Exception] { val v = buildView(); spark.sql(s"SELECT * FROM $v").collect() } - assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(_.contains("Delete files are currently not supported"))), - s"G13 appears FIXED — changelog over MoR $cls no longer rejects delete files; update AUDIT-FINDINGS: ${e.getMessage.take(160)}") - println(s"DIAG changelog.$cls.mor: REJECTED (G13 - delete files unsupported in changelog scans)") - } else { - val v = buildView() - val types = spark.sql(s"SELECT _change_type, count(*) AS c FROM $v GROUP BY _change_type") - .collect().toSeq.map(r => r.getString(0) -> r.getLong(1)).toMap - println(s"DIAG changelog.$cls${if (mor) ".mor" else ""}: $types") - cls match { - case "append" => assert(types.getOrElse("INSERT", 0L) == 1 && !types.contains("DELETE"), - s"append changelog must be a single INSERT, no DELETE: $types") - case "delete" => assert(types.getOrElse("DELETE", 0L) == 1 && !types.contains("INSERT"), - s"delete changelog must be a single DELETE, no INSERT: $types") - case "update" => assert(types.getOrElse("DELETE", 0L) >= 1 && types.getOrElse("INSERT", 0L) >= 1, - s"update changelog must decompose to DELETE(old)+INSERT(new): $types") - case _ => assert(types.values.sum >= 1, s"$cls changelog must be non-empty: $types") + + val readerWriterCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + val cowPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => cowCreate(table, format))() + .insert(3)()) + val morPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => morCreate(table, format))() + .insert(3)()) + + List( + cowPreparation.test("readerWriter.changelog.append") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.append: $changeTypes") + assert( + changeTypes.getOrElse("INSERT", 0L) == 1 && + !changeTypes.contains("DELETE"), + s"append changelog must contain one INSERT and no DELETE: $changeTypes") + }, + morPreparation.test("readerWriter.changelog.append.mor") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.append.mor: $changeTypes") + assert( + changeTypes.getOrElse("INSERT", 0L) == 1 && + !changeTypes.contains("DELETE"), + s"MoR append changelog must contain one INSERT and no DELETE: $changeTypes") + }, + cowPreparation.test("readerWriter.changelog.overwrite") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT OVERWRITE ${table.name} " + + s"SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.overwrite: $changeTypes") + assert( + changeTypes.values.sum >= 1, + s"overwrite changelog must be non-empty: $changeTypes") + }, + morPreparation.test("readerWriter.changelog.overwrite.mor") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT OVERWRITE ${table.name} " + + s"SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.overwrite.mor: $changeTypes") + assert( + changeTypes.values.sum >= 1, + s"MoR overwrite changelog must be non-empty: $changeTypes") + }, + cowPreparation.test("readerWriter.changelog.delete") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.delete: $changeTypes") + assert( + changeTypes.getOrElse("DELETE", 0L) == 1 && + !changeTypes.contains("INSERT"), + s"delete changelog must contain one DELETE and no INSERT: $changeTypes") + }, + morPreparation.test("readerWriter.changelog.delete.mor") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.delete.mor: $changeTypes") + assert( + changeTypes.getOrElse("DELETE", 0L) == 1 && + !changeTypes.contains("INSERT"), + s"MoR delete changelog must contain one DELETE and no INSERT: $changeTypes") + }, + cowPreparation.test("readerWriter.changelog.update") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.update: $changeTypes") + assert( + changeTypes.getOrElse("DELETE", 0L) >= 1 && + changeTypes.getOrElse("INSERT", 0L) >= 1, + s"update changelog must decompose to DELETE and INSERT: $changeTypes") + }, + morPreparation.test("readerWriter.changelog.update.mor") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2") + val exception = Check.intercept[Exception] { + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + table.spark.sql(s"SELECT * FROM $view").collect() } + + assert( + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage) + .exists(_.contains("Delete files are currently not supported"))), + "MoR update changelog should reject position-delete files") + println( + "DIAG changelog.update.mor: " + + "REJECTED (delete files unsupported in changelog scans)") + }, + cowPreparation.test("readerWriter.changelog.merge") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"MERGE INTO ${table.name} target " + + "USING (SELECT CAST(2 AS BIGINT) key " + + "UNION ALL SELECT CAST(9 AS BIGINT)) source " + + s"ON target.${Core.long0.columnName} = source.key " + + s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + + "WHEN NOT MATCHED THEN INSERT " + + s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + + s"${Core.string0.columnName}, ${Core.double0.columnName}, " + + s"${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + + "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.merge: $changeTypes") + assert( + changeTypes.values.sum >= 1, + s"merge changelog must be non-empty: $changeTypes") + }, + morPreparation.test("readerWriter.changelog.merge.mor") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"MERGE INTO ${table.name} target " + + "USING (SELECT CAST(2 AS BIGINT) key " + + "UNION ALL SELECT CAST(9 AS BIGINT)) source " + + s"ON target.${Core.long0.columnName} = source.key " + + s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + + "WHEN NOT MATCHED THEN INSERT " + + s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + + s"${Core.string0.columnName}, ${Core.double0.columnName}, " + + s"${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + + "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") + val exception = Check.intercept[Exception] { + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + table.spark.sql(s"SELECT * FROM $view").collect() + } + + assert( + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage) + .exists(_.contains("Delete files are currently not supported"))), + "MoR merge changelog should reject position-delete files") + println( + "DIAG changelog.merge.mor: " + + "REJECTED (delete files unsupported in changelog scans)") + }, + cowPreparation.test("readerWriter.incremental.append") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.append: added=$addedRowCount") + assert( + addedRowCount == 1, + s"append incremental scan should contain one row, got $addedRowCount") + }, + cowPreparation.test("readerWriter.incremental.delete") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.delete: added=$addedRowCount") + assert( + addedRowCount >= 0, + s"delete incremental scan returned $addedRowCount") + }, + cowPreparation.test("readerWriter.incremental.overwrite") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT OVERWRITE ${table.name} " + + s"SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.overwrite: added=$addedRowCount") + assert( + addedRowCount >= 0, + s"overwrite incremental scan returned $addedRowCount") + }, + cowPreparation.test("readerWriter.incremental.update") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.update: added=$addedRowCount") + assert( + addedRowCount >= 0, + s"update incremental scan returned $addedRowCount") + }, + cowPreparation.test("readerWriter.stream.append") { table => + val destination = s"${table.name}_s" + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + table.spark.sql(cowCreate(destination, format)) + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-rw").toString + def runStream(): Unit = { + val query = table.spark.readStream + .table(table.name) + .writeStream + .format("iceberg") + .outputMode("append") + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .toTable(destination) + assert(query.awaitTermination(120000), "stream did not finish") + query.stop() + } + + try { + runStream() + assert( + countOf(table.spark, s"SELECT count(*) FROM $destination") == "3", + "initial stream did not deliver the seed") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + runStream() + assert( + countOf(table.spark, s"SELECT count(*) FROM $destination") == "4", + "stream restart did not deliver the appended row") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + } + }, + cowPreparation.test("readerWriter.stream.deleteRejected") { table => + val destination = s"${table.name}_sd" + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + table.spark.sql(cowCreate(destination, format)) + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-rwd").toString + def runStream(): Unit = { + val query = table.spark.readStream + .table(table.name) + .writeStream + .format("iceberg") + .outputMode("append") + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .toTable(destination) + assert(query.awaitTermination(120000), "stream did not finish") + query.stop() + } + + try { + runStream() + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val exception = Check.intercept[Exception](runStream()) + + println( + "DIAG stream.afterDelete: " + + s"${exception.getClass.getSimpleName} :: " + + Option(exception.getMessage).getOrElse("").take(140)) + assert( + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage).exists(message => + message.toLowerCase.contains("delete") || + message.toLowerCase.contains("overwrite"))), + "append-only stream should reject a delete snapshot") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + } + }) + } + + private def localizedHazardCases(format: String): List[Plan.Case] = { + val basePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => cowCreate(table, format))() + .insert(3)()) + val taggedReplacePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => cowCreate(table, format))() + .insert(3)() + .sql("enableReplace")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")() + .sql("tagPii")(table => + s"ALTER TABLE $table MODIFY COLUMN " + + s"${Core.string0.columnName} SET TAG = (PII)")()) + val partitionedPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"PARTITIONED BY (${Core.datePartition.columnName}) " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + val twoSnapshotPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => cowCreate(table, format))() + .insert(3)() + .sql("insertMore")(table => + s"INSERT INTO $table VALUES " + + "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + + "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) + val wapPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => cowCreate(table, format))() + .insert(3)() + .sql("enableWap")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")()) + + List( + basePreparation.test("hazard.stream.expiredCheckpoint") { table => + val destination = s"${table.name}_sink" + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + table.spark.sql(cowCreate(destination, format)) + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-hazard").toString + def runStream(): Unit = { + val query = table.spark.readStream + .table(table.name) + .writeStream + .format("iceberg") + .outputMode("append") + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .toTable(destination) + assert(query.awaitTermination(120000), "stream did not finish") + query.stop() } - }() - - // Incremental read (append scan) must reflect the writer: appends add rows; a delete/overwrite - // changes the incremental row set. Bound start=seed. - private def incrementalWriterTest(cls: String, fmt: String): TableTest[CoreTable.type] = - TableTest(Core).sql("create")(t => cowCreate(t, fmt))().insert(3)() - .step(s"readerWriter.incremental.$cls") { (spark, table) => - val s0 = snapshotIds(spark, table).head - spark.sql(writerClasses.toMap.apply(cls)(table)) - val s1 = snapshotIds(spark, table).last - val added = spark.read.format("iceberg").option("start-snapshot-id", s0).option("end-snapshot-id", s1) - .load(table).count() - println(s"DIAG incremental.$cls: added=$added") - cls match { - case "append" => assert(added == 1, s"append incremental must scan the 1 appended row: $added") - case _ => assert(added >= 0, s"$cls incremental read must not error: $added") - } - }() - - // Streaming read must represent the writer: an append is delivered; a delete/overwrite snapshot is - // rejected by the stream unless streaming-skip-* is set (characterize the two paths). - def readerWriterStreamAppend(fmt: String): TableTest[CoreTable.type] = - TableTest(Core).sql("create")(t => cowCreate(t, fmt))().insert(3)() - .step("readerWriter.stream.append") { (spark, table) => - val dst = s"${table}_s"; spark.sql(s"DROP TABLE IF EXISTS $dst"); spark.sql(cowCreate(dst, fmt)) - val ckpt = java.nio.file.Files.createTempDirectory("ck-rw").toString - def run(): Unit = { val q = spark.readStream.table(table).writeStream.format("iceberg") - .outputMode("append").trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", ckpt).toTable(dst); assert(q.awaitTermination(120000)); q.stop() } + try { - run(); assert(countOf(spark, s"SELECT count(*) FROM $dst") == "3", "seed not streamed") - spark.sql(writerClasses.toMap.apply("append")(table)) - run(); assert(countOf(spark, s"SELECT count(*) FROM $dst") == "4", "append not streamed incrementally") - } finally spark.sql(s"DROP TABLE IF EXISTS $dst") - }() - - def readerWriterStreamDelete(fmt: String): TableTest[CoreTable.type] = - TableTest(Core).sql("create")(t => cowCreate(t, fmt))().insert(3)() - .step("readerWriter.stream.deleteRejected") { (spark, table) => - val dst = s"${table}_sd"; spark.sql(s"DROP TABLE IF EXISTS $dst"); spark.sql(cowCreate(dst, fmt)) - val ckpt = java.nio.file.Files.createTempDirectory("ck-rwd").toString - def run(): Unit = { val q = spark.readStream.table(table).writeStream.format("iceberg") - .outputMode("append").trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", ckpt).toTable(dst); assert(q.awaitTermination(120000)); q.stop() } + runStream() + assert( + countOf( + table.spark, + s"SELECT count(*) FROM $destination") == "3", + "initial stream should deliver the seed") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + runStream() + assert( + countOf( + table.spark, + s"SELECT count(*) FROM $destination") == "4", + "control restart should deliver one incremental row") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + val exception = Check.intercept[Exception](runStream()) + + assert( + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage).exists(message => + message.contains("expired or removed") || + message.contains("Cannot load current offset") || + message.contains("Cannot find snapshot"))), + "stream restart should report the expired checkpoint offset") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + } + }, + basePreparation.test("hazard.cdc.expiredRange") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + val snapshots = snapshotIds(table.spark, table.name) + val firstTimestamp = table.spark + .sql( + s"SELECT committed_at FROM ${table.name}.snapshots " + + "ORDER BY committed_at LIMIT 1") + .collect()(0) + .getTimestamp(0) + val middleTimestamp = table.spark + .sql( + s"SELECT committed_at FROM ${table.name}.snapshots " + + s"WHERE snapshot_id = ${snapshots(1)}") + .collect()(0) + .getTimestamp(0) + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + def changelog( + optionKey: String, + optionValue: String, + trueChangeCount: Long): String = + try { + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('$optionKey', '$optionValue'))") + .collect()(0) + .getString(0) + val actualChangeCount = table.spark + .sql(s"SELECT count(*) FROM $view") + .collect()(0) + .getLong(0) + if (actualChangeCount < trueChangeCount) { + s"SILENT under-report: $actualChangeCount of " + + s"$trueChangeCount true changes" + } else { + s"FULL: $actualChangeCount of $trueChangeCount" + } + } catch { + case exception: Throwable => + s"TYPED: ${exception.getClass.getSimpleName} :: " + + Option(exception.getMessage).getOrElse("").take(140) + } + val explicitSnapshotOutcome = + changelog("start-snapshot-id", snapshots.head.toString, 5) + val beforeHistoryOutcome = + changelog( + "start-timestamp", + (firstTimestamp.getTime - 1000).toString, + 5) + val middleHistoryOutcome = + changelog( + "start-timestamp", + (middleTimestamp.getTime - 1).toString, + 2) + + println(s"DIAG cdc.explicitExpiredId: $explicitSnapshotOutcome") + println(s"DIAG cdc.tsBeforeHistory: $beforeHistoryOutcome") + println(s"DIAG cdc.tsMidExpired: $middleHistoryOutcome") + Seq( + "explicitId" -> explicitSnapshotOutcome, + "tsBeforeHistory" -> beforeHistoryOutcome, + "tsMidExpired" -> middleHistoryOutcome).foreach { + case (label, outcome) => + assert( + !outcome.startsWith("FULL"), + s"expired-lineage changelog returned full truth for $label") + assert( + !outcome.toLowerCase.contains("expir"), + s"expired-lineage message now names expiration for $label") + } + }, + taggedReplacePreparation.test( + "hazard.rtas.wipesColumnTags") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN " + + s"${Core.string0.columnName} COMMENT 'contains-pii'") + val policiesBefore = + tableProps(table.spark, table.name).getOrElse("policies", "") + assert( + policiesBefore.toLowerCase.contains("pii") || + policiesBefore.toLowerCase.contains("columntags"), + s"PII tag was not stored before RTAS: $policiesBefore") + + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val policiesAfter = + tableProps(table.spark, table.name).getOrElse("policies", "") + val comment = table.spark + .sql(s"DESCRIBE TABLE ${table.name}") + .collect() + .find(_.getString(0) == Core.string0.columnName) + .map(_.getString(2)) + .getOrElse("") + + assert( + !policiesAfter.toLowerCase.contains("pii"), + s"PII column tag survived RTAS: $policiesAfter") + println( + s"DIAG rtas.columnComment after replace: '$comment' " + + "(was 'contains-pii')") + }, + partitionedPreparation.test( + "hazard.retentionBranch.defended") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH rbb") + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} <= 2") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + table.spark.sql( + "CALL openhouse.system.remove_orphan_files(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2020-01-01 00:00:00')") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rbb'") == "3", + "branch should remain readable after retention cleanup") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "1", + "main should reflect the retention-shaped delete") + }, + twoSnapshotPreparation.test("hazard.rename.consumers") { table => + val snapshots = snapshotIds(table.spark, table.name) + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH rnb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_rnb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val renamedTable = s"${table.name}_rn" + table.spark.sql( + s"ALTER TABLE ${table.name} RENAME TO $renamedTable") try { - run() // consume the seed - spark.sql(writerClasses.toMap.apply("delete")(table)) // a delete snapshot - val e = Check.intercept[Exception](run()) - println(s"DIAG stream.afterDelete: ${e.getClass.getSimpleName} :: ${Option(e.getMessage).getOrElse("").take(140)}") - assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(m => - m.toLowerCase.contains("delete") || m.toLowerCase.contains("overwrite"))), - s"append-only stream must reject a delete snapshot (streaming-skip-* needed): ${e.getMessage.take(140)}") - } finally spark.sql(s"DROP TABLE IF EXISTS $dst") - }() - - def readerWriterOps(fmt: String): List[(String, TableTest[CoreTable.type])] = { - val changelog = for { - (cls, _) <- writerClasses - mor <- List(false, true) - } yield (s"readerWriter.changelog.$cls${if (mor) ".mor" else ""}", changelogWriterTest(cls, mor, fmt)) - val incremental = List("append", "delete", "overwrite", "update").map(c => - (s"readerWriter.incremental.$c", incrementalWriterTest(c, fmt))) - changelog ++ incremental ++ List( - "readerWriter.stream.append" -> readerWriterStreamAppend(fmt), - "readerWriter.stream.deleteRejected" -> readerWriterStreamDelete(fmt)) + assert( + countOf( + table.spark, + s"SELECT count(*) FROM $renamedTable " + + "VERSION AS OF 'rnb'") == "6", + "branch should survive table rename") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM $renamedTable " + + s"VERSION AS OF ${snapshots.head}") == "3", + "time travel should survive table rename") + + table.spark.sql( + s"INSERT INTO $renamedTable VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM $renamedTable") == "6", + "renamed table should remain writable") + } finally { + table.spark.sql( + s"ALTER TABLE $renamedTable RENAME TO ${table.name}") + } + }, + wapPreparation.test("hazard.wapToggle.branchesSurvive") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH wtb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_wtb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='false')") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_wtb VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'wtb'") == "5", + "named branch should survive disabling WAP") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "branch writes should leave main unchanged") + }, + basePreparation.test("hazard.addColumn.breaksWriters") { table => + val allColumns = + Core.tableColumns.map(_.columnName).mkString(", ") + val writerStatement = + s"INSERT INTO ${table.name} ($allColumns) VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')" + table.spark.sql(writerStatement) + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "4", + "explicit-column writer should work before schema evolution") + + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + val exception = Check.intercept[AnalysisException]( + table.spark.sql(writerStatement)) + assert( + exception.getMessage.contains("extra_col") && + (exception.getMessage.contains("CANNOT_FIND_DATA") || + exception.getMessage.toLowerCase.contains("cannot find data")), + "pre-evolution explicit-column writer should fail after ADD COLUMN") + }) } - val hazardOps: List[(String, TableTest[CoreTable.type])] = List( - "hazard.stream.expiredCheckpoint" -> hazardStreamExpiredCheckpoint, - "hazard.cdc.expiredRange" -> hazardCdcExpiredRange, - "hazard.rtas.wipesColumnTags" -> hazardRtasWipesColumnTags, - "hazard.retentionBranch.defended" -> hazardRetentionBranchDefended, - "hazard.rename.consumers" -> hazardRenameConsumers, - "hazard.wapToggle.branchesSurvive" -> hazardWapToggleBranchesSurvive, - "hazard.addColumn.breaksWriters" -> hazardAddColumnBreaksWriters - ) + val hazardCases: List[Plan.Case] = + List("parquet", "orc").flatMap(localizedHazardCases) // H4 — lock starves maintenance (needs the REST lock → Ctx-based). The same gate G2 shows the // replace path SKIPS is hit by every maintenance commit: upkeep is blocked, replacement is not. @@ -333,8 +798,10 @@ trait HazardReaderWriterScenarios extends ScenarioKit { } } - val hazardCtxOps: List[(String, Ctx => Unit)] = List( - "hazard.lock.starvesMaintenance" -> hazardLockStarvesMaintenance - ) + val hazardContextCases: List[Plan.Case] = + List( + Plan.Case( + "hazard.lock.starvesMaintenance @ embedded", + hazardLockStarvesMaintenance)) } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala index 3f227548c..fcc5af782 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala @@ -14,426 +14,896 @@ trait InteractionScenarios extends ScenarioKit { import Rows._ - // ── DDL × history ────────────────────────────────────────────────────────────────────────── - val interactTtAfterAddColumn: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("interact.ddl.ttAfterAddColumn") { (spark, table) => - val s0 = snapshotIds(spark, table).last - spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") - spark.sql(s"INSERT INTO $table VALUES $extraColInsert9") - val current = spark.sql(s"SELECT * FROM $table LIMIT 1").columns.toSeq - val travel = spark.sql(s"SELECT * FROM $table VERSION AS OF $s0 LIMIT 1").columns.toSeq - assert(current.contains("extra_col"), s"current read missing evolved column: $current") - assert(!travel.contains("extra_col") && travel.size == Core.tableColumns.size, - s"time travel must read with the SNAPSHOT's schema (no extra_col): $travel") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF $s0").collect()(0).getLong(0) == 3, - "pre-DDL snapshot row count wrong") - }() - - val interactRestoreAfterAddColumn: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("interact.ddl.restoreAfterAddColumn") { (spark, table) => - val s0 = snapshotIds(spark, table).last - spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") - spark.sql(s"INSERT INTO $table VALUES $extraColInsert9") - spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', $s0)") - val cols = spark.sql(s"SELECT * FROM $table LIMIT 1").columns.toSeq - assert(cols.contains("extra_col"), s"rollback rolls back DATA only — schema keeps the evolved column: $cols") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "data not rolled back") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE extra_col IS NOT NULL").collect()(0).getLong(0) == 0, - "rolled-back rows must read the evolved column as null") - spark.sql(s"INSERT INTO $table VALUES $extraColInsert10") // table stays writable at the evolved arity - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 4, "post-rollback insert failed") - }() - - // E1: data in the evolved column, then the (currently pinned-rejected) DROP — table stays intact. - // Gating pin: if DROP COLUMN support ever lands this fails → extend to full post-drop coverage. - val interactDropColAfterData: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("interact.ddl.dropColAfterData") { (spark, table) => - spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") - spark.sql(s"INSERT INTO $table VALUES $extraColInsert9") - val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table DROP COLUMN extra_col")) - assert(e.getMessage.contains("not found in newSchema"), s"drop rejection message changed: ${e.getMessage.take(200)}") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE extra_col = 42").collect()(0).getLong(0) == 1, - "rejected drop must leave the column's data readable") - spark.sql(s"INSERT INTO $table VALUES $extraColInsert10") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 5, - "rejected drop must leave the table writable") - }() - - // ── RTAS × history / lineage ─────────────────────────────────────────────────────────────── - - val interactRtasHistoryPreserved: TableTest[CoreTable.type] = - rtasPrep.step("interact.rtas.historyPreserved") { (spark, table) => - val pre = snapshotIds(spark, table).last - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - assert(spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) == 2, - "pre-RTAS snapshots must survive the replace") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF $pre").collect()(0).getLong(0) == 3, - "time travel to a pre-RTAS snapshot must work") - }() - - val interactRtasRestoreRejected: TableTest[CoreTable.type] = - rtasPrep.step("interact.rtas.restoreRejected") { (spark, table) => - val pre = snapshotIds(spark, table).last - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - val e = Check.intercept[ValidationException]( - spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', $pre)")) - assert(e.getMessage.contains("not an ancestor"), - s"rollback across RTAS: expected the new-lineage/ancestry rejection, got: ${e.getMessage.take(200)}") - }() - - // The recovery path rollback can't provide: set_current_snapshot has no ancestry requirement. - val interactRtasSetCurrentRecovery: TableTest[CoreTable.type] = - rtasPrep.step("interact.rtas.setCurrentRecovery") { (spark, table) => - val pre = snapshotIds(spark, table).last - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - spark.sql(s"CALL openhouse.system.set_current_snapshot('${catalogRelative(table)}', $pre)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, - "set_current_snapshot must recover the pre-RTAS state (no ancestry requirement)") - }() + private def interactionDdlCases(format: String): List[Plan.Case] = { + val preparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + + List( + preparation.test("interact.ddl.ttAfterAddColumn") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + val currentColumns = table.spark + .sql(s"SELECT * FROM ${table.name} LIMIT 1") + .columns + .toSeq + val historicalColumns = table.spark + .sql( + s"SELECT * FROM ${table.name} " + + s"VERSION AS OF $seedSnapshotId LIMIT 1") + .columns + .toSeq + val historicalRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF $seedSnapshotId") + .collect()(0) + .getLong(0) + + assert( + currentColumns.contains("extra_col"), + s"current read is missing the evolved column: $currentColumns") + assert( + !historicalColumns.contains("extra_col") && + historicalColumns.size == Core.tableColumns.size, + s"time travel should use the snapshot schema: $historicalColumns") + assert( + historicalRowCount == 3, + s"pre-DDL snapshot should contain 3 rows, got $historicalRowCount") + }, + preparation.test("interact.ddl.restoreAfterAddColumn") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $seedSnapshotId)") + val currentColumns = table.spark + .sql(s"SELECT * FROM ${table.name} LIMIT 1") + .columns + .toSeq + val currentRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val nonNullEvolvedValueCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + "WHERE extra_col IS NOT NULL") + .collect()(0) + .getLong(0) + + assert( + currentColumns.contains("extra_col"), + s"rollback should retain the evolved schema: $currentColumns") + assert( + currentRowCount == 3, + s"rollback should restore 3 rows, got $currentRowCount") + assert( + nonNullEvolvedValueCount == 0, + "rolled-back rows should read the evolved column as null") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert10") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "the rolled-back table should accept evolved-schema writes") + }, + preparation.test("interact.ddl.dropColAfterData") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} DROP COLUMN extra_col")) + + assert( + exception.getMessage.contains("not found in newSchema"), + s"drop rejection message changed: ${exception.getMessage.take(200)}") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} WHERE extra_col = 42") + .collect()(0) + .getLong(0) == 1, + "rejected drop should leave the column data readable") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert10") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 5, + "rejected drop should leave the table writable") + }) + } - val interactRtasWriteAfter: TableTest[CoreTable.type] = - rtasPrep.step("interact.rtas.writeAfter") { (spark, table) => - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, - "replaced table must stay writable (DML-after-RTAS)") - }() - - // G9 (partition half): the replace path skips checkPartitionSpecEvolution — RTAS CAN change the - // spec where ALTER is pinned-rejected. Characterizes the bypass; if this ever fails, the guard - // was extended to the replace path — update AUDIT-FINDINGS G9. - val interactRtasPartitionSpecChange: TableTest[CoreTable.type] = - rtasPrep.step("interact.rtas.partitionSpecChange") { (spark, table) => - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource PARTITIONED BY (datepartition) AS SELECT * FROM $table") - val desc = spark.sql(s"DESCRIBE TABLE $table").collect().toSeq - // Confirmed live: the table gains a "# Partition Information" section (datepartition listed - // both as a column and as a partition field) — the spec changed where ALTER is pinned-rejected. - assert(desc.exists(_.getString(0) == "# Partition Information") && - desc.count(_.getString(0) == "datepartition") == 2, - s"G9 appears FIXED — RTAS no longer changes the partition spec; update AUDIT-FINDINGS G9. DESCRIBE:\n" + - desc.map(_.mkString(" | ")).mkString("\n")) - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "rows lost in re-spec RTAS") - }() - - // G9 (schema half): column drop via RTAS projection, where ALTER DROP COLUMN is pinned-rejected. - // Confirmed live (first run failed on the harness's own read-back because the column was GONE). - // Runs on a side table so the pipeline's implicit full-schema read-back stays valid. - val interactRtasDropsColumn: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("interact.rtas.dropsColumn") { (spark, table) => - val side = s"${table}_dropcol" - spark.sql(s"DROP TABLE IF EXISTS $side") + private def interactionRtasCases(format: String): List[Plan.Case] = { + val basePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + val replacePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableReplace")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")()) + val userPropertyPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', " + + "'replace.enabled'='true', 'user.key'='v1')")() + .insert(3)()) + val retentionPolicyPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"PARTITIONED BY (${Core.datePartition.columnName}) " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', 'replace.enabled'='true')")() + .insert(3)() + .sql("setRetention")(table => + s"ALTER TABLE $table SET POLICY " + + s"(RETENTION = 30d ON COLUMN ${Core.datePartition.columnName} " + + "WHERE pattern = 'yyyy-MM-dd-HH')")()) + + List( + replacePreparation.test("interact.rtas.historyPreserved") { table => + val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val snapshotCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.snapshots") + .collect()(0) + .getLong(0) + val historicalRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF $preReplaceSnapshotId") + .collect()(0) + .getLong(0) + + assert( + snapshotCount == 2, + s"replace should retain two snapshots, got $snapshotCount") + assert( + historicalRowCount == 3, + s"pre-replace snapshot should contain 3 rows, got $historicalRowCount") + }, + replacePreparation.test("interact.rtas.restoreRejected") { table => + val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val exception = Check.intercept[ValidationException]( + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $preReplaceSnapshotId)")) + + assert( + exception.getMessage.contains("not an ancestor"), + "rollback across replacement should reject the old lineage") + }, + replacePreparation.test("interact.rtas.setCurrentRecovery") { table => + val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + table.spark.sql( + "CALL openhouse.system.set_current_snapshot(" + + s"'${catalogRelative(table.name)}', $preReplaceSnapshotId)") + val recoveredRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + recoveredRowCount == 3, + s"set_current_snapshot should recover 3 rows, got $recoveredRowCount") + }, + replacePreparation.test("interact.rtas.writeAfter") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + rowCount == 3, + s"replaced table should contain 3 rows after insert, got $rowCount") + }, + replacePreparation.test("interact.rtas.partitionSpecChange") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"PARTITIONED BY (${Core.datePartition.columnName}) " + + s"AS SELECT * FROM ${table.name}") + val description = table.spark + .sql(s"DESCRIBE TABLE ${table.name}") + .collect() + .toSeq + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + description.exists(_.getString(0) == "# Partition Information") && + description.count( + _.getString(0) == Core.datePartition.columnName) == 2, + "RTAS should replace the partition specification") + assert( + rowCount == 3, + s"partition-spec replacement should preserve 3 rows, got $rowCount") + }, + basePreparation.test("interact.rtas.dropsColumn") { table => + val sideTable = s"${table.name}_dropcol" + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") try { - spark.sql(s"CREATE TABLE $side USING $dataSource TBLPROPERTIES ('replace.enabled'='true') AS SELECT * FROM $table") - spark.sql(s"CREATE OR REPLACE TABLE $side USING $dataSource AS " + - s"SELECT ${Core.long0.columnName}, ${Core.string0.columnName} FROM $side") - val cols = spark.sql(s"SELECT * FROM $side LIMIT 1").columns.toSeq - assert(cols == Seq(Core.long0.columnName, Core.string0.columnName), - s"G9 appears FIXED — RTAS no longer drops columns (ALTER DROP stays rejected); update AUDIT-FINDINGS G9: $cols") - assert(spark.sql(s"SELECT count(*) FROM $side").collect()(0).getLong(0) == 3, "rows lost in column-drop RTAS") - } finally spark.sql(s"DROP TABLE IF EXISTS $side") - }() - - // ── RTAS × table-property merge semantics (the THIRD property path beside CREATE and ALTER) ── - val interactRtasPropsUserSurvival: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$seedFmt', 'replace.enabled'='true', 'user.key'='v1')")() - .insert(3)() - .step("interact.rtas.props.userSurvival") { (spark, table) => - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - val p = tableProps(spark, table) - assert(p.get("user.key").contains("v1"), s"user prop lost across RTAS: user.key=${p.get("user.key")}") - assert(p.get("replace.enabled").contains("true"), s"replace.enabled lost across RTAS: ${p.get("replace.enabled")}") - }() - - val interactRtasPropsStatementWins: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$seedFmt', 'replace.enabled'='true', 'user.key'='v1')")() - .insert(3)() - .step("interact.rtas.props.statementWins") { (spark, table) => - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource TBLPROPERTIES ('user.key'='v2') " + - s"AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - val p = tableProps(spark, table) - assert(p.get("user.key").contains("v2"), s"statement TBLPROPERTIES must win over the old value: ${p.get("user.key")}") - assert(p.get("replace.enabled").contains("true"), - s"props NOT named in the statement must still survive (merge, not wholesale replace): ${p.get("replace.enabled")}") - }() - - val interactRtasPropsCreateDefaulting: TableTest[CoreTable.type] = - rtasPrep.step("interact.rtas.props.createDefaulting") { (spark, table) => - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource TBLPROPERTIES ('write.format.default'='orc') " + - s"AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - val p = tableProps(spark, table) - assert(p.get("write.format.default").contains("orc"), - s"RTAS can change the storage format where ALTER can't rewrite: ${p.get("write.format.default")}") - assert(p.get("format-version").forall(_ == "2"), s"forced format-version drifted: ${p.get("format-version")}") - spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "orc-format table not writable") - }() - - val interactRtasPropsReservedPlane: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource PARTITIONED BY (datepartition) TBLPROPERTIES (" + - s"'write.format.default'='$seedFmt', 'replace.enabled'='true')")() - .insert(3)() - .sql("setRetention")(t => s"ALTER TABLE $t SET POLICY (RETENTION = 30d ON COLUMN datepartition WHERE pattern = 'yyyy-MM-dd-HH')")() - .step("interact.rtas.props.reservedPlane") { (spark, table) => - val uuidBefore = tableProps(spark, table).getOrElse("openhouse.tableUUID", "") - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource PARTITIONED BY (datepartition) " + - s"AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - val p = tableProps(spark, table) - assert(p.getOrElse("openhouse.tableUUID", "") == uuidBefore, - s"tableUUID must be preserved across RTAS: $uuidBefore -> ${p.get("openhouse.tableUUID")}") - // G10 (confirmed live): RTAS silently WIPES the policies plane — the retention policy set - // before the replace is gone after it (while tableUUID survives). Characterizes the bug; - // if this fails, G10 was fixed — flip to a survival assertion and update AUDIT-FINDINGS. - val policiesAfter = p.get("policies") - assert(policiesAfter.forall(b => !b.toLowerCase.contains("retention")), - s"G10 appears FIXED — retention policy survived RTAS; update AUDIT-FINDINGS G10 and flip this test: $policiesAfter") - }() - - // RTAS on a table with an existing branch: refs travel in the replace payload — branch survives, - // still readable at its (old-lineage) head. - val interactRtasWithBranch: TableTest[CoreTable.type] = - rtasPrep.step("interact.rtas.withBranch") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH keepbr") - spark.sql(s"INSERT INTO $table.branch_keepbr VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - val refs = spark.sql(s"SELECT name FROM $table.refs").collect().toSeq.map(_.getString(0)).toSet - assert(refs.contains("keepbr"), s"branch ref lost across RTAS: $refs") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'keepbr'").collect()(0).getLong(0) == 4, - "branch head (old lineage) unreadable after RTAS") - }() - - // ── branch × history / maintenance ───────────────────────────────────────────────────────── - val interactBranchTtBeforeBranchPoint: TableTest[CoreTable.type] = - coreTwoSnapshots.step("interact.branch.ttBeforeBranchPoint") { (spark, table) => - val snaps = snapshotIds(spark, table) - val ts0 = spark.sql(s"SELECT committed_at FROM $table.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getTimestamp(0) - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") - spark.sql(s"ALTER TABLE $table CREATE BRANCH tb") - spark.sql(s"INSERT INTO $table.branch_tb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'tb'").collect()(0).getLong(0) == 6, "branch head") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF ${snaps.head}").collect()(0).getLong(0) == 3, - "snapshot-id travel to a pre-branch-point ancestor must work") - spark.conf.set("spark.wap.branch", "tb") - try { - assert(spark.sql(s"SELECT count(*) FROM $table TIMESTAMP AS OF '$ts0'").collect()(0).getLong(0) == 3, - "explicit TIMESTAMP AS OF must override spark.wap.branch and resolve against main history") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF ${snaps.head}").collect()(0).getLong(0) == 3, - "explicit VERSION AS OF must override spark.wap.branch") - } finally spark.conf.unset("spark.wap.branch") - }() - - // E5 characterization (mirror of G8): DDL on MAIN hits branches immediately — schema is - // table-global, and an old-arity branch writer is broken mid-flight. - val interactBranchMainDdlImmediate: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("interact.branch.mainDdlImmediate") { (spark, table) => - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") - spark.sql(s"ALTER TABLE $table CREATE BRANCH mb") - spark.sql(s"INSERT INTO $table.branch_mb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") // DDL on MAIN - val branchCols = spark.sql(s"SELECT * FROM $table VERSION AS OF 'mb' LIMIT 1").columns.toSeq - assert(branchCols.contains("extra_col"), s"main DDL is table-global — branch reads see it immediately: $branchCols") - val e = Check.intercept[AnalysisException]( - spark.sql(s"INSERT INTO $table.branch_mb VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')")) - assert(e.getMessage.toLowerCase.contains("not enough data columns"), - s"old-arity branch writer must break after main DDL (characterizes the hazard): ${e.getMessage.take(200)}") - spark.sql(s"INSERT INTO $table.branch_mb VALUES (CAST(8 AS BIGINT), 8, 'row-8', 8.5, true, '2024-01-08-07', 44)") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'mb'").collect()(0).getLong(0) == 5, - "new-arity branch write after main DDL") - }() - - // E10: expiration is ref-aware — branch heads survive, shared ancestry prunes. - val interactBranchExpireProtectsRefs: TableTest[CoreTable.type] = - coreTwoSnapshots.step("interact.branch.expireProtectsRefs") { (spark, table) => - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") - spark.sql(s"ALTER TABLE $table CREATE BRANCH eb") - spark.sql(s"INSERT INTO $table.branch_eb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - spark.sql(s"INSERT INTO $table VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert(spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) == 4, "expected 4 snapshots pre-expire") - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - val refs = spark.sql(s"SELECT name FROM $table.refs").collect().toSeq.map(_.getString(0)).toSet - assert(refs == Set("main", "eb"), s"branch/tag refs must survive expiration: $refs") - assert(spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) == 2, - "shared ancestry prunes to the two ref heads") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'eb'").collect()(0).getLong(0) == 6, "branch readable post-expire") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 6, "main readable post-expire") - }() - - // C4: restore procedures target MAIN even while spark.wap.branch is set (procedures are not - // branch-conf-routed) — the branch is untouched. - val interactBranchRollbackWhileWapConf: TableTest[CoreTable.type] = - coreTwoSnapshots.step("interact.branch.rollbackWhileWapConf") { (spark, table) => - val s0 = snapshotIds(spark, table).head - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") - spark.sql(s"ALTER TABLE $table CREATE BRANCH rb") - spark.sql(s"INSERT INTO $table.branch_rb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - spark.conf.set("spark.wap.branch", "rb") - try spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', $s0)") - finally spark.conf.unset("spark.wap.branch") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, - "rollback under wap.branch conf still targets MAIN (procedures are not branch-routed)") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'rb'").collect()(0).getLong(0) == 6, - "branch untouched by the main rollback") - }() - - // C1: rolled-past snapshots are unreferenced — expiration makes the rollback permanent. - val interactRestoreExpireAfterRollback: TableTest[CoreTable.type] = - coreTwoSnapshots.step("interact.restore.expireAfterRollback") { (spark, table) => - val snaps = snapshotIds(spark, table) - spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', ${snaps.head})") - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - assert(spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) == 1, - "the rolled-past snapshot must be expired (unreferenced)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "current state intact") - val e = Check.intercept[Exception]( - spark.sql(s"SELECT count(*) FROM $table VERSION AS OF ${snaps(1)}").collect()) - assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(_.toLowerCase.contains("snapshot"))), - s"travel to the expired snapshot must fail (rollback is now PERMANENT): ${e.getMessage.take(200)}") - }() - - // ── THE COMPOSITE DEFECT: branch × expiration × merge (G11; INTERACTION-AUDIT §6) ─────────── - // Bytecode-confirmed mechanism: RemoveSnapshots retention is per-ref and head-anchored (no - // protection for the ancestry BETWEEN live refs), and SnapshotUtil's ancestry walk SILENTLY - // TRUNCATES at an expired hole and returns false. So policy-driven expiration between branch - // work and the merge makes fast_forward spuriously reject with "not an ancestor" — even when - // main never advanced — and, with no rebase in Iceberg, the branch is permanently stranded. - // The pair test (branch × expire) PASSES because reads don't consume ancestry; only the merge does. - val interactExpireMergeSpuriousReject: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("interact.branch.expireMerge.spuriousReject") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH mb") - spark.sql(s"INSERT INTO $table.branch_mb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") // B1 - spark.sql(s"INSERT INTO $table.branch_mb VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") // B2 (head) - assert(countOf(spark, s"SELECT count(*) FROM $table.snapshots") == "3", "expected P, B1, B2") - // main NEVER advances. This merge is valid right now (branch.fastForward.merge is the - // no-expiration control proving it). Interpose the destroyer: - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - // P2 VIOLATED: retention is per-ref head-anchored — the intermediate branch commit B1 - // (merge connectivity) is expired even though both refs are alive. - assert(countOf(spark, s"SELECT count(*) FROM $table.snapshots") == "2", - "retention keeps only the two ref heads; the intermediate branch snapshot is expired") - // The pair-test ILLUSION: refs alive, branch fully readable — nothing looks broken. - val refs = spark.sql(s"SELECT name FROM $table.refs").collect().toSeq.map(_.getString(0)).toSet - assert(refs == Set("main", "mb"), s"both refs alive: $refs") - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'mb'") == "5", "branch readable") - // P1 VIOLATED: the merge is now spuriously rejected — the ancestry walk from B2 hits the - // B1 hole, silently truncates, and concludes main's head "is not an ancestor" of the branch. - val e = Check.intercept[Exception]( - spark.sql(s"CALL openhouse.system.fast_forward('${catalogRelative(table)}', 'main', 'mb')")) - assert(Option(e.getMessage).exists(_.contains("not an ancestor")), - s"G11 appears FIXED — fast_forward survived expiration (or failed differently); update AUDIT-FINDINGS G11: " + - s"${e.getClass.getName} ${Option(e.getMessage).getOrElse("").take(180)}") - // P6 VIOLATED: no recovery path merges the branch. Characterize the cherry-pick fallback: - val b2 = spark.sql(s"SELECT snapshot_id FROM $table.refs WHERE name = 'mb'").collect()(0).getLong(0) - val cherry = try { - spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${b2}L)") - s"SUCCEEDED — main now ${countOf(spark, s"SELECT count(*) FROM $table")} rows (B1's commit silently LOST in the 'merge')" - } catch { case t: Throwable => s"REJECTED ${t.getClass.getName} :: ${Option(t.getMessage).getOrElse("").take(160)}" } - println(s"DIAG expireMerge.cherrypickFallback: $cherry") - val mainCount = countOf(spark, s"SELECT count(*) FROM $table").toLong - assert(mainCount == 3 || mainCount == 4, s"main must stay consistent (3, or 4 if cherry-pick half-merged): $mainCount") - // Copy-out is the ONLY full recovery (data files survive: expiration ran cleanExpiredFiles(false)). - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'mb'") == "5", - "branch data must remain readable for copy-out recovery") - }() - - // P3 VIOLATED: WAP-staged snapshots are UNREFERENCED, so age-based expiration silently deletes - // them before publish; the loss only becomes loud at publish time ("Cannot find snapshot"). - // OpenHouse's scheduled expiration job (default 3-day TTL) makes this automatic, not hypothetical. - val interactExpireMergeStagedWapLoss: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("interact.branch.expireMerge.stagedWapLoss") { (spark, table) => - spark.conf.set("spark.wap.id", "w2") - try spark.sql(s"INSERT INTO $table VALUES (CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - finally spark.conf.unset("spark.wap.id") - assert(countOf(spark, s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'w2'") == "1", "staged") - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - // The SILENT loss: expiration reports nothing about the staged work it destroyed. - assert(countOf(spark, s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'w2'") == "0", - "P3 appears FIXED — staged WAP snapshot survived expiration; update AUDIT-FINDINGS G11") - // Loud only NOW, at publish — after the work is unrecoverable: - val e = Check.intercept[Exception]( - spark.sql(s"CALL openhouse.system.publish_changes(table => '${catalogRelative(table)}', wap_id => 'w2')")) - println(s"DIAG stagedWapLoss.publish: ${e.getClass.getName} :: ${Option(e.getMessage).getOrElse("").take(180)}") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "main unchanged; the staged write is gone") - }() - - // ── flags at CREATE + ALTER-to-MoR + compaction over evolved schema ──────────────────────── - val interactFlagsWapReplaceAtCreate: TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$seedFmt', 'write.wap.enabled'='true', 'replace.enabled'='true')")() - .insert(3)() - .step("interact.flags.wapReplaceAtCreate") { (spark, table) => - val p = tableProps(spark, table) - assert(p.get("write.wap.enabled").contains("true") && p.get("replace.enabled").contains("true"), - s"flags set at CREATE must be honored: wap=${p.get("write.wap.enabled")} replace=${p.get("replace.enabled")}") - spark.sql(s"ALTER TABLE $table CREATE BRANCH cb") // wap-at-create usable immediately - val e = Check.intercept[BadRequestException]( - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table")) - assert(e.getMessage.contains("while WAP"), - s"RTAS-while-WAP guard must fire from create-time flags too: ${e.getMessage.take(200)}") - }() - - val interactMorAlterToMor: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)() - .sql("seed(3, one-file)")(t => - s"INSERT INTO $t SELECT /*+ COALESCE(1) */ * FROM (${RowGenerator.valuesClause(Core, 3)}) AS seed")() - .step("interact.mor.alterToMor") { (spark, table) => - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')") - spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") - val deleteFiles = spark.sql(s"SELECT count(*) FROM $table.all_delete_files").collect()(0).getLong(0) - assert(deleteFiles == 1, - s"ALTER-to-MoR must govern subsequent deletes (expected 1 position-delete file, got $deleteFiles)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "row not deleted") - }() - - val interactMaintCompactEvolved: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("interact.maint.compactEvolved") { (spark, table) => - spark.sql(s"ALTER TABLE $table ADD COLUMN extra_col INT") - spark.sql(s"INSERT INTO $table VALUES $extraColInsert9") - spark.sql(s"INSERT INTO $table VALUES $extraColInsert10") - spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}')") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 5, "compaction changed row count") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE extra_col IN (42, 43)").collect()(0).getLong(0) == 2, - "compaction over mixed-schema files must preserve evolved-column values") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE extra_col IS NULL").collect()(0).getLong(0) == 3, - "pre-evolution rows must stay null in the evolved column") - }() - - val interactions: List[(String, TableTest[CoreTable.type])] = List( - "interact.ddl.ttAfterAddColumn" -> interactTtAfterAddColumn, - "interact.ddl.restoreAfterAddColumn" -> interactRestoreAfterAddColumn, - "interact.ddl.dropColAfterData" -> interactDropColAfterData, - "interact.rtas.historyPreserved" -> interactRtasHistoryPreserved, - "interact.rtas.restoreRejected" -> interactRtasRestoreRejected, - "interact.rtas.setCurrentRecovery" -> interactRtasSetCurrentRecovery, - "interact.rtas.writeAfter" -> interactRtasWriteAfter, - "interact.rtas.partitionSpecChange" -> interactRtasPartitionSpecChange, - "interact.rtas.dropsColumn" -> interactRtasDropsColumn, - "interact.rtas.props.userSurvival" -> interactRtasPropsUserSurvival, - "interact.rtas.props.statementWins" -> interactRtasPropsStatementWins, - "interact.rtas.props.createDefaulting" -> interactRtasPropsCreateDefaulting, - "interact.rtas.props.reservedPlane" -> interactRtasPropsReservedPlane, - "interact.rtas.withBranch" -> interactRtasWithBranch, - "interact.branch.ttBeforeBranchPoint" -> interactBranchTtBeforeBranchPoint, - "interact.branch.mainDdlImmediate" -> interactBranchMainDdlImmediate, - "interact.branch.expireProtectsRefs" -> interactBranchExpireProtectsRefs, - "interact.branch.rollbackWhileWapConf" -> interactBranchRollbackWhileWapConf, - "interact.restore.expireAfterRollback" -> interactRestoreExpireAfterRollback, - "interact.branch.expireMerge.spuriousReject" -> interactExpireMergeSpuriousReject, - "interact.branch.expireMerge.stagedWapLoss" -> interactExpireMergeStagedWapLoss, - "interact.flags.wapReplaceAtCreate" -> interactFlagsWapReplaceAtCreate, - "interact.mor.alterToMor" -> interactMorAlterToMor, - "interact.maint.compactEvolved" -> interactMaintCompactEvolved - ) + table.spark.sql( + s"CREATE TABLE $sideTable USING $dataSource " + + "TBLPROPERTIES ('replace.enabled'='true') " + + s"AS SELECT * FROM ${table.name}") + table.spark.sql( + s"CREATE OR REPLACE TABLE $sideTable USING $dataSource AS " + + s"SELECT ${Core.long0.columnName}, ${Core.string0.columnName} " + + s"FROM $sideTable") + val columns = table.spark + .sql(s"SELECT * FROM $sideTable LIMIT 1") + .columns + .toSeq + val rowCount = table.spark + .sql(s"SELECT count(*) FROM $sideTable") + .collect()(0) + .getLong(0) + + assert( + columns == Seq(Core.long0.columnName, Core.string0.columnName), + s"RTAS should project the table to two columns, got $columns") + assert( + rowCount == 3, + s"column-drop RTAS should preserve 3 rows, got $rowCount") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + } + }, + userPropertyPreparation.test( + "interact.rtas.props.userSurvival") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val properties = tableProps(table.spark, table.name) + + assert( + properties.get("user.key").contains("v1"), + s"user.key did not survive RTAS: ${properties.get("user.key")}") + assert( + properties.get("replace.enabled").contains("true"), + "replace.enabled did not survive RTAS") + }, + userPropertyPreparation.test( + "interact.rtas.props.statementWins") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + "TBLPROPERTIES ('user.key'='v2') " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val properties = tableProps(table.spark, table.name) + + assert( + properties.get("user.key").contains("v2"), + s"statement property should win, got ${properties.get("user.key")}") + assert( + properties.get("replace.enabled").contains("true"), + "properties omitted from RTAS should survive") + }, + replacePreparation.test( + "interact.rtas.props.createDefaulting") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + "TBLPROPERTIES ('write.format.default'='orc') " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val properties = tableProps(table.spark, table.name) + + assert( + properties.get("write.format.default").contains("orc"), + "RTAS should set write.format.default to orc") + assert( + properties.get("format-version").forall(_ == "2"), + s"format-version drifted: ${properties.get("format-version")}") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 3, + "RTAS table using ORC should remain writable") + }, + retentionPolicyPreparation.test( + "interact.rtas.props.reservedPlane") { table => + val tableUuidBefore = tableProps(table.spark, table.name) + .getOrElse("openhouse.tableUUID", "") + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"PARTITIONED BY (${Core.datePartition.columnName}) " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val properties = tableProps(table.spark, table.name) + val policiesAfter = properties.get("policies") + + assert( + properties.getOrElse("openhouse.tableUUID", "") == + tableUuidBefore, + "table UUID should survive RTAS") + assert( + policiesAfter.forall(policy => + !policy.toLowerCase.contains("retention")), + s"RTAS should currently remove the retention policy: $policiesAfter") + }, + replacePreparation.test("interact.rtas.withBranch") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH keepbr") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_keepbr VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val refs = table.spark + .sql(s"SELECT name FROM ${table.name}.refs") + .collect() + .map(_.getString(0)) + .toSet + val branchRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'keepbr'") + .collect()(0) + .getLong(0) + + assert( + refs.contains("keepbr"), + s"branch ref did not survive RTAS: $refs") + assert( + branchRowCount == 4, + s"branch should retain 4 rows after RTAS, got $branchRowCount") + }) + } + + private def interactionBranchCases(format: String): List[Plan.Case] = { + val basePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + val twoSnapshotPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("insertMore")(table => + s"INSERT INTO $table VALUES " + + "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + + "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) + val wapPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableWap")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")()) + + List( + twoSnapshotPreparation.test( + "interact.branch.ttBeforeBranchPoint") { table => + val snapshots = snapshotIds(table.spark, table.name) + val firstCommitTimestamp = table.spark + .sql( + s"SELECT committed_at FROM ${table.name}.snapshots " + + "ORDER BY committed_at LIMIT 1") + .collect()(0) + .getTimestamp(0) + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH tb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_tb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'tb'") + .collect()(0) + .getLong(0) == 6, + "branch head should contain 6 rows") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF ${snapshots.head}") + .collect()(0) + .getLong(0) == 3, + "snapshot ID should resolve before the branch point") + + table.spark.conf.set("spark.wap.branch", "tb") + try { + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"TIMESTAMP AS OF '$firstCommitTimestamp'") + .collect()(0) + .getLong(0) == 3, + "explicit timestamp should override spark.wap.branch") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF ${snapshots.head}") + .collect()(0) + .getLong(0) == 3, + "explicit snapshot ID should override spark.wap.branch") + } finally { + table.spark.conf.unset("spark.wap.branch") + } + }, + basePreparation.test("interact.branch.mainDdlImmediate") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_mb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + val branchColumns = table.spark + .sql( + s"SELECT * FROM ${table.name} VERSION AS OF 'mb' LIMIT 1") + .columns + .toSeq + + assert( + branchColumns.contains("extra_col"), + s"main DDL should change the table-global schema: $branchColumns") + + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"INSERT INTO ${table.name}.branch_mb VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')")) + assert( + exception.getMessage.toLowerCase.contains("not enough data columns"), + "old-arity branch writer should fail after main DDL") + + table.spark.sql( + s"INSERT INTO ${table.name}.branch_mb VALUES " + + "(CAST(8 AS BIGINT), 8, 'row-8', 8.5, true, " + + "'2024-01-08-07', 44)") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") + .collect()(0) + .getLong(0) == 5, + "new-arity branch write should succeed after main DDL") + }, + twoSnapshotPreparation.test( + "interact.branch.expireProtectsRefs") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH eb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_eb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}.snapshots") + .collect()(0) + .getLong(0) == 4, + "expected four snapshots before expiration") + + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + val refs = table.spark + .sql(s"SELECT name FROM ${table.name}.refs") + .collect() + .map(_.getString(0)) + .toSet + val snapshotCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.snapshots") + .collect()(0) + .getLong(0) + val branchRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'eb'") + .collect()(0) + .getLong(0) + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert(refs == Set("main", "eb"), s"refs changed: $refs") + assert( + snapshotCount == 2, + s"expiration should retain two ref heads, got $snapshotCount") + assert( + branchRowCount == 6, + s"branch should remain readable with 6 rows, got $branchRowCount") + assert( + mainRowCount == 6, + s"main should remain readable with 6 rows, got $mainRowCount") + }, + twoSnapshotPreparation.test( + "interact.branch.rollbackWhileWapConf") { table => + val firstSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH rb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_rb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.conf.set("spark.wap.branch", "rb") + try { + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $firstSnapshotId)") + } finally { + table.spark.conf.unset("spark.wap.branch") + } + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val branchRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb'") + .collect()(0) + .getLong(0) + + assert( + mainRowCount == 3, + s"rollback should target main and restore 3 rows, got $mainRowCount") + assert( + branchRowCount == 6, + s"rollback should leave branch at 6 rows, got $branchRowCount") + }, + twoSnapshotPreparation.test( + "interact.restore.expireAfterRollback") { table => + val snapshots = snapshotIds(table.spark, table.name) + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', ${snapshots.head})") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + val snapshotCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.snapshots") + .collect()(0) + .getLong(0) + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + snapshotCount == 1, + s"rolled-past snapshot should expire, got $snapshotCount snapshots") + assert( + rowCount == 3, + s"rollback should preserve 3 current rows, got $rowCount") + + val exception = Check.intercept[Exception]( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF ${snapshots(1)}") + .collect()) + assert( + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage) + .exists(_.toLowerCase.contains("snapshot"))), + "time travel to the expired rolled-past snapshot should fail") + }, + basePreparation.test( + "interact.branch.expireMerge.spuriousReject") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_mb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_mb VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots") == "3", + "expected parent and two branch snapshots") + + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots") == "2", + "expiration should remove the intermediate branch snapshot") + val refs = table.spark + .sql(s"SELECT name FROM ${table.name}.refs") + .collect() + .map(_.getString(0)) + .toSet + assert(refs == Set("main", "mb"), s"refs changed: $refs") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") == "5", + "branch should remain readable after expiration") + + val exception = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.fast_forward(" + + s"'${catalogRelative(table.name)}', 'main', 'mb')")) + assert( + Option(exception.getMessage).exists(_.contains("not an ancestor")), + "fast_forward should reject the punctured branch ancestry") + + val branchHeadSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.refs WHERE name = 'mb'") + .collect()(0) + .getLong(0) + val cherryPickOutcome = + try { + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', " + + s"${branchHeadSnapshotId}L)") + s"SUCCEEDED: main now ${countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}")} rows" + } catch { + case exception: Throwable => + s"REJECTED ${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(160) + } + println( + s"DIAG expireMerge.cherrypickFallback: $cherryPickOutcome") + val mainRowCount = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}").toLong + + assert( + mainRowCount == 3 || mainRowCount == 4, + s"main should remain consistent, got $mainRowCount rows") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") == "5", + "branch data should remain available for copy-out recovery") + }, + wapPreparation.test( + "interact.branch.expireMerge.stagedWapLoss") { table => + table.spark.conf.set("spark.wap.id", "w2") + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + } finally { + table.spark.conf.unset("spark.wap.id") + } + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w2'") == "1", + "WAP write should create one staged snapshot") + + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w2'") == "0", + "expiration should remove the unreferenced staged snapshot") + + val exception = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.publish_changes(" + + s"table => '${catalogRelative(table.name)}', wap_id => 'w2')")) + println( + "DIAG stagedWapLoss.publish: " + + s"${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(180)) + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "main should remain unchanged after staged snapshot loss") + }) + } + + private def interactionMiscellaneousCases( + format: String): List[Plan.Case] = { + val flagPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', " + + "'write.wap.enabled'='true', 'replace.enabled'='true')")() + .insert(3)()) + val oneFilePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .sql("seed")(table => + s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM " + + s"(${RowGenerator.valuesClause(Core, 3)}) AS seed")()) + val basePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + + List( + flagPreparation.test("interact.flags.wapReplaceAtCreate") { table => + val properties = tableProps(table.spark, table.name) + assert( + properties.get("write.wap.enabled").contains("true") && + properties.get("replace.enabled").contains("true"), + "WAP and replace flags should be active when set at CREATE") + + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH cb") + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name}")) + assert( + exception.getMessage.contains("while WAP"), + "RTAS should reject a table with WAP enabled at CREATE") + }, + oneFilePreparation.test("interact.mor.alterToMor") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.delete.mode'='merge-on-read')") + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val deleteFileCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.all_delete_files") + .collect()(0) + .getLong(0) + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + deleteFileCount == 1, + s"ALTER-to-MoR should create one delete file, got $deleteFileCount") + assert( + rowCount == 2, + s"ALTER-to-MoR delete should leave 2 rows, got $rowCount") + }, + basePreparation.test("interact.maint.compactEvolved") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert10") + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}')") + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val evolvedValueCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + "WHERE extra_col IN (42, 43)") + .collect()(0) + .getLong(0) + val nullValueCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} WHERE extra_col IS NULL") + .collect()(0) + .getLong(0) + + assert( + rowCount == 5, + s"compaction should preserve 5 rows, got $rowCount") + assert( + evolvedValueCount == 2, + s"compaction should preserve two evolved values, got $evolvedValueCount") + assert( + nullValueCount == 3, + s"pre-evolution rows should remain null, got $nullValueCount") + }) + } + + val interactionCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + interactionDdlCases(format) ++ + interactionRtasCases(format) ++ + interactionBranchCases(format) ++ + interactionMiscellaneousCases(format) + } // G2 characterization needs the REST lock (no SQL surface) → Ctx-based like controlPlane. // Sanity-checks the lock DOES block a normal write, then demonstrates RTAS sails through it. @@ -462,9 +932,11 @@ trait InteractionScenarios extends ScenarioKit { } } - val interactionCtxOps: List[(String, Ctx => Unit)] = List( - "interact.rtas.onLockedTable" -> interactRtasOnLockedTable - ) + val interactionContextCases: List[Plan.Case] = + List( + Plan.Case( + "interact.rtas.onLockedTable @ embedded", + interactRtasOnLockedTable)) // ═══ Surface-completion axis: queued follow-ups + untested Iceberg surface ═══════════════════ diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala index 026650c38..3160632af 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala @@ -17,98 +17,141 @@ trait MaintControlScenarios extends ScenarioKit { // A two-snapshot base: seed 3 rows (snapshot A), then insert 2 more (snapshot B). // Format is a PARAMETER, not baked in — so any block built on this base can multiplex across formats. - def timeTravelVersionAsOf(fmt: String): TableTest[CoreTable.type] = - coreTwoSnapshots(fmt).check("timeTravel.versionAsOf") { view => - val snaps = snapshotIds(view.spark, view.table) - assert(view.spark.sql(s"SELECT count(*) FROM ${view.table} VERSION AS OF ${snaps(0)}").collect()(0).getLong(0) == 3) - assert(view.spark.sql(s"SELECT count(*) FROM ${view.table} VERSION AS OF ${snaps(1)}").collect()(0).getLong(0) == 5) - } + val timeTravelCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + val preparation = TablePreparation( + format, + coreTwoSnapshots(format)) - def timeTravelTimestampAsOf(fmt: String): TableTest[CoreTable.type] = - coreTwoSnapshots(fmt).check("timeTravel.timestampAsOf") { view => - val ts0 = view.spark.sql(s"SELECT committed_at FROM ${view.table}.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getTimestamp(0) - assert(view.spark.sql(s"SELECT count(*) FROM ${view.table} TIMESTAMP AS OF '$ts0'").collect()(0).getLong(0) == 3) - } + List( + preparation.test("timeTravel.versionAsOf") { table => + val snapshots = snapshotIds(table.spark, table.name) - def timeTravelMetadataTables(fmt: String): TableTest[CoreTable.type] = - coreTwoSnapshots(fmt).check("timeTravel.metadataTables") { view => - def count(meta: String): Long = view.spark.sql(s"SELECT count(*) FROM ${view.table}.$meta").collect()(0).getLong(0) - assert(count("snapshots") == 2) - assert(count("history") == 2) - assert(count("files") >= 1 && count("manifests") >= 1) - } + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF ${snapshots(0)}") + .collect()(0) + .getLong(0) == 3) + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF ${snapshots(1)}") + .collect()(0) + .getLong(0) == 5) + }, + preparation.test("timeTravel.timestampAsOf") { table => + val firstCommitTimestamp = table.spark + .sql( + s"SELECT committed_at FROM ${table.name}.snapshots " + + "ORDER BY committed_at LIMIT 1") + .collect()(0) + .getTimestamp(0) - def timeTravelIncrementalRead(fmt: String): TableTest[CoreTable.type] = - coreTwoSnapshots(fmt).check("timeTravel.incrementalRead") { view => - val snaps = snapshotIds(view.spark, view.table) - val added = view.spark.read.format("iceberg") - .option("start-snapshot-id", snaps(0)).option("end-snapshot-id", snaps(1)) - .load(view.table).count() - assert(added == 2) // only the rows added between snapshot A and B - } + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"TIMESTAMP AS OF '$firstCommitTimestamp'") + .collect()(0) + .getLong(0) == 3) + }, + preparation.test("timeTravel.metadataTables") { table => + def metadataRowCount(metadataTable: String): Long = + table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.$metadataTable") + .collect()(0) + .getLong(0) - def timeTravelOps(fmt: String): List[(String, TableTest[CoreTable.type])] = List( - "timeTravel.versionAsOf" -> timeTravelVersionAsOf(fmt), - "timeTravel.timestampAsOf" -> timeTravelTimestampAsOf(fmt), - "timeTravel.metadataTables" -> timeTravelMetadataTables(fmt), - "timeTravel.incrementalRead" -> timeTravelIncrementalRead(fmt) - ) - - // Restore/rollback via stored procedures (gated: OpenHouse may not expose CALL procedures). - - def restoreRollbackToSnapshot(fmt: String): TableTest[CoreTable.type] = - coreTwoSnapshots(fmt).step("restore.rollbackToSnapshot") { (spark, table) => - val first = snapshotIds(spark, table).head - spark.sql(s"CALL openhouse.system.rollback_to_snapshot('${catalogRelative(table)}', $first)") - } { view => - assert(view.after.size == 3) // rolled back to the 3-row snapshot - } + assert(metadataRowCount("snapshots") == 2) + assert(metadataRowCount("history") == 2) + assert( + metadataRowCount("files") >= 1 && + metadataRowCount("manifests") >= 1) + }, + preparation.test("timeTravel.incrementalRead") { table => + val snapshots = snapshotIds(table.spark, table.name) + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", snapshots(0)) + .option("end-snapshot-id", snapshots(1)) + .load(table.name) + .count() - def restoreSetCurrentSnapshot(fmt: String): TableTest[CoreTable.type] = - coreTwoSnapshots(fmt).step("restore.setCurrentSnapshot") { (spark, table) => - val first = snapshotIds(spark, table).head - spark.sql(s"CALL openhouse.system.set_current_snapshot('${catalogRelative(table)}', $first)") - } { view => - assert(view.after.size == 3) + assert(addedRowCount == 2) + }) } - def restoreRollbackOps(fmt: String): List[(String, TableTest[CoreTable.type])] = List( - "restore.rollbackToSnapshot" -> restoreRollbackToSnapshot(fmt), - "restore.setCurrentSnapshot" -> restoreSetCurrentSnapshot(fmt) - ) - - // ── Maintenance OPERATIONS (Iceberg CALL procedures; jobs merely orchestrate these) ────────── - // SE / OFD / compaction are stored procedures, reachable from Spark SQL like rollback/set_current. - // Each mutates physical state; we assert the current DATA is preserved and observe the metadata delta. - def maintenanceExpireSnapshots(fmt: String): TableTest[CoreTable.type] = - coreTwoSnapshots(fmt).step("maintenance.expireSnapshots") { (spark, table) => - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - } { view => - assert(view.after.size == 5, "expire_snapshots changed the current data") - assert(view.snapshotsAfter < view.snapshotsBefore, s"expire did not drop a snapshot: ${view.snapshotsBefore} -> ${view.snapshotsAfter}") - } + val restoreRollbackCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + val preparation = TablePreparation( + format, + coreTwoSnapshots(format)) - def maintenanceRewriteDataFiles(fmt: String): TableTest[CoreTable.type] = - coreTwoSnapshots(fmt).step("maintenance.rewriteDataFiles") { (spark, table) => - spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}')") - } { view => - assert(view.after.size == 5, "compaction changed rows") // rows preserved - } + List( + preparation.test("restore.rollbackToSnapshot") { table => + val firstSnapshotId = + snapshotIds(table.spark, table.name).head + + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $firstSnapshotId)") - def maintenanceRemoveOrphanFiles(fmt: String): TableTest[CoreTable.type] = - coreTwoSnapshots(fmt).step("maintenance.removeOrphanFiles") { (spark, table) => - // older_than must be ≥24h in the past (a safety guard); a far-past ts is a valid no-op that - // still exercises the procedure end-to-end without corrupting live files. - spark.sql(s"CALL openhouse.system.remove_orphan_files(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2020-01-01 00:00:00')") - } { view => - assert(view.after.size == 5, "orphan removal changed rows") + assert(table.rows.size == 3) + }, + preparation.test("restore.setCurrentSnapshot") { table => + val firstSnapshotId = + snapshotIds(table.spark, table.name).head + + table.spark.sql( + "CALL openhouse.system.set_current_snapshot(" + + s"'${catalogRelative(table.name)}', $firstSnapshotId)") + + assert(table.rows.size == 3) + }) } - def maintenanceOps(fmt: String): List[(String, TableTest[CoreTable.type])] = List( - "maintenance.expireSnapshots" -> maintenanceExpireSnapshots(fmt), - "maintenance.rewriteDataFiles" -> maintenanceRewriteDataFiles(fmt), - "maintenance.removeOrphanFiles" -> maintenanceRemoveOrphanFiles(fmt) - ) + val maintenanceCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + val preparation = TablePreparation( + format, + coreTwoSnapshots(format)) + + List( + preparation.test("maintenance.expireSnapshots") { table => + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + assert( + table.rows.size == 5, + "expire_snapshots changed the current data") + assert( + table.snapshotCount < table.preparedSnapshotCount, + "expire_snapshots did not remove a snapshot: " + + s"${table.preparedSnapshotCount} -> ${table.snapshotCount}") + }, + preparation.test("maintenance.rewriteDataFiles") { table => + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}')") + + assert(table.rows.size == 5, "compaction changed rows") + }, + preparation.test("maintenance.removeOrphanFiles") { table => + table.spark.sql( + "CALL openhouse.system.remove_orphan_files(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2020-01-01 00:00:00')") + + assert(table.rows.size == 5, "orphan removal changed rows") + }) + } // ── Control-plane (REST) ops with no SQL surface — driven via the embedded server's HTTP API ── // Lock enforcement: POST /lock (a real public entry), then a Spark mutation is rejected server-side @@ -159,10 +202,14 @@ trait MaintControlScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $table") } - val controlPlane: List[(String, Ctx => Unit)] = List( - "control.lock.enforcement" -> controlLockEnforcement, - "control.undrop.lifecycle" -> controlUndropLifecycle - ) + val controlPlaneCases: List[Plan.Case] = + List( + Plan.Case( + "control.lock.enforcement @ embedded", + controlLockEnforcement), + Plan.Case( + "control.undrop.lifecycle @ embedded", + controlUndropLifecycle)) // ── Undrop admin-lifecycle block (Phase 5 — REAL HTS only, HtsAdmin.enabled) ───────────────── // With an embedded real HTS the full soft-delete → list → restore / purge lifecycle is exercisable @@ -204,11 +251,21 @@ trait MaintControlScenarios extends ScenarioKit { assert(rs >= 400, s"restore after purge must be rejected, got $rs") } - val undropAdminOps: List[(String, Ctx => Unit)] = List( - "undropAdmin.restoreRoundTrip" -> undropAdminRestoreRoundTrip, - "undropAdmin.listSoftDeleted" -> undropAdminListSoftDeleted, - "undropAdmin.restoreAfterPurgeRejected" -> undropAdminRestoreAfterPurgeRejected - ) + def undropAdminCases: List[Plan.Case] = + if (HtsAdmin.enabled) { + List( + Plan.Case( + "undropAdmin.restoreRoundTrip", + undropAdminRestoreRoundTrip), + Plan.Case( + "undropAdmin.listSoftDeleted", + undropAdminListSoftDeleted), + Plan.Case( + "undropAdmin.restoreAfterPurgeRejected", + undropAdminRestoreAfterPurgeRejected)) + } else { + Nil + } } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala index f006cf1c0..8d9769875 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala @@ -20,44 +20,123 @@ trait MorMaintScenarios extends ScenarioKit { // operating on a table that ALREADY carries a live position-delete file — data-file/delete-file // COEXISTENCE. `createAndSeedMorDeleted` leaves 2 rows (keys 2,3) with a live delete for key 1; // these ops then act on that state. - val morCoexistOps: List[(String, TableTest[CoreTable.type])] = List( - // A new data file must coexist with the existing delete file; the read applies the delete to - // OLD data only, not the appended rows. - "coexist.append" -> TableTest(Core).step("coexist.append") { (spark, table) => - spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "append over live delete file wrong count") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 0, "deleted row resurrected by append") - }(), - // A second delete adds a second position-delete file over the same data file. - "coexist.secondDelete" -> TableTest(Core).step("coexist.secondDelete") { (spark, table) => - spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 2") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 1, "second delete over existing delete file wrong count") - assert(spark.sql(s"SELECT count(*) FROM $table.all_delete_files").collect()(0).getLong(0) >= 1, "delete files missing after second delete") - }(), - // Update a surviving row while a delete file is live. - "coexist.update" -> TableTest(Core).step("coexist.update") { (spark, table) => - spark.sql(s"UPDATE $table SET ${Core.string0.columnName} = 'cx' WHERE ${Core.long0.columnName} = 3") - assert(spark.sql(s"SELECT ${Core.string0.columnName} FROM $table WHERE ${Core.long0.columnName} = 3").collect()(0).getString(0) == "cx", "update over live delete failed") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "update over live delete changed count") - }(), - // A filtered read must apply the position delete (the deleted key must never appear). - "coexist.readFilter" -> TableTest(Core).step("coexist.readFilter") { (spark, table) => - val keys = spark.sql(s"SELECT ${Core.long0.columnName} FROM $table WHERE ${Core.long0.columnName} <= 2 ORDER BY ${Core.long0.columnName}").collect().toSeq.map(_.getLong(0)) - assert(keys == Seq(2L), s"filter must apply the position delete (key 1 gone): $keys") - }(), - // Compacting the position deletes materializes them; the row set is unchanged. - "coexist.compactDeletes" -> TableTest(Core).step("coexist.compactDeletes") { (spark, table) => - spark.sql(s"CALL openhouse.system.rewrite_position_delete_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "compact position deletes changed row set") - }(), - // Merge onto a table with a live delete file. - "coexist.merge" -> TableTest(Core).step("coexist.merge") { (spark, table) => - spark.sql(s"MERGE INTO $table t USING (SELECT CAST(3 AS BIGINT) k) s ON t.${Core.long0.columnName} = s.k " + - s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'mg'") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "merge over live delete changed count") - assert(spark.sql(s"SELECT ${Core.string0.columnName} FROM $table WHERE ${Core.long0.columnName} = 3").collect()(0).getString(0) == "mg", "merge over live delete failed") - }() - ) + val morCoexistCases: List[Plan.Case] = + morVerifyLayouts + .map(layout => + TablePreparation( + layout.label, + createAndSeedMorDeleted(layout, 3))) + .flatMap { preparation => + List( + preparation.test("coexist.append") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 3, + "append over a live delete file returned the wrong row count") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + .collect()(0) + .getLong(0) == 0, + "append resurrected the deleted row") + }, + preparation.test("coexist.secondDelete") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 1, + "second delete returned the wrong row count") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}.all_delete_files") + .collect()(0) + .getLong(0) >= 1, + "delete files are missing after the second delete") + }, + preparation.test("coexist.update") { table => + table.spark.sql( + s"UPDATE ${table.name} " + + s"SET ${Core.string0.columnName} = 'cx' " + + s"WHERE ${Core.long0.columnName} = 3") + + assert( + table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 3") + .collect()(0) + .getString(0) == "cx", + "update over a live delete file failed") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "update over a live delete file changed the row count") + }, + preparation.test("coexist.readFilter") { table => + val keys = table.spark + .sql( + s"SELECT ${Core.long0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2 " + + s"ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) + + assert( + keys == Seq(2L), + s"filter did not apply the position delete: $keys") + }, + preparation.test("coexist.compactDeletes") { table => + table.spark.sql( + "CALL openhouse.system.rewrite_position_delete_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "position-delete compaction changed the row set") + }, + preparation.test("coexist.merge") { table => + table.spark.sql( + s"MERGE INTO ${table.name} target " + + "USING (SELECT CAST(3 AS BIGINT) key) source " + + s"ON target.${Core.long0.columnName} = source.key " + + "WHEN MATCHED THEN UPDATE " + + s"SET ${Core.string0.columnName} = 'mg'") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "merge over a live delete file changed the row count") + assert( + table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 3") + .collect()(0) + .getString(0) == "mg", + "merge over a live delete file failed") + }) + } // ── Maintenance × MoR-with-live-delete (BUILD-STATUS block 8 deepening) ────────────────────── // The maintenance.* block runs on plain CoW; the genuinely-distinct surface is maintenance over a @@ -71,95 +150,276 @@ trait MorMaintScenarios extends ScenarioKit { // carries a live delete-file reference that points at data already removed; it lingers until // rewrite_position_delete_files or expire_snapshots. Reads stay correct throughout. Crossed × 3 MoR // formats to confirm the behavior is format-consistent (the delete decode differs per format). - val maintenanceMorFoldOps: List[(String, TableTest[CoreTable.type])] = List( - "maint.mor.rewriteDataFilesDanglingDelete" -> TableTest(Core).step("maint.mor.rewriteDataFilesDanglingDelete") { (spark, table) => - spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") - // the delete IS applied logically — row set is correct - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "rewrite_data_files changed the live row set over a MoR delete") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 0, "rewrite_data_files RESURRECTED the deleted row") - // G14 PIN: the position delete is NOT removed from the current snapshot — it dangles. - val delFiles = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) - assert(delFiles == 1, s"characterized: rewrite_data_files leaves the position delete dangling in the current snapshot (expected 1), got $delFiles — if this is 0, the build now folds deletes and the pin should flip") - // despite the dangling delete, reads remain correct (the removed row never reappears) - val keys = spark.sql(s"SELECT ${Core.long0.columnName} FROM $table WHERE ${Core.long0.columnName} <= 2 ORDER BY ${Core.long0.columnName}").collect().toSeq.map(_.getLong(0)) - assert(keys == Seq(2L), s"read after rewrite_data_files must stay correct despite the dangling delete: $keys") - }(), - // D5 DECIDER (owner: G14 is a BUG unless the recovery path works, then a PIN): does - // `rewrite_position_delete_files` actually FOLD OUT the dangling position delete that - // rewrite_data_files leaves behind (delete_files 1 -> 0)? If yes, the operator has a working - // additional-maintenance recovery (G14 = pin); if no, the dangling delete is unrecoverable via the - // documented procedure (G14 = bug). Reads must stay correct throughout. × 3 MoR formats. - "maint.mor.rewritePositionDeleteFolds" -> TableTest(Core).step("maint.mor.rewritePositionDeleteFolds") { (spark, table) => - // 1) rewrite_data_files leaves a dangling position delete (the G14 state). - spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") - val danglingBefore = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) - // 2) the recovery path: rewrite_position_delete_files — does it fold the dangling delete out? - spark.sql(s"CALL openhouse.system.rewrite_position_delete_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") - val danglingAfter = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) - println(s"DIAG maint.mor.rewritePositionDeleteFolds: delete_files before=$danglingBefore after=$danglingAfter") - // reads must stay correct regardless (key 1 removed, 2 live rows). - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "rewrite_position_delete_files changed the live row set") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 0, "rewrite_position_delete_files resurrected the deleted row") - // D5 PIN: the recovery WORKS — rewrite_position_delete_files folds the dangling delete out. - assert(danglingBefore == 1 && danglingAfter == 0, - s"D5: expected rewrite_position_delete_files to FOLD the dangling delete (before=1 -> after=0); got before=$danglingBefore after=$danglingAfter — if after>0 the recovery path does NOT work and G14 must be reclassified from pin to BUG") - }() - ) + val maintenanceMorFoldCases: List[Plan.Case] = + morVerifyLayouts + .map(layout => + TablePreparation( + layout.label, + createAndSeedMorDeleted(layout, 3))) + .flatMap { preparation => + List( + preparation.test("maint.mor.rewriteDataFilesDanglingDelete") { table => + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "rewrite_data_files changed the live row set") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + .collect()(0) + .getLong(0) == 0, + "rewrite_data_files resurrected the deleted row") + + val deleteFileCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.delete_files") + .collect()(0) + .getLong(0) + val keys = table.spark + .sql( + s"SELECT ${Core.long0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2 " + + s"ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) + + assert( + deleteFileCount == 1, + "rewrite_data_files should leave one dangling position delete, " + + s"got $deleteFileCount") + assert( + keys == Seq(2L), + s"read after rewrite_data_files returned incorrect keys: $keys") + }, + preparation.test("maint.mor.rewritePositionDeleteFolds") { table => + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + val deleteFilesBefore = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.delete_files") + .collect()(0) + .getLong(0) + + table.spark.sql( + "CALL openhouse.system.rewrite_position_delete_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + val deleteFilesAfter = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.delete_files") + .collect()(0) + .getLong(0) + + println( + "DIAG maint.mor.rewritePositionDeleteFolds: " + + s"delete_files before=$deleteFilesBefore after=$deleteFilesAfter") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "rewrite_position_delete_files changed the live row set") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + .collect()(0) + .getLong(0) == 0, + "rewrite_position_delete_files resurrected the deleted row") + assert( + deleteFilesBefore == 1 && deleteFilesAfter == 0, + "rewrite_position_delete_files should fold the dangling delete: " + + s"before=$deleteFilesBefore after=$deleteFilesAfter") + }) + } // Metadata-only maintenance over a live delete — format is vacuous (these never decode the delete // file), so × 1 MoR layout. Each must PRESERVE the delete (2 live rows, key 1 still gone). - val maintenanceMorMetaOps: List[(String, TableTest[CoreTable.type])] = List( - "maint.mor.expireSnapshots" -> TableTest(Core).step("maint.mor.expireSnapshots") { (spark, table) => - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "expire_snapshots changed the live row set over a MoR delete") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 0, "expire_snapshots resurrected the deleted row") - }(), - "maint.mor.rewriteManifests" -> TableTest(Core).step("maint.mor.rewriteManifests") { (spark, table) => - spark.sql(s"CALL openhouse.system.rewrite_manifests(table => '${catalogRelative(table)}', use_caching => false)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "rewrite_manifests changed the live row set over a MoR delete") - }(), - "maint.mor.removeOrphanFiles" -> TableTest(Core).step("maint.mor.removeOrphanFiles") { (spark, table) => - spark.sql(s"CALL openhouse.system.remove_orphan_files(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2020-01-01 00:00:00')") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "remove_orphan_files changed the live row set over a MoR delete") - }(), - // Modality: compact the position deletes, THEN expire the pre-compact snapshot — the folded - // state must survive (the deleted row must not reappear via the retained/expired lineage). - "maint.mor.compactThenExpire" -> TableTest(Core).step("maint.mor.compactThenExpire") { (spark, table) => - spark.sql(s"CALL openhouse.system.rewrite_position_delete_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "compact-then-expire changed the live row set") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 0, "compact-then-expire resurrected the deleted row") - }() - ) + val maintenanceMorMetaCases: List[Plan.Case] = + morVerifyLayouts + .filter(layout => + layout.label == "mor-verify/parquet" || + layout.label == "mor-verify/orc") + .map(layout => + TablePreparation( + layout.label, + createAndSeedMorDeleted(layout, 3))) + .flatMap { preparation => + List( + preparation.test("maint.mor.expireSnapshots") { table => + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "expire_snapshots changed the live row set") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + .collect()(0) + .getLong(0) == 0, + "expire_snapshots resurrected the deleted row") + }, + preparation.test("maint.mor.rewriteManifests") { table => + table.spark.sql( + "CALL openhouse.system.rewrite_manifests(" + + s"table => '${catalogRelative(table.name)}', " + + "use_caching => false)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "rewrite_manifests changed the live row set") + }, + preparation.test("maint.mor.removeOrphanFiles") { table => + table.spark.sql( + "CALL openhouse.system.remove_orphan_files(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2020-01-01 00:00:00')") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "remove_orphan_files changed the live row set") + }, + preparation.test("maint.mor.compactThenExpire") { table => + table.spark.sql( + "CALL openhouse.system.rewrite_position_delete_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "compact-then-expire changed the live row set") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + .collect()(0) + .getLong(0) == 0, + "compact-then-expire resurrected the deleted row") + }) + } // ── MoR delete-file modality hazards (BUILD-STATUS block 10 deepening) ─────────────────────── // A live position delete is snapshot-scoped state. These hunt for it being mis-resolved across the // history/restore axes: a delete must NOT be retroactive (pre-delete snapshots still see the row), // rollback must UNDO it, and it must SURVIVE expiration of older snapshots. Time-travel/rollback // logic is format-vacuous (it resolves snapshots, not file bytes) → × 1 MoR layout. - val morHazardOps: List[(String, TableTest[CoreTable.type])] = List( - // The delete is snapshot-scoped: time-travel to the pre-delete snapshot still sees key 1. - "hazard.mor.timeTravelBeforeDelete" -> TableTest(Core).step("hazard.mor.timeTravelBeforeDelete") { (spark, table) => - val seedSnap = spark.sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getLong(0) - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "current MoR state should have the delete applied") - assert(spark.sql(s"SELECT count(*) FROM $table VERSION AS OF $seedSnap").collect()(0).getLong(0) == 3, - "pre-delete snapshot must still see the deleted row (delete must not be retroactive)") - }(), - // Rollback to the pre-delete snapshot UNDOES the delete — the row returns and no delete is live. - "hazard.mor.rollbackUndoesDelete" -> TableTest(Core).step("hazard.mor.rollbackUndoesDelete") { (spark, table) => - val seedSnap = spark.sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getLong(0) - spark.sql(s"CALL openhouse.system.rollback_to_snapshot(table => '${catalogRelative(table)}', snapshot_id => ${seedSnap}L)") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "rollback did not undo the MoR delete") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1").collect()(0).getLong(0) == 1, "rolled-back row not restored") - }(), - // The delete must SURVIVE expiration of the older (pre-delete) snapshot — a filtered read still - // excludes key 1 after expire. - "hazard.mor.expireThenDeleteHolds" -> TableTest(Core).step("hazard.mor.expireThenDeleteHolds") { (spark, table) => - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - val keys = spark.sql(s"SELECT ${Core.long0.columnName} FROM $table WHERE ${Core.long0.columnName} <= 2 ORDER BY ${Core.long0.columnName}").collect().toSeq.map(_.getLong(0)) - assert(keys == Seq(2L), s"delete must survive expiration of the pre-delete snapshot (key 1 gone): $keys") - }() - ) + val morHazardCases: List[Plan.Case] = + morVerifyLayouts + .filter(layout => + layout.label == "mor-verify/parquet" || + layout.label == "mor-verify/orc") + .map(layout => + TablePreparation( + layout.label, + createAndSeedMorDeleted(layout, 3))) + .flatMap { preparation => + List( + preparation.test("hazard.mor.timeTravelBeforeDelete") { table => + val seedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "ORDER BY committed_at LIMIT 1") + .collect()(0) + .getLong(0) + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "current merge-on-read state should apply the delete") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF $seedSnapshotId") + .collect()(0) + .getLong(0) == 3, + "the snapshot before the delete should still contain the row") + }, + preparation.test("hazard.mor.rollbackUndoesDelete") { table => + val seedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "ORDER BY committed_at LIMIT 1") + .collect()(0) + .getLong(0) + + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"table => '${catalogRelative(table.name)}', " + + s"snapshot_id => ${seedSnapshotId}L)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 3, + "rollback did not undo the merge-on-read delete") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + .collect()(0) + .getLong(0) == 1, + "rollback did not restore the deleted row") + }, + preparation.test("hazard.mor.expireThenDeleteHolds") { table => + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + val keys = table.spark + .sql( + s"SELECT ${Core.long0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2 " + + s"ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) + + assert( + keys == Seq(2L), + s"delete did not survive snapshot expiration: $keys") + }) + } // ── MoR × branch MERGE (position deletes carried across fast_forward / cherry_pick / REPLACE BRANCH) ── // A DELETE/UPDATE on a branch of a MoR table writes position-delete files ON THE BRANCH; merging the @@ -169,53 +429,145 @@ trait MorMaintScenarios extends ScenarioKit { // is a real position delete, not a file elimination. Merge is a ref/snapshot carry → format-vacuous // (× 1 MoR layout). Each hunts for: deletes lost/not-carried, deleted rows resurrecting on main, // cherry-pick rejecting row-delete snapshots. - val morBranchMergeOps: List[(String, TableTest[CoreTable.type])] = List( - // fast_forward must carry a branch position-delete into main: after merge the deleted row is gone. - "mbranch.fastForwardDelete" -> TableTest(Core).step("mbranch.fastForwardDelete") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH mfb") - spark.sql(s"DELETE FROM $table.branch_mfb WHERE ${Core.long0.columnName} = 1") // position delete on branch - assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "main advanced before merge") - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'mfb'") == "2", "branch delete not applied on the branch") - spark.sql(s"CALL openhouse.system.fast_forward('${catalogRelative(table)}', 'main', 'mfb')") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "2", "fast_forward did not carry the branch position-delete to main") - assert(countOf(spark, s"SELECT count(*) FROM $table WHERE ${Core.long0.columnName} = 1") == "0", "deleted row resurrected on main after fast_forward") - }(), - // fast_forward must carry a branch UPDATE (MoR update = position delete + new data file). - "mbranch.fastForwardUpdate" -> TableTest(Core).step("mbranch.fastForwardUpdate") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH mub") - spark.sql(s"UPDATE $table.branch_mub SET ${Core.string0.columnName} = 'br-upd' WHERE ${Core.long0.columnName} = 2") - spark.sql(s"CALL openhouse.system.fast_forward('${catalogRelative(table)}', 'main', 'mub')") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "fast_forward of a MoR update changed the row count on main") - assert(spark.sql(s"SELECT ${Core.string0.columnName} FROM $table WHERE ${Core.long0.columnName} = 2").collect()(0).getString(0) == "br-upd", - "MoR update not carried to main by fast_forward") - }(), - // Cherry-pick a branch ROW-DELETE snapshot onto main — CHARACTERIZE (the fragile path): it either - // applies the delete (main → 2) or is rejected; pin the outcome and assert the row set matches it. - "mbranch.cherrypickDelete" -> TableTest(Core).step("mbranch.cherrypickDelete") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH mcb") - spark.sql(s"DELETE FROM $table.branch_mcb WHERE ${Core.long0.columnName} = 1") - val delSnap = spark.sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at DESC LIMIT 1").collect()(0).getLong(0) - val outcome = - try { spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${delSnap}L)"); "ok" } - catch { case NonFatal(e) => s"rejected:${Exceptions.root(e).getClass.getSimpleName}" } - val mainCount = countOf(spark, s"SELECT count(*) FROM $table") - println(s"DIAG mbranch.cherrypickDelete: $outcome, mainCount=$mainCount") - if (outcome == "ok") - assert(mainCount == "2", s"cherrypick reported ok but did not apply the branch delete to main (got $mainCount)") - else - assert(mainCount == "3", s"cherrypick was rejected but main changed anyway (got $mainCount)") - }(), - // REPLACE BRANCH retargets a MoR branch to a pre-delete snapshot — the delete must follow the target. - "mbranch.replaceBranchDelete" -> TableTest(Core).step("mbranch.replaceBranchDelete") { (spark, table) => - val preSnap = spark.sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at DESC LIMIT 1").collect()(0).getLong(0) // seed (3 rows) - spark.sql(s"ALTER TABLE $table CREATE BRANCH mrb") - spark.sql(s"DELETE FROM $table.branch_mrb WHERE ${Core.long0.columnName} = 1") - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'mrb'") == "2", "branch delete not applied") - spark.sql(s"ALTER TABLE $table REPLACE BRANCH mrb AS OF VERSION $preSnap") - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'mrb'") == "3", - "REPLACE BRANCH to the pre-delete snapshot did not undo the branch position-delete") - }() - ) + val morBranchMergeCases: List[Plan.Case] = + morVerifyLayouts + .filter(layout => + layout.label == "mor-verify/parquet" || + layout.label == "mor-verify/orc") + .map(layout => + TablePreparation( + layout.label, + createAndSeedSingleFile(layout, 3))) + .flatMap { preparation => + List( + preparation.test("mbranch.fastForwardDelete") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mfb") + table.spark.sql( + s"DELETE FROM ${table.name}.branch_mfb " + + s"WHERE ${Core.long0.columnName} = 1") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "main advanced before fast-forward") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mfb'") == "2", + "branch delete was not applied") + + table.spark.sql( + "CALL openhouse.system.fast_forward(" + + s"'${catalogRelative(table.name)}', 'main', 'mfb')") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "2", + "fast-forward did not carry the branch position delete") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") == "0", + "deleted row reappeared after fast-forward") + }, + preparation.test("mbranch.fastForwardUpdate") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mub") + table.spark.sql( + s"UPDATE ${table.name}.branch_mub " + + s"SET ${Core.string0.columnName} = 'br-upd' " + + s"WHERE ${Core.long0.columnName} = 2") + table.spark.sql( + "CALL openhouse.system.fast_forward(" + + s"'${catalogRelative(table.name)}', 'main', 'mub')") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "fast-forward of an update changed the main row count") + assert( + table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 2") + .collect()(0) + .getString(0) == "br-upd", + "fast-forward did not carry the branch update") + }, + preparation.test("mbranch.cherrypickDelete") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mcb") + table.spark.sql( + s"DELETE FROM ${table.name}.branch_mcb " + + s"WHERE ${Core.long0.columnName} = 1") + val deleteSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "ORDER BY committed_at DESC LIMIT 1") + .collect()(0) + .getLong(0) + val outcome = + try { + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', ${deleteSnapshotId}L)") + "ok" + } catch { + case NonFatal(exception) => + s"rejected:${Exceptions.root(exception).getClass.getSimpleName}" + } + val mainCount = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") + + println( + s"DIAG mbranch.cherrypickDelete: $outcome, mainCount=$mainCount") + if (outcome == "ok") { + assert( + mainCount == "2", + "cherry-pick reported success without applying the branch delete") + } else { + assert( + mainCount == "3", + "cherry-pick was rejected after changing main") + } + }, + preparation.test("mbranch.replaceBranchDelete") { table => + val seedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "ORDER BY committed_at DESC LIMIT 1") + .collect()(0) + .getLong(0) + + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mrb") + table.spark.sql( + s"DELETE FROM ${table.name}.branch_mrb " + + s"WHERE ${Core.long0.columnName} = 1") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mrb'") == "2", + "branch delete was not applied") + + table.spark.sql( + s"ALTER TABLE ${table.name} REPLACE BRANCH mrb " + + s"AS OF VERSION $seedSnapshotId") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mrb'") == "3", + "replacing the branch target did not undo its position delete") + }) + } // Encryption capability PIN (characterization). OpenHouse delegates table-data encryption to an // external KMS plugin (private repo); in OSS the catalog never wires a KeyManagementClient, so @@ -224,18 +576,34 @@ trait MorMaintScenarios extends ScenarioKit { // encryption — robust regardless of compression. This pins that OSS writes plaintext; it FLIPS to // "PARE" the moment table-data encryption is wired (then update BUGS.md and this pin). An off-the- // shelf KMS does NOT change this — nothing in the OpenHouse write path invokes the encryption hook. - val encryptionPlaintextPin: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.pin.dataPlaintext") { (spark, table) => - val path = spark.sql(s"SELECT file_path FROM $table.data_files LIMIT 1").collect()(0).getString(0) - val local = path.stripPrefix("file:") - val bytes = java.nio.file.Files.readAllBytes(java.nio.file.Paths.get(local)) - assert(bytes.length >= 8, s"data file too small to inspect: ${bytes.length} bytes") + val encryptionPinCases: List[Plan.Case] = { + val preparation = TablePreparation( + "parquet", + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "TBLPROPERTIES ('write.format.default'='parquet')")() + .insert(3)()) + + List( + preparation.test("surface.pin.dataPlaintext") { table => + val dataFilePath = table.spark + .sql(s"SELECT file_path FROM ${table.name}.data_files LIMIT 1") + .collect()(0) + .getString(0) + .stripPrefix("file:") + val bytes = java.nio.file.Files.readAllBytes( + java.nio.file.Paths.get(dataFilePath)) + + assert( + bytes.length >= 8, + s"data file is too small to inspect: ${bytes.length} bytes") val footerMagic = new String(bytes.takeRight(4), "US-ASCII") - assert(footerMagic == "PAR1", - s"expected UNENCRYPTED parquet footer magic PAR1 (OSS encryption is un-wired — capability gap, BUGS.md); " + - s"got '$footerMagic' — if 'PARE', table-data encryption is now active and this pin should flip to assert ciphertext") - }() + assert( + footerMagic == "PAR1", + s"expected plaintext Parquet footer PAR1, got $footerMagic") + }) + } } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala index 081b4f492..0799084d9 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala @@ -13,390 +13,455 @@ import scala.util.control.NonFatal trait NegativeDdlScenarios extends ScenarioKit { import Rows._ - // ── negative / contract tests ─────────────────────────────────────────────────────────── - // Create + seed a valid CoreTable, then assert the bad operation is rejected. - private def coreNegative(label: String)(bad: (SparkSession, String) => Unit): TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt')")() - .insert(3)() - .step(label)(bad)() - private val S = CoreTable.string0.columnName - // Each negative asserts BOTH the exception type and a message substring, so it verifies the - // operation was rejected for the RIGHT reason (not merely that something threw). - val negNonExistentColumn: TableTest[CoreTable.type] = - coreNegative("negative.nonExistentColumn") { (spark, table) => - val e = Check.intercept[AnalysisException](spark.sql(s"DELETE FROM $table WHERE no_such_column = 1")) - assert(e.getMessage.contains("no_such_column")) - } - - val negNonDeterministicDelete: TableTest[CoreTable.type] = - coreNegative("negative.nonDeterministicDelete") { (spark, table) => - val e = Check.intercept[AnalysisException](spark.sql(s"DELETE FROM $table WHERE rand() < 0.5")) - assert(e.getMessage.toLowerCase.contains("deterministic")) - } - - val negNonDeterministicUpdate: TableTest[CoreTable.type] = - coreNegative("negative.nonDeterministicUpdate") { (spark, table) => - val e = Check.intercept[AnalysisException](spark.sql(s"UPDATE $table SET $S = 'x' WHERE rand() < 0.5")) - assert(e.getMessage.toLowerCase.contains("deterministic")) - } - - val negInsertArity: TableTest[CoreTable.type] = - coreNegative("negative.insertArity") { (spark, table) => - val e = Check.intercept[AnalysisException](spark.sql(s"INSERT INTO $table VALUES (CAST(1 AS BIGINT), 1)")) // too few columns - assert(e.getMessage.toLowerCase.contains("not enough data columns")) - } - - // Two UPDATE assignments to the same column in one MERGE clause → analysis error. - val negMergeConflictingUpdates: TableTest[CoreTable.type] = - coreNegative("negative.mergeConflictingUpdates") { (spark, table) => - val e = Check.intercept[AnalysisException](spark.sql( - s"""MERGE INTO $table t USING (SELECT * FROM VALUES (CAST(2 AS BIGINT)) AS s($L)) s - ON t.$L = s.$L - WHEN MATCHED THEN UPDATE SET t.$S = 'a', t.$S = 'b'""")) - assert(e.getMessage.contains("Multiple assignments")) - } - - // Source has two rows matching the same target row → cardinality violation at RUNTIME. The - // concrete runtime exception class (SparkRuntimeException) is package-private, so we anchor on - // the specific message across the cause chain (the error may be wrapped in a task failure). - val negMergeCardinalityViolation: TableTest[CoreTable.type] = - coreNegative("negative.mergeCardinalityViolation") { (spark, table) => - val e = Check.intercept[Exception](spark.sql( - s"""MERGE INTO $table t USING ( - SELECT * FROM VALUES (CAST(2 AS BIGINT), 'a'), (CAST(2 AS BIGINT), 'b') AS s($L, $S) - ) s ON t.$L = s.$L - WHEN MATCHED THEN UPDATE SET t.$S = s.$S""")) - assert( - Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(_.contains("matched a single row from the target table"))), - s"expected a MERGE cardinality-violation message, got: ${e.getMessage}") - } - - // CREATE partitioned by a non-existent column (on a scratch name, valid managed table stays). - val negPartitionByNonExistent: TableTest[CoreTable.type] = - coreNegative("negative.partitionByNonExistent") { (spark, table) => - val scratch = table + "_x" - val e = Check.intercept[AnalysisException](spark.sql( - s"CREATE TABLE $scratch ($columnDefinitions) USING $dataSource PARTITIONED BY (no_such_column) TBLPROPERTIES ('write.format.default'='$seedFmt')")) - spark.sql(s"DROP TABLE IF EXISTS $scratch") - assert(e.getMessage.contains("no_such_column")) - } - - val negatives: List[(String, TableTest[CoreTable.type])] = List( - "negative.nonExistentColumn" -> negNonExistentColumn, - "negative.nonDeterministicDelete" -> negNonDeterministicDelete, - "negative.nonDeterministicUpdate" -> negNonDeterministicUpdate, - "negative.insertArity" -> negInsertArity, - "negative.mergeConflictingUpdates" -> negMergeConflictingUpdates, - "negative.mergeCardinalityViolation" -> negMergeCardinalityViolation, - "negative.partitionByNonExistent" -> negPartitionByNonExistent - ) - - // ── DDL Phase 13: schema-evolution negatives ──────────────────────────────────────────── - // DROP COLUMN fails at COMMIT (server 400 → Iceberg BadRequestException); the message carries the - // full body incl. schema dump (AUDIT-FINDINGS B — a "dumb" message), so we anchor on the meaningful - // "Some columns are dropped" reason. Narrowing / SET NOT NULL are caught earlier at Spark analysis - // (ExtendedAnalysisException, a subtype of AnalysisException) with clean messages. - // NOTE: RENAME COLUMN is NOT rejected — it is supported (see ddlRenameColumn in Phase 12). - // DROP COLUMN rejects — but the message is `Column[foo_col_int] not found in newSchema` (buried in a - // double schema dump); it never says "you cannot drop columns" (AUDIT-FINDINGS B, a readability gap). - val ddlNegDropColumn: TableTest[CoreTable.type] = - coreNegative("ddl.neg.dropColumn") { (spark, table) => - val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table DROP COLUMN ${Core.int0.columnName}")) - assert(e.getMessage.contains("not found in newSchema"), s"unexpected message: ${e.getMessage.take(160)}") - assert(e.getMessage.contains(Core.int0.columnName), s"message should name the dropped column: ${e.getMessage.take(160)}") + val negativeCases: List[Plan.Case] = + preparedCoreFormats.flatMap { preparation => + List( + preparation.test("negative.nonExistentColumn") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"DELETE FROM ${table.name} WHERE no_such_column = 1")) + + assert(exception.getMessage.contains("no_such_column")) + }, + preparation.test("negative.nonDeterministicDelete") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"DELETE FROM ${table.name} WHERE rand() < 0.5")) + + assert( + exception.getMessage.toLowerCase.contains("deterministic")) + }, + preparation.test("negative.nonDeterministicUpdate") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"UPDATE ${table.name} SET $S = 'x' WHERE rand() < 0.5")) + + assert( + exception.getMessage.toLowerCase.contains("deterministic")) + }, + preparation.test("negative.insertArity") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"INSERT INTO ${table.name} VALUES (CAST(1 AS BIGINT), 1)")) + + assert( + exception.getMessage.toLowerCase.contains( + "not enough data columns")) + }, + preparation.test("negative.mergeConflictingUpdates") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"""MERGE INTO ${table.name} target USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT)) AS source($L) + ) source + ON target.$L = source.$L + WHEN MATCHED THEN UPDATE + SET target.$S = 'a', target.$S = 'b'""")) + + assert(exception.getMessage.contains("Multiple assignments")) + }, + preparation.test("negative.mergeCardinalityViolation") { table => + val exception = Check.intercept[Exception]( + table.spark.sql( + s"""MERGE INTO ${table.name} target USING ( + SELECT * FROM VALUES + (CAST(2 AS BIGINT), 'a'), + (CAST(2 AS BIGINT), 'b') + AS source($L, $S) + ) source + ON target.$L = source.$L + WHEN MATCHED THEN UPDATE SET target.$S = source.$S""")) + + assert( + Exceptions.causeChain(exception).exists { cause => + Option(cause.getMessage).exists( + _.contains("matched a single row from the target table")) + }, + "expected a MERGE cardinality-violation message, got: " + + exception.getMessage) + }, + preparation.test("negative.partitionByNonExistent") { table => + val scratchTable = table.name + "_x" + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"CREATE TABLE $scratchTable ($columnDefinitions) " + + s"USING $dataSource PARTITIONED BY (no_such_column) " + + s"TBLPROPERTIES ('write.format.default'='${preparation.label}')")) + + table.spark.sql(s"DROP TABLE IF EXISTS $scratchTable") + assert(exception.getMessage.contains("no_such_column")) + }) } - val ddlNegNarrowType: TableTest[CoreTable.type] = - coreNegative("ddl.neg.narrowType") { (spark, table) => - val e = Check.intercept[AnalysisException](spark.sql(s"ALTER TABLE $table ALTER COLUMN ${Core.long0.columnName} TYPE int")) - assert(e.getMessage.contains("NOT_SUPPORTED_CHANGE_COLUMN"), s"unexpected message: ${e.getMessage.take(160)}") - } - - val ddlNegSetNotNull: TableTest[CoreTable.type] = - coreNegative("ddl.neg.setNotNull") { (spark, table) => - val e = Check.intercept[AnalysisException](spark.sql(s"ALTER TABLE $table ALTER COLUMN ${Core.string0.columnName} SET NOT NULL")) - assert(e.getMessage.contains("Cannot change nullable column to non-nullable"), s"unexpected message: ${e.getMessage.take(160)}") - } - - val ddlNegatives: List[(String, TableTest[CoreTable.type])] = List( - "ddl.neg.dropColumn" -> ddlNegDropColumn, - "ddl.neg.narrowType" -> ddlNegNarrowType, - "ddl.neg.setNotNull" -> ddlNegSetNotNull - ) - - // ── DDL Phase 14: table properties (user keys, reserved-key rejection, forced-override findings) ─ - // Self-contained pipelines (parquet) — property behavior is layout-invariant. `tableProps` reads - // back via SHOW TBLPROPERTIES. - - private def propsCreate(label: String, tblprops: String)(check: StepView[CoreTable.type] => Unit): TableTest[CoreTable.type] = - TableTest(Core).sql(label)(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ($tblprops)")(check) - - // user key round-trips: SET then read back, UNSET removes it - val ddlPropsUserRoundTrip: TableTest[CoreTable.type] = - TableTest(Core) - .sql("ddl.props.userRoundTrip.create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt')")() - .sql("ddl.props.userRoundTrip.set")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('my_key'='my_val')") { view => - assert(tableProps(view.spark, view.table).get("my_key").contains("my_val"), "user prop not set") - } - .sql("ddl.props.userRoundTrip.unset")(t => s"ALTER TABLE $t UNSET TBLPROPERTIES ('my_key')") { view => - assert(!tableProps(view.spark, view.table).contains("my_key"), "user prop not removed") - } - - // reserved-key rejection: an openhouse.* key hits the clean server guard (ALTER_RESERVED_TBLPROPS → - // 400 → BadRequestException). NOTE: `policies` specifically is value-parsed on the CLIENT first, so - // SET('policies'='x') throws a Gson JsonParseException before the guard — recorded in AUDIT-FINDINGS. - val ddlPropsReservedOpenhouse: TableTest[CoreTable.type] = - coreNegative("ddl.props.reservedOpenhouse") { (spark, table) => - val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('openhouse.tableUUID'='deadbeef')")) - assert(e.getMessage.toLowerCase.contains("restriction"), s"msg: ${e.getMessage.take(200)}") - } - - // finding: format-version is forced to the cluster default (2) — a create with '1' still reads 2 - val ddlPropsFormatVersionForced: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt', 'format-version'='1')")() - .insert(3)() - .check("ddl.props.formatVersionForced") { view => - val fv = tableProps(view.spark, view.table).get("format-version") - assert(fv.contains("2"), s"expected forced format-version=2, got $fv") - assert(view.after.size == 3, "table not writable at the forced format-version") // DML-after-DDL - } - - // honored-if-set: previous-versions-max the user provides survives - val ddlPropsPreviousVersionsHonored: TableTest[CoreTable.type] = - propsCreate("ddl.props.previousVersionsHonored", "'write.format.default'='$seedFmt', 'write.metadata.previous-versions-max'='7'") { view => - val v = tableProps(view.spark, view.table).get("write.metadata.previous-versions-max") - assert(v.contains("7"), s"expected previous-versions-max=7, got $v") - } - - val ddlPropsOperations: List[(String, TableTest[CoreTable.type])] = List( - "ddl.props.userRoundTrip" -> ddlPropsUserRoundTrip, - "ddl.props.reservedOpenhouse" -> ddlPropsReservedOpenhouse, - "ddl.props.formatVersionForced" -> ddlPropsFormatVersionForced, - "ddl.props.previousVersionsHonored"-> ddlPropsPreviousVersionsHonored - ) + val ddlNegativeCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => + List( + preparation.test("ddl.neg.dropColumn") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} DROP COLUMN ${Core.int0.columnName}")) + + assert( + exception.getMessage.contains("not found in newSchema"), + s"unexpected message: ${exception.getMessage.take(160)}") + assert( + exception.getMessage.contains(Core.int0.columnName), + s"message should name the dropped column: ${exception.getMessage.take(160)}") + }, + preparation.test("ddl.neg.narrowType") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.long0.columnName} TYPE int")) + + assert( + exception.getMessage.contains("NOT_SUPPORTED_CHANGE_COLUMN"), + s"unexpected message: ${exception.getMessage.take(160)}") + }, + preparation.test("ddl.neg.setNotNull") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.string0.columnName} SET NOT NULL")) + + assert( + exception.getMessage.contains("Cannot change nullable column to non-nullable"), + s"unexpected message: ${exception.getMessage.take(160)}") + }) + } + + val ddlPropertyCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => + val format = preparation.label + val formatVersionPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'format-version'='1')")() + .insert(3)()) + val previousVersionsPreparation = TablePreparation( + format, + TableTest(Core).sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'write.metadata.previous-versions-max'='7')")()) + + List( + preparation.test("ddl.props.userRoundTrip") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('my_key'='my_val')") + assert( + tableProps(table.spark, table.name).get("my_key").contains("my_val"), + "user prop not set") + + table.spark.sql(s"ALTER TABLE ${table.name} UNSET TBLPROPERTIES ('my_key')") + assert( + !tableProps(table.spark, table.name).contains("my_key"), + "user prop not removed") + }, + preparation.test("ddl.props.reservedOpenhouse") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + + "'openhouse.tableUUID'='deadbeef')")) + + assert( + exception.getMessage.toLowerCase.contains("restriction"), + s"msg: ${exception.getMessage.take(200)}") + }, + formatVersionPreparation.test("ddl.props.formatVersionForced") { table => + val formatVersion = tableProps(table.spark, table.name).get("format-version") + + assert( + formatVersion.contains("2"), + s"expected forced format-version=2, got $formatVersion") + assert( + table.rows.size == 3, + "table not writable at the forced format-version") + }, + previousVersionsPreparation.test("ddl.props.previousVersionsHonored") { table => + val previousVersions = + tableProps(table.spark, table.name).get("write.metadata.previous-versions-max") + + assert( + previousVersions.contains("7"), + s"expected previous-versions-max=7, got $previousVersions") + }) + } // Per-case "current seed format" (default parquet). The assembly's `crossFmt` sets it around each case // so a block multiplexes across formats WITHOUT every builder taking an explicit fmt param. Safe because // each case runs sequentially on its own worker thread (session-per-worker, parallel runner). This is // how format-INERT-by-hypothesis blocks (DDL/props/policy/branch/surface/negatives) get run on ORC too — - // ── DDL Phase 16: sort order / write distribution ─────────────────────────────────────── - // WRITE ORDERED BY sets the sort order; the observable side effect is write.distribution-mode=range - // (the recon's CatalogOperationTest asserts this). WRITE UNORDERED clears the order. - val ddlWriteOrderedBy: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("ddl.sortOrder.orderedBy")(t => s"ALTER TABLE $t WRITE ORDERED BY ${Core.long0.columnName}") { view => - assert(tableProps(view.spark, view.table).get("write.distribution-mode").contains("range"), - s"distribution-mode not range: ${tableProps(view.spark, view.table).get("write.distribution-mode")}") - } - - val ddlWriteOrderedByMulti: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("ddl.sortOrder.orderedByMulti")(t => - s"ALTER TABLE $t WRITE ORDERED BY ${Core.string0.columnName} DESC NULLS FIRST, ${Core.long0.columnName}") { view => - assert(tableProps(view.spark, view.table).get("write.distribution-mode").contains("range"), "multi-col ordered-by should set range") - } - .insert(2) { view => assert(view.after.size == 5, "multi-col ordered write path failed") } // DML-after-DDL - - // ── DDL Phase 17: rename table (rename to scratch + back, so the harness's fixed table name resolves) ─ - val ddlRenameTable: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("ddl.renameTable") { (spark, table) => - val scratch = s"${table}_ren" - spark.sql(s"ALTER TABLE $table RENAME TO $scratch") - assert(spark.sql(s"SELECT count(*) FROM $scratch").collect()(0).getLong(0) == 3, "renamed table lost rows") - Check.intercept[Exception](spark.sql(s"SELECT 1 FROM $table LIMIT 1")) // old name is gone - spark.sql(s"ALTER TABLE $scratch RENAME TO $table") // restore for teardown - }() - - val ddlRenameTableConflict: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("ddl.renameTable.conflict") { (spark, table) => - val other = s"${table}_other" - spark.sql(s"DROP TABLE IF EXISTS $other") - spark.sql(coreCreateParquet(other)) - val e = Check.intercept[WebClientResponseWithMessageException](spark.sql(s"ALTER TABLE $table RENAME TO $other")) // target exists - assert(e.getMessage.contains("already exists"), s"msg: ${e.getMessage.take(160)}") - spark.sql(s"DROP TABLE IF EXISTS $other") - }() - - // ── DDL Phase 19: namespace DDL negatives (OpenHouse rejects create/drop) ────────────────── - // Both CREATE and DROP NAMESPACE surface `UnsupportedOperationException: "Describing database is not - // supported"` — Spark calls loadNamespaceMetadata first, so the user gets a *describe* message for a - // create/drop (a misleading message — AUDIT-FINDINGS B). We anchor on the stable "not supported". - val ddlNegCreateNamespace: TableTest[CoreTable.type] = - coreNegative("ddl.ns.createRejected") { (spark, _) => - val e = Check.intercept[UnsupportedOperationException](spark.sql("CREATE NAMESPACE openhouse.a_new_db")) - assert(e.getMessage.contains("not supported"), s"msg: ${e.getMessage.take(160)}") - } - - val ddlNegDropNamespace: TableTest[CoreTable.type] = - coreNegative("ddl.ns.dropRejected") { (spark, _) => - val e = Check.intercept[UnsupportedOperationException](spark.sql("DROP NAMESPACE openhouse.dbMatrix")) - assert(e.getMessage.contains("not supported"), s"msg: ${e.getMessage.take(160)}") - } - - val ddlMiscOperations: List[(String, TableTest[CoreTable.type])] = List( - "ddl.sortOrder.orderedBy" -> ddlWriteOrderedBy, - "ddl.sortOrder.orderedByMulti" -> ddlWriteOrderedByMulti, - "ddl.renameTable" -> ddlRenameTable, - "ddl.renameTable.conflict" -> ddlRenameTableConflict, - "ddl.ns.createRejected" -> ddlNegCreateNamespace, - "ddl.ns.dropRejected" -> ddlNegDropNamespace - ) - - // ── DDL Phase 20: policy DDL (OpenHouse SQL extension: ALTER TABLE … SET/UNSET POLICY) ────── - private def policiesBlob(view: StepView[CoreTable.type]): String = - tableProps(view.spark, view.table).getOrElse("policies", "") - - val ddlPolicySharing: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("ddl.policy.sharing")(t => s"ALTER TABLE $t SET POLICY (SHARING=TRUE)") { view => - assert(policiesBlob(view).toLowerCase.contains("true") || policiesBlob(view).toLowerCase.contains("sharing"), - s"sharing policy not stored: ${policiesBlob(view)}") - assert(view.after.size == 3, "table not queryable after SET POLICY (SHARING)") // DML-after-DDL - } - - val ddlPolicyHistory: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("ddl.policy.history")(t => s"ALTER TABLE $t SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20)") { view => - assert(policiesBlob(view).contains("20") || policiesBlob(view).toLowerCase.contains("history"), - s"history policy not stored: ${policiesBlob(view)}") - assert(view.after.size == 3, "table not queryable after SET POLICY (HISTORY)") // DML-after-DDL - } - - val ddlPolicyReplicationRoundTrip: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("ddl.policy.replication.set")(t => s"ALTER TABLE $t SET POLICY (REPLICATION = ({destination:'WAR'}))")() - .sql("ddl.policy.replication.unset")(t => s"ALTER TABLE $t UNSET POLICY (REPLICATION)") { view => - assert(view.after.size == 3) // survives set+unset - } - - val ddlPolicyNegHistoryMaxAge: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("ddl.policy.neg.historyMaxAge") { (spark, table) => - val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table SET POLICY (HISTORY MAX_AGE=5D)")) // > 3 days - assert(e.getMessage.contains("max age must be between 1 to 3 days"), s"msg: ${e.getMessage.take(160)}") - }() - - val ddlPolicyNegHistoryVersions: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("ddl.policy.neg.historyVersions") { (spark, table) => - val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table SET POLICY (HISTORY VERSIONS=200)")) // > 100 - assert(e.getMessage.contains("must be between 2 to 100 versions"), s"msg: ${e.getMessage.take(160)}") - }() - - // Retention on a (string) time-partitioned column requires a column pattern (a valid DateTimeFormatter). - val ddlPolicyRetention: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource PARTITIONED BY (datepartition) TBLPROPERTIES ('write.format.default'='$seedFmt')")().insert(3)() - .sql("ddl.policy.retention")(t => s"ALTER TABLE $t SET POLICY (RETENTION = 30d ON COLUMN datepartition WHERE pattern = 'yyyy-MM-dd-HH')") { view => - assert(policiesBlob(view).toLowerCase.contains("retention") || policiesBlob(view).contains("30"), - s"retention policy not stored: ${policiesBlob(view)}") - assert(view.after.size == 3, "table not queryable after SET POLICY (RETENTION)") // DML-after-DDL - } - - val ddlPolicyOperations: List[(String, TableTest[CoreTable.type])] = List( - "ddl.policy.sharing" -> ddlPolicySharing, - "ddl.policy.history" -> ddlPolicyHistory, - "ddl.policy.replication" -> ddlPolicyReplicationRoundTrip, - "ddl.policy.retention" -> ddlPolicyRetention, - "ddl.policy.neg.historyMaxAge" -> ddlPolicyNegHistoryMaxAge, - "ddl.policy.neg.historyVersions" -> ddlPolicyNegHistoryVersions - ) - - // ── DDL Phase 18: CTAS / RTAS ─────────────────────────────────────────────────────────── - val ddlCtas: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("ddl.ctas") { (spark, table) => - val tgt = s"${table}_ctas" - spark.sql(s"DROP TABLE IF EXISTS $tgt") - spark.sql(s"CREATE TABLE $tgt USING $dataSource AS SELECT * FROM $table") - assert(spark.sql(s"SELECT count(*) FROM $tgt").collect()(0).getLong(0) == 3, "CTAS lost rows") - spark.sql(s"DROP TABLE IF EXISTS $tgt") - }() - - val ddlRtasEnabled: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("ddl.rtas.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('replace.enabled'='true')")() - .step("ddl.rtas.enabled") { (spark, table) => - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, "RTAS did not replace") - }() - - val ddlRtasDisabled: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("ddl.rtas.disabled") { (spark, table) => - val e = Check.intercept[BadRequestException](spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table")) - assert(e.getMessage.contains("REPLACE TABLE AS SELECT is not enabled"), s"msg: ${e.getMessage.take(160)}") - }() - - val ddlRtasReplicationConflict: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("ddl.rtas.repl.enable")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('replace.enabled'='true')")() - .sql("ddl.rtas.repl.policy")(t => s"ALTER TABLE $t SET POLICY (REPLICATION = ({destination:'WAR'}))")() - .step("ddl.rtas.replicationConflict") { (spark, table) => - val e = Check.intercept[BadRequestException](spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table")) - assert(e.getMessage.contains("while replication is enabled"), s"msg: ${e.getMessage.take(160)}") - }() - - val ddlCtasRtasOperations: List[(String, TableTest[CoreTable.type])] = List( - "ddl.ctas" -> ddlCtas, - "ddl.rtas.enabled" -> ddlRtasEnabled, - "ddl.rtas.disabled" -> ddlRtasDisabled, - "ddl.rtas.replicationConflict" -> ddlRtasReplicationConflict - ) - - // ── DDL Phase 22: column tags + ACL (metadata/ACL-plane; tags do NOT mask query results) ──── - val ddlColumnTag: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("ddl.colTag")(t => s"ALTER TABLE $t MODIFY COLUMN ${Core.string0.columnName} SET TAG = (PII)") { view => - val vals = view.spark.sql(s"SELECT ${Core.string0.columnName} FROM ${view.table} ORDER BY ${Core.long0.columnName}").collect().toSeq.map(_.getString(0)) - assert(vals == Seq("row-1", "row-2", "row-3"), s"SET TAG changed query results (should not mask): $vals") - } - - val ddlAclGrantUnshared: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("ddl.acl.grantUnshared") { (spark, table) => - val e = Check.intercept[IllegalArgumentException](spark.sql(s"GRANT SELECT ON TABLE $table TO PUBLIC")) - assert(e.getMessage.contains("is not a shared table"), s"msg: ${e.getMessage.take(160)}") - }() - - // After SHARING=TRUE the grant is accepted (the embedded auth handler records it, no throw). - val ddlAclGrantShared: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("ddl.acl.share")(t => s"ALTER TABLE $t SET POLICY (SHARING=TRUE)")() - .sql("ddl.acl.grantShared")(t => s"GRANT SELECT ON TABLE $t TO PUBLIC") { view => - assert(view.after.size == 3, "shared/granted table not queryable") // DML-after-DDL - } - - // ── DDL Phase 15: feature-flag property (write.distribution-mode governs the write path) ─ - val ddlFeatureDistributionMode: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt', 'write.distribution-mode'='none')")() - .insert(3)() - .check("ddl.featureFlag.distributionMode") { view => - assert(tableProps(view.spark, view.table).get("write.distribution-mode").contains("none"), - s"distribution-mode not honored: ${tableProps(view.spark, view.table).get("write.distribution-mode")}") - assert(view.after.size == 3, "table not writable under distribution-mode=none") // DML-after-DDL - } - - // ── DDL Phase 23: replication / table-type contract (SQL-reachable) ───────────────────────── - val ddlReplTableTypeImmutable: TableTest[CoreTable.type] = - coreNegative("ddl.repl.tableTypeImmutable") { (spark, table) => - val e = Check.intercept[BadRequestException](spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('openhouse.tableType'='REPLICA_TABLE')")) - assert(e.getMessage.contains("restriction"), s"msg: ${e.getMessage.take(160)}") - } - - val ddlTagAclFeatureOperations: List[(String, TableTest[CoreTable.type])] = List( - "ddl.colTag" -> ddlColumnTag, - "ddl.acl.grantUnshared" -> ddlAclGrantUnshared, - "ddl.acl.grantShared" -> ddlAclGrantShared, - "ddl.featureFlag.distributionMode" -> ddlFeatureDistributionMode, - "ddl.repl.tableTypeImmutable" -> ddlReplTableTypeImmutable - ) + val ddlMiscellaneousCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => + val format = preparation.label + + List( + preparation.test("ddl.sortOrder.orderedBy") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} WRITE ORDERED BY ${Core.long0.columnName}") + + val distributionMode = + tableProps(table.spark, table.name).get("write.distribution-mode") + + assert( + distributionMode.contains("range"), + s"distribution-mode not range: $distributionMode") + }, + preparation.test("ddl.sortOrder.orderedByMulti") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} WRITE ORDERED BY " + + s"${Core.string0.columnName} DESC NULLS FIRST, ${Core.long0.columnName}") + + assert( + tableProps(table.spark, table.name).get("write.distribution-mode").contains("range"), + "multi-col ordered-by should set range") + + table.spark.sql( + s"INSERT INTO ${table.name} ${RowGenerator.valuesClause(Core, 2)}") + + assert(table.rows.size == 5, "multi-col ordered write path failed") + }, + preparation.test("ddl.renameTable") { table => + val renamedTable = s"${table.name}_ren" + + table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $renamedTable") + assert( + table.spark.sql(s"SELECT count(*) FROM $renamedTable").collect()(0).getLong(0) == 3, + "renamed table lost rows") + Check.intercept[Exception]( + table.spark.sql(s"SELECT 1 FROM ${table.name} LIMIT 1")) + table.spark.sql(s"ALTER TABLE $renamedTable RENAME TO ${table.name}") + }, + preparation.test("ddl.renameTable.conflict") { table => + val conflictingTable = s"${table.name}_other" + + table.spark.sql(s"DROP TABLE IF EXISTS $conflictingTable") + table.spark.sql( + s"CREATE TABLE $conflictingTable ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')") + val exception = Check.intercept[WebClientResponseWithMessageException]( + table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $conflictingTable")) + + assert( + exception.getMessage.contains("already exists"), + s"msg: ${exception.getMessage.take(160)}") + table.spark.sql(s"DROP TABLE IF EXISTS $conflictingTable") + }, + preparation.test("ddl.ns.createRejected") { table => + val exception = Check.intercept[UnsupportedOperationException]( + table.spark.sql("CREATE NAMESPACE openhouse.a_new_db")) + + assert( + exception.getMessage.contains("not supported"), + s"msg: ${exception.getMessage.take(160)}") + }, + preparation.test("ddl.ns.dropRejected") { table => + val exception = Check.intercept[UnsupportedOperationException]( + table.spark.sql("DROP NAMESPACE openhouse.dbMatrix")) + + assert( + exception.getMessage.contains("not supported"), + s"msg: ${exception.getMessage.take(160)}") + }) + } + + val ddlPolicyCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => + val format = preparation.label + val retentionPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "PARTITIONED BY (datepartition) " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + + List( + preparation.test("ddl.policy.sharing") { table => + table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") + + val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + + assert( + policies.toLowerCase.contains("true") || + policies.toLowerCase.contains("sharing"), + s"sharing policy not stored: $policies") + assert( + table.rows.size == 3, + "table not queryable after SET POLICY (SHARING)") + }, + preparation.test("ddl.policy.history") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20)") + + val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + + assert( + policies.contains("20") || policies.toLowerCase.contains("history"), + s"history policy not stored: $policies") + assert( + table.rows.size == 3, + "table not queryable after SET POLICY (HISTORY)") + }, + preparation.test("ddl.policy.replication") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") + table.spark.sql( + s"ALTER TABLE ${table.name} UNSET POLICY (REPLICATION)") + + assert(table.rows.size == 3) + }, + retentionPreparation.test("ddl.policy.retention") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (" + + "RETENTION = 30d ON COLUMN datepartition WHERE pattern = 'yyyy-MM-dd-HH')") + + val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + + assert( + policies.toLowerCase.contains("retention") || policies.contains("30"), + s"retention policy not stored: $policies") + assert( + table.rows.size == 3, + "table not queryable after SET POLICY (RETENTION)") + }, + preparation.test("ddl.policy.neg.historyMaxAge") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=5D)")) + + assert( + exception.getMessage.contains("max age must be between 1 to 3 days"), + s"msg: ${exception.getMessage.take(160)}") + }, + preparation.test("ddl.policy.neg.historyVersions") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (HISTORY VERSIONS=200)")) + + assert( + exception.getMessage.contains("must be between 2 to 100 versions"), + s"msg: ${exception.getMessage.take(160)}") + }) + } + + val ddlCtasRtasCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => + List( + preparation.test("ddl.ctas") { table => + val targetTable = s"${table.name}_ctas" + + table.spark.sql(s"DROP TABLE IF EXISTS $targetTable") + table.spark.sql( + s"CREATE TABLE $targetTable USING $dataSource AS SELECT * FROM ${table.name}") + + assert( + table.spark.sql(s"SELECT count(*) FROM $targetTable").collect()(0).getLong(0) == 3, + "CTAS lost rows") + + table.spark.sql(s"DROP TABLE IF EXISTS $targetTable") + }, + preparation.test("ddl.rtas.enabled") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('replace.enabled'='true')") + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} WHERE ${Core.long0.columnName} <= 2") + + assert( + table.spark.sql(s"SELECT count(*) FROM ${table.name}").collect()(0).getLong(0) == 2, + "RTAS did not replace") + }, + preparation.test("ddl.rtas.disabled") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name}")) + + assert( + exception.getMessage.contains("REPLACE TABLE AS SELECT is not enabled"), + s"msg: ${exception.getMessage.take(160)}") + }, + preparation.test("ddl.rtas.replicationConflict") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('replace.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") + + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name}")) + + assert( + exception.getMessage.contains("while replication is enabled"), + s"msg: ${exception.getMessage.take(160)}") + }) + } + + val ddlTagAclFeatureCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => + val format = preparation.label + val distributionModePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'write.distribution-mode'='none')")() + .insert(3)()) + + List( + preparation.test("ddl.colTag") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} MODIFY COLUMN " + + s"${Core.string0.columnName} SET TAG = (PII)") + + val values = table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getString(0)) + + assert( + values == Seq("row-1", "row-2", "row-3"), + s"SET TAG changed query results (should not mask): $values") + }, + preparation.test("ddl.acl.grantUnshared") { table => + val exception = Check.intercept[IllegalArgumentException]( + table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC")) + + assert( + exception.getMessage.contains("is not a shared table"), + s"msg: ${exception.getMessage.take(160)}") + }, + preparation.test("ddl.acl.grantShared") { table => + table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") + table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC") + + assert(table.rows.size == 3, "shared/granted table not queryable") + }, + distributionModePreparation.test("ddl.featureFlag.distributionMode") { table => + val distributionMode = + tableProps(table.spark, table.name).get("write.distribution-mode") + + assert( + distributionMode.contains("none"), + s"distribution-mode not honored: $distributionMode") + assert( + table.rows.size == 3, + "table not writable under distribution-mode=none") + }, + preparation.test("ddl.repl.tableTypeImmutable") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + + "'openhouse.tableType'='REPLICA_TABLE')")) + + assert( + exception.getMessage.contains("restriction"), + s"msg: ${exception.getMessage.take(160)}") + }) + } // ── DDL Phase 24b: encryption — asserts the INTENDED behavior, tagged SKIP in OSS ───────────── // The KMS plugin is external/private (a repo-wide search finds no EncryptionManager / @@ -404,20 +469,31 @@ trait NegativeDdlScenarios extends ScenarioKit { // with encryption configured, the data file must NOT be readable as plaintext parquet. In OSS the // hook is un-wired so files are plaintext and this would fail; it is tagged in Plan.knownBugs and // reports SKIP until the private plugin is present (then unskip to validate encryption-ON). - val ddlEncryptionActive: TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='parquet', 'encryption.key-id'='k1', 'write.metadata.encryption.gcm-key-id'='k1')")() - .insert(3)() - .check("ddl.encryption.active") { view => - val filePath = view.spark.sql(s"SELECT file_path FROM ${view.table}.files LIMIT 1").collect()(0).getString(0).stripPrefix("file:") - val head = new String(java.nio.file.Files.readAllBytes(java.nio.file.Paths.get(filePath)).take(4)) - assert(head != "PAR1", s"encryption not in force — data file is plaintext parquet (magic=$head); requires the private KMS plugin") - } - - val ddlEncryptionOperations: List[(String, TableTest[CoreTable.type])] = List( - "ddl.encryption.active" -> ddlEncryptionActive - ) + val ddlEncryptionCases: List[Plan.Case] = { + val preparation = TablePreparation( + "parquet", + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + "'write.format.default'='parquet', 'encryption.key-id'='k1', " + + "'write.metadata.encryption.gcm-key-id'='k1')")() + .insert(3)()) + + List(preparation.test("ddl.encryption.active") { table => + val filePath = table.spark + .sql(s"SELECT file_path FROM ${table.name}.files LIMIT 1") + .collect()(0) + .getString(0) + .stripPrefix("file:") + val fileHeader = new String( + java.nio.file.Files.readAllBytes(java.nio.file.Paths.get(filePath)).take(4)) + + assert( + fileHeader != "PAR1", + s"encryption not in force: data file is plaintext parquet (magic=$fileHeader); " + + "requires the private KMS plugin") + }) + } // ═══ Feature-INTERACTION axis (INTERACTION-AUDIT.md) — behaviors, single layout ══════════════ // Characterization stance: rejections are PINS of current behavior (tripwires), not contracts; diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala index e41fb5bb3..8de27468f 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala @@ -21,71 +21,130 @@ trait NestedTypesScenarios extends ScenarioKit { def createAndSeedNested(layout: Layout, numberOfRows: Int): TableTest[NestedTable.type] = TableTest(NestedTable).sql("create")(layout.create)().insert(numberOfRows)() - // Read every nested column back and check the seeded values roundtrip. - val nestedRoundtrip: TableTest[NestedTable.type] = - TableTest(NestedTable).check("nested.roundtrip") { view => - val got = view.spark.sql(s"SELECT id, s.x, s.y, arr, m['k'], nested.inner.z FROM ${view.table} ORDER BY id").collect().toSeq - val actual = got.map(r => (r.getLong(0), r.getInt(1), r.getString(2), r.getSeq[Int](3), r.getInt(4), r.getInt(5))) - assert(actual == (1 to 3).map(i => (i.toLong, i, s"row-$i", Seq(i, i + 1), i, i))) - } + val nestedCases: List[Plan.Case] = + nestedLayouts + .map(layout => + TablePreparation( + layout.label, + createAndSeedNested(layout, 3))) + .flatMap { preparation => + List( + preparation.test("nested.roundtrip") { table => + val actual = table.spark + .sql( + s"SELECT id, s.x, s.y, arr, m['k'], nested.inner.z " + + s"FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(row => + ( + row.getLong(0), + row.getInt(1), + row.getString(2), + row.getSeq[Int](3), + row.getInt(4), + row.getInt(5))) + val expected = (1 to 3).map { value => + ( + value.toLong, + value, + s"row-$value", + Seq(value, value + 1), + value, + value) + } - val nestedProjectField: TableTest[NestedTable.type] = - TableTest(NestedTable).check("nested.projectField") { view => - val xs = view.spark.sql(s"SELECT s.x FROM ${view.table} ORDER BY id").collect().map(_.getInt(0)).toSeq - assert(xs == Seq(1, 2, 3)) - } + assert(actual == expected) + }, + preparation.test("nested.projectField") { table => + val actual = table.spark + .sql(s"SELECT s.x FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(_.getInt(0)) - val nestedFilterField: TableTest[NestedTable.type] = - TableTest(NestedTable).check("nested.filterNestedField") { view => - val ids = view.spark.sql(s"SELECT id FROM ${view.table} WHERE s.x = 2 ORDER BY id").collect().map(_.getLong(0)).toSeq - assert(ids == Seq(2L)) - } + assert(actual == Seq(1, 2, 3)) + }, + preparation.test("nested.filterNestedField") { table => + val actual = table.spark + .sql(s"SELECT id FROM ${table.name} WHERE s.x = 2 ORDER BY id") + .collect() + .toSeq + .map(_.getLong(0)) - // Update a nested struct field. - val nestedUpdateStructField: TableTest[NestedTable.type] = - TableTest(NestedTable).sql("nested.updateStructField")(table => s"UPDATE $table SET s.x = 99 WHERE id = 2") { view => - assert(view.spark.sql(s"SELECT s.x FROM ${view.table} WHERE id = 2").collect()(0).getInt(0) == 99) - assert(view.spark.sql(s"SELECT s.x FROM ${view.table} WHERE id = 1").collect()(0).getInt(0) == 1) - } + assert(actual == Seq(2L)) + }, + preparation.test("nested.updateStructField") { table => + table.spark.sql( + s"UPDATE ${table.name} SET s.x = 99 WHERE id = 2") - val nestedMergeInsert: TableTest[NestedTable.type] = - TableTest(NestedTable).sql("nested.mergeInsert")(table => - s"""MERGE INTO $table tgt USING ( - SELECT * FROM VALUES - (CAST(4 AS BIGINT), named_struct('x', 4, 'y', 'row-4'), array(4, 5), map('k', 4), named_struct('inner', named_struct('z', 4))) - AS v(id, s, arr, m, nested) - ) src ON tgt.id = src.id - WHEN NOT MATCHED THEN INSERT *""") { view => - val ids = view.spark.sql(s"SELECT id FROM ${view.table} ORDER BY id").collect().map(_.getLong(0)).toSeq - assert(ids == Seq(1L, 2L, 3L, 4L)) - assert(view.spark.sql(s"SELECT s.x FROM ${view.table} WHERE id = 4").collect()(0).getInt(0) == 4) - } + assert( + table.spark + .sql(s"SELECT s.x FROM ${table.name} WHERE id = 2") + .collect()(0) + .getInt(0) == 99) + assert( + table.spark + .sql(s"SELECT s.x FROM ${table.name} WHERE id = 1") + .collect()(0) + .getInt(0) == 1) + }, + preparation.test("nested.mergeInsert") { table => + table.spark.sql( + s"""MERGE INTO ${table.name} target USING ( + SELECT * FROM VALUES + ( + CAST(4 AS BIGINT), + named_struct('x', 4, 'y', 'row-4'), + array(4, 5), + map('k', 4), + named_struct('inner', named_struct('z', 4))) + AS source(id, s, arr, m, nested) + ) source ON target.id = source.id + WHEN NOT MATCHED THEN INSERT *""") - val nestedDeleteByField: TableTest[NestedTable.type] = - TableTest(NestedTable).sql("nested.deleteByNestedField")(table => s"DELETE FROM $table WHERE s.x = 2") { view => - val ids = view.spark.sql(s"SELECT id FROM ${view.table} ORDER BY id").collect().map(_.getLong(0)).toSeq - assert(ids == Seq(1L, 3L)) - } + val ids = table.spark + .sql(s"SELECT id FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(_.getLong(0)) - // Insert a row with a null struct and empty array/map. - val nestedNullValues: TableTest[NestedTable.type] = - TableTest(NestedTable).sql("nested.nullValues")(table => - s"INSERT INTO $table VALUES (CAST(4 AS BIGINT), CAST(NULL AS struct), " + - s"CAST(array() AS array), CAST(map() AS map), CAST(NULL AS struct>))") { view => - val row4 = view.spark.sql(s"SELECT id, s, arr FROM ${view.table} WHERE id = 4").collect()(0) - assert(row4.isNullAt(1)) // s is null - assert(row4.getSeq[Int](2).isEmpty) // arr is empty - } + assert(ids == Seq(1L, 2L, 3L, 4L)) + assert( + table.spark + .sql(s"SELECT s.x FROM ${table.name} WHERE id = 4") + .collect()(0) + .getInt(0) == 4) + }, + preparation.test("nested.deleteByNestedField") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE s.x = 2") - val nestedOperations: List[(String, TableTest[NestedTable.type])] = List( - "nested.roundtrip" -> nestedRoundtrip, - "nested.projectField" -> nestedProjectField, - "nested.filterNestedField" -> nestedFilterField, - "nested.updateStructField" -> nestedUpdateStructField, - "nested.mergeInsert" -> nestedMergeInsert, - "nested.deleteByNestedField" -> nestedDeleteByField, - "nested.nullValues" -> nestedNullValues - ) + val ids = table.spark + .sql(s"SELECT id FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(_.getLong(0)) + + assert(ids == Seq(1L, 3L)) + }, + preparation.test("nested.nullValues") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES (" + + "CAST(4 AS BIGINT), " + + "CAST(NULL AS struct), " + + "CAST(array() AS array), " + + "CAST(map() AS map), " + + "CAST(NULL AS struct>))") + + val insertedRow = table.spark + .sql(s"SELECT id, s, arr FROM ${table.name} WHERE id = 4") + .collect()(0) + + assert(insertedRow.isNullAt(1)) + assert(insertedRow.getSeq[Int](2).isEmpty) + }) + } // ── type-edge coverage (TypesTable) ───────────────────────────────────────────────────── val typesLayouts: List[Layout] = @@ -100,118 +159,203 @@ trait NestedTypesScenarios extends ScenarioKit { s"(CAST($id AS BIGINT), $n, $x, $dec, $str, CAST('b' AS binary), DATE '2024-01-01', " + s"TIMESTAMP '2024-01-01 00:00:00', TIMESTAMP_NTZ '2024-01-01 00:00:00')" - val typesRoundtrip: TableTest[TypesTable.type] = - TableTest(TypesTable).check("types.roundtrip") { view => - val r = view.spark.sql(s"SELECT id, n, x, dec, str FROM ${view.table} WHERE id = 1").collect()(0) - assert(r.getLong(0) == 1L && r.getInt(1) == 1 && r.getDouble(2) == 1.5) - assert(r.getDecimal(3).compareTo(new java.math.BigDecimal("1.50")) == 0) - assert(r.getString(4) == "row-1") - } + val typesCases: List[Plan.Case] = + typesLayouts + .map(layout => + TablePreparation( + layout.label, + createAndSeedTypes(layout, 3))) + .flatMap { preparation => + List( + preparation.test("types.roundtrip") { table => + val row = table.spark + .sql( + s"SELECT id, n, x, dec, str FROM ${table.name} WHERE id = 1") + .collect()(0) - val typesNulls: TableTest[TypesTable.type] = - TableTest(TypesTable).sql("types.nulls")(table => - s"INSERT INTO $table VALUES (CAST(10 AS BIGINT), NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)") { view => - val r = view.spark.sql(s"SELECT n, x, str, ts, tsntz FROM ${view.table} WHERE id = 10").collect()(0) - assert((0 to 4).forall(r.isNullAt)) - } + assert( + row.getLong(0) == 1L && + row.getInt(1) == 1 && + row.getDouble(2) == 1.5) + assert( + row.getDecimal(3).compareTo( + new java.math.BigDecimal("1.50")) == 0) + assert(row.getString(4) == "row-1") + }, + preparation.test("types.nulls") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES (" + + "CAST(10 AS BIGINT), NULL, NULL, NULL, NULL, " + + "NULL, NULL, NULL, NULL)") - val typesSpecialFloats: TableTest[TypesTable.type] = - TableTest(TypesTable).sql("types.specialFloats")(table => - s"INSERT INTO $table VALUES ${typesRow(11, "0", "double('NaN')", "CAST(0 AS decimal(10,2))", "'x'")}, " + - s"${typesRow(12, "0", "double('Infinity')", "CAST(0 AS decimal(10,2))", "'y'")}") { view => - assert(view.spark.sql(s"SELECT x FROM ${view.table} WHERE id = 11").collect()(0).getDouble(0).isNaN) - assert(view.spark.sql(s"SELECT x FROM ${view.table} WHERE id = 12").collect()(0).getDouble(0).isInfinite) - } + val row = table.spark + .sql( + s"SELECT n, x, str, ts, tsntz FROM ${table.name} WHERE id = 10") + .collect()(0) - val typesBoundaries: TableTest[TypesTable.type] = - TableTest(TypesTable).sql("types.boundaries")(table => - s"INSERT INTO $table VALUES " + - s"${typesRow(9223372036854775807L, "2147483647", "0.0", "CAST(99999999.99 AS decimal(10,2))", "'max'")}") { view => - val r = view.spark.sql(s"SELECT id, n, dec FROM ${view.table} WHERE str = 'max'").collect()(0) - assert(r.getLong(0) == Long.MaxValue && r.getInt(1) == Int.MaxValue) - assert(r.getDecimal(2).compareTo(new java.math.BigDecimal("99999999.99")) == 0) - } + assert((0 to 4).forall(row.isNullAt)) + }, + preparation.test("types.specialFloats") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + s"${typesRow(11, "0", "double('NaN')", "CAST(0 AS decimal(10,2))", "'x'")}, " + + s"${typesRow(12, "0", "double('Infinity')", "CAST(0 AS decimal(10,2))", "'y'")}") - val typesUnicodeAndEmpty: TableTest[TypesTable.type] = - TableTest(TypesTable).sql("types.unicodeAndEmpty")(table => - s"INSERT INTO $table VALUES ${typesRow(13, "0", "0.0", "CAST(0 AS decimal(10,2))", "'日本語 🎉'")}, " + - s"${typesRow(14, "0", "0.0", "CAST(0 AS decimal(10,2))", "''")}") { view => - assert(view.spark.sql(s"SELECT str FROM ${view.table} WHERE id = 13").collect()(0).getString(0) == "日本語 🎉") - assert(view.spark.sql(s"SELECT str FROM ${view.table} WHERE id = 14").collect()(0).getString(0) == "") - } + assert( + table.spark + .sql(s"SELECT x FROM ${table.name} WHERE id = 11") + .collect()(0) + .getDouble(0) + .isNaN) + assert( + table.spark + .sql(s"SELECT x FROM ${table.name} WHERE id = 12") + .collect()(0) + .getDouble(0) + .isInfinite) + }, + preparation.test("types.boundaries") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + typesRow( + Long.MaxValue, + Int.MaxValue.toString, + "0.0", + "CAST(99999999.99 AS decimal(10,2))", + "'max'")) - val typesOperations: List[(String, TableTest[TypesTable.type])] = List( - "types.roundtrip" -> typesRoundtrip, - "types.nulls" -> typesNulls, - "types.specialFloats" -> typesSpecialFloats, - "types.boundaries" -> typesBoundaries, - "types.unicodeAndEmpty" -> typesUnicodeAndEmpty - ) + val row = table.spark + .sql( + s"SELECT id, n, dec FROM ${table.name} WHERE str = 'max'") + .collect()(0) + + assert( + row.getLong(0) == Long.MaxValue && + row.getInt(1) == Int.MaxValue) + assert( + row.getDecimal(2).compareTo( + new java.math.BigDecimal("99999999.99")) == 0) + }, + preparation.test("types.unicodeAndEmpty") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + s"${typesRow(13, "0", "0.0", "CAST(0 AS decimal(10,2))", "'日本語 🎉'")}, " + + s"${typesRow(14, "0", "0.0", "CAST(0 AS decimal(10,2))", "''")}") + + assert( + table.spark + .sql(s"SELECT str FROM ${table.name} WHERE id = 13") + .collect()(0) + .getString(0) == "日本語 🎉") + assert( + table.spark + .sql(s"SELECT str FROM ${table.name} WHERE id = 14") + .collect()(0) + .getString(0) == "") + }) + } // ── partition transforms + evolution ──────────────────────────────────────────────────── // Each transform test is self-contained: create partitioned by the transform, seed, and verify // the rows roundtrip and a partition spec is registered. - def partitionTransform(transform: String): TableTest[TypesTable.type] = - TableTest(TypesTable) - .sql("create")(table => - s"CREATE TABLE $table (${TypesTable.columnDefinitions}) USING $dataSource PARTITIONED BY ($transform) " + - s"TBLPROPERTIES ('write.format.default'='$seedFmt')")() - .insert(3)() - .check("verify") { view => - assert(view.after.size == 3) - assert(view.spark.sql(s"SELECT * FROM ${view.table}.partitions").collect().nonEmpty) - } + val partitionTransformCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + val supported = List( + "partition.identity" -> "id", + "partition.bucket" -> "bucket(4, id)", + "partition.truncate" -> "truncate(2, str)", + "partition.years" -> "years(ts)", + "partition.months" -> "months(ts)", + "partition.days" -> "days(ts)", + "partition.hours" -> "hours(ts)") + .map { + case (caseName, transform) => + TablePreparation( + format, + TableTest(TypesTable) + .sql("create")(table => + s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + + s"USING $dataSource PARTITIONED BY ($transform) " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + .test(caseName) { table => + assert(table.rows.size == 3) + assert( + table.spark + .sql(s"SELECT * FROM ${table.name}.partitions") + .collect() + .nonEmpty) + } + } + val rejected = List( + ("partition.void.rejected", "void(n)", "not supported"), + ( + "partition.dateDay.rejected", + "days(dt)", + "Unsupported column")) + .map { + case (caseName, transform, expectedMessage) => + TablePreparation( + format, + TableTest(TypesTable) + .sql("create")(table => + s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + + s"USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")()) + .test(caseName) { table => + val scratchTable = table.name + "_x" + val exception = Check.intercept[RuntimeException]( + table.spark.sql( + s"CREATE TABLE $scratchTable " + + s"(${TypesTable.columnDefinitions}) " + + s"USING $dataSource PARTITIONED BY ($transform) " + + s"TBLPROPERTIES ('write.format.default'='$format')")) + + table.spark.sql(s"DROP TABLE IF EXISTS $scratchTable") + assert(exception.getMessage.contains(expectedMessage)) + } + } - // A CREATE with an unsupported partition transform is rejected. Run it on a scratch name so the - // pipeline's managed (valid) table still exists for snapshotting. - private def partitionTransformRejected(label: String, transform: String, expectMessage: String): TableTest[TypesTable.type] = - TableTest(TypesTable) - .sql("create")(table => s"CREATE TABLE $table (${TypesTable.columnDefinitions}) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt')")() - .step(label) { (spark, table) => - val scratch = table + "_x" - val error = Check.intercept[RuntimeException](spark.sql( - s"CREATE TABLE $scratch (${TypesTable.columnDefinitions}) USING $dataSource PARTITIONED BY ($transform) TBLPROPERTIES ('write.format.default'='$seedFmt')")) - spark.sql(s"DROP TABLE IF EXISTS $scratch") - assert(error.getMessage.contains(expectMessage)) - }() - - val partitionTransforms: List[(String, TableTest[TypesTable.type])] = List( - "partition.identity" -> partitionTransform("id"), - "partition.bucket" -> partitionTransform("bucket(4, id)"), - "partition.truncate" -> partitionTransform("truncate(2, str)"), - "partition.years" -> partitionTransform("years(ts)"), - "partition.months" -> partitionTransform("months(ts)"), - "partition.days" -> partitionTransform("days(ts)"), - "partition.hours" -> partitionTransform("hours(ts)"), - // OpenHouse contract: these transforms are rejected (negative tests). - "partition.void.rejected" -> partitionTransformRejected("partition.void.rejected", "void(n)", "not supported"), - "partition.dateDay.rejected" -> partitionTransformRejected("partition.dateDay.rejected", "days(dt)", "Unsupported column") - ) + supported ++ rejected + } // OpenHouse contract: partition evolution is NOT supported — ALTER … ADD/DROP PARTITION FIELD is // rejected with a 400 telling you to recreate the table. Captured as negative tests. - val partitionEvolutionAddRejected: TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt')")() - .insert(3)() - .step("partition.evolutionAdd.rejected") { (spark, table) => - val error = Check.intercept[Exception](spark.sql(s"ALTER TABLE $table ADD PARTITION FIELD datepartition")) - assert(error.getMessage.contains("Evolution of table partitioning")) - }() - - val partitionEvolutionDropRejected: TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource PARTITIONED BY (datepartition) TBLPROPERTIES ('write.format.default'='$seedFmt')")() - .insert(3)() - .step("partition.evolutionDrop.rejected") { (spark, table) => - val error = Check.intercept[Exception](spark.sql(s"ALTER TABLE $table DROP PARTITION FIELD datepartition")) - assert(error.getMessage.contains("Evolution of table partitioning")) - }() - - val partitionEvolution: List[(String, TableTest[CoreTable.type])] = List( - "partition.evolutionAdd.rejected" -> partitionEvolutionAddRejected, - "partition.evolutionDrop.rejected" -> partitionEvolutionDropRejected - ) + val partitionEvolutionCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + List( + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + .test("partition.evolutionAdd.rejected") { table => + val exception = Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} ADD PARTITION FIELD datepartition")) + + assert( + exception.getMessage.contains("Evolution of table partitioning")) + }, + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "PARTITIONED BY (datepartition) " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + .test("partition.evolutionDrop.rejected") { table => + val exception = Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} DROP PARTITION FIELD datepartition")) + + assert( + exception.getMessage.contains("Evolution of table partitioning")) + }) + } } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala index ab65751b1..4ec1e77dd 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala @@ -1,23 +1,14 @@ package harness -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -/** - * The concrete tests, all on CoreTable. An operation is a HEADLESS pipeline segment (no create); - * the run crosses every operation with every `Layout` by composing `createAndSeed(layout)` before - * it via `andThen`. Every operation asserts the DELTA against the observed pre-state (rows and/or - * commit count), never an absolute row set — so a test holds under any layout. Operation sources - * are written as EXPLICIT literals. - */ -// The tests are authored across cohesive per-domain traits (see *Scenarios.scala + ScenarioKit.scala); -// this object assembles them. Trait mixin order == original top-to-bottom source order, so val -// initialization order is preserved. `object Plan` consumes the public members declared here. -object Scenarios extends MorMaintScenarios with DmlScenarios with NestedTypesScenarios with MaintControlScenarios with ForkScenarios with BranchWapScenarios with NegativeDdlScenarios with InteractionScenarios with SurfaceScenarios with HazardReaderWriterScenarios +/** Mixes the scenario-owned case lists and shared preparation kit into one catalog source. */ +object Scenarios + extends MorMaintScenarios + with DmlScenarios + with NestedTypesScenarios + with MaintControlScenarios + with ForkScenarios + with BranchWapScenarios + with NegativeDdlScenarios + with InteractionScenarios + with SurfaceScenarios + with HazardReaderWriterScenarios diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala index 23a5924c3..879038667 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala @@ -1,19 +1,12 @@ package harness -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -/** Assembles the run: every operation x every layout, plus create.schema per layout. */ +/** Defines the ordered catalog of scenario-owned test cases. */ object Plan { final case class Case(id: String, run: Ctx => Unit) + /** The deterministic ordered case catalog. Reading it does not execute a case or start Spark. */ + def caseIds: List[String] = cases.map(_.id) + // Known PRODUCT bugs: any case whose id contains the key is reported SKIP (bug: reason) instead // of failing the suite, and is tracked in BUGS.md. This is how we "tag a failing test and filter // it": a genuine bug is tagged here, deferred for follow-up, and never plowed past silently. @@ -35,248 +28,58 @@ object Plan { def bugReason(id: String): Option[String] = knownBugs.collectFirst { case (key, reason) if id.contains(key) => s"bug: $reason" } - def cases: List[Case] = { - val dml = for { - layout <- Scenarios.layouts - (name, op) <- Scenarios.operations - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeed(layout, 3).andThen(op).run) - - val partitioned = for { - layout <- Scenarios.layouts.filter(_.label.startsWith("partitioned/")) - (name, op) <- Scenarios.partitionedOperations - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeed(layout, 3).andThen(op).run) - - // Merge-on-read: the same mutation operations, prepared on a MoR table. - val mor = for { - layout <- Scenarios.morLayouts - (name, op) <- Scenarios.mutationOperations - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeed(layout, 3).andThen(op).run) - - // MoR discriminator: prove merge-on-read wrote delete files, and copy-on-write did not. - val morVerify = Scenarios.morVerifyLayouts.map(layout => - Case(s"mor.writesDeleteFiles @ ${layout.label}", Scenarios.createAndSeedSingleFile(layout, 3).andThen(Scenarios.morWritesDeleteFiles).run)) - val cowVerify = Scenarios.cowVerifyLayouts.map(layout => - Case(s"cow.writesNoDeleteFiles @ ${layout.label}", Scenarios.createAndSeedSingleFile(layout, 3).andThen(Scenarios.cowWritesNoDeleteFiles).run)) - - // Nested / complex types, on their own schema and layouts. - val nested = for { - layout <- Scenarios.nestedLayouts - (name, op) <- Scenarios.nestedOperations - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedNested(layout, 3).andThen(op).run) - - // Type-edge coverage, on TypesTable. - val types = for { - layout <- Scenarios.typesLayouts - (name, op) <- Scenarios.typesOperations - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedTypes(layout, 3).andThen(op).run) - - // Format multiplex. Blocks whose tables are seeded via the format-aware create helpers (coreCreateParquet - // / coreCreate / propsCreate / the ddl inline creates now reading $seedFmt) run on parquet AND orc: any - // table-creating op has a real format axis, and "format-inert" is a HYPOTHESIS this harness verifies, not - // assumes. `crossFmt` sets the per-case seed format around each case (safe — cases are sequential per worker). - val dataFormats = List("parquet", "orc") - def crossFmt[S <: Schema](block: List[(String, TableTest[S])]): List[Plan.Case] = - for { f <- dataFormats; (name, t) <- block } yield Case(s"$name @ $f", ctx => Scenarios.withSeedFmt(f)(t.run(ctx))) - - // Partition transforms + evolution — multiplex (format is a hypothesis to verify, not assume). - val partitionTransforms = crossFmt(Scenarios.partitionTransforms) - val partitionEvolution = crossFmt(Scenarios.partitionEvolution) - - val timeTravel = for { f <- dataFormats; (name, t) <- Scenarios.timeTravelOps(f) } yield Case(s"$name @ $f", t.run) - val restoreRollback = for { f <- dataFormats; (name, t) <- Scenarios.restoreRollbackOps(f) } yield Case(s"$name @ $f", t.run) - val maintenance = for { f <- dataFormats; (name, t) <- Scenarios.maintenanceOps(f) } yield Case(s"$name @ $f", t.run) - val control = Scenarios.controlPlane.map { case (name, f) => Case(s"$name @ embedded", f) } - val forkColDefault = Scenarios.forkColDefaultOps.map { case (name, f) => Case(name, f) } - val forkPartitionDist = Scenarios.forkPartitionDistOps.map { case (name, f) => Case(name, f) } - val forkDeleteFileReplication = Scenarios.forkDeleteFileReplicationOps.map { case (name, f) => Case(name, f) } - val forkFileReplicationFactor = Scenarios.forkFileReplicationFactorOps.map { case (name, f) => Case(name, f) } - val forkSplitSize = Scenarios.forkSplitSizeOps.map { case (name, f) => Case(name, f) } - val forkBinPackByLength = Scenarios.forkBinPackByLengthOps.map { case (name, f) => Case(name, f) } - val forkCompactionOrder = Scenarios.forkCompactionOrderOps.map { case (name, f) => Case(name, f) } - val branching = crossFmt(Scenarios.branching) - val branchDdl = crossFmt(Scenarios.branchDdlOps) // WAP mega-axis Stage B (G8 leak, systematic) - val wapStaged = crossFmt(Scenarios.wapStagedOps) // WAP mega-axis Stage C (staged → publish) - val interactions = crossFmt(Scenarios.interactions) ++ - Scenarios.interactionCtxOps.map { case (name, f) => Case(s"$name @ embedded", f) } - val surface = crossFmt(Scenarios.surfaceOps) - val hazards = crossFmt(Scenarios.hazardOps) ++ - Scenarios.hazardCtxOps.map { case (name, f) => Case(s"$name @ embedded", f) } - val readerWriter = for { f <- dataFormats; (name, t) <- Scenarios.readerWriterOps(f) } yield Case(s"$name @ $f", t.run) - val negatives = crossFmt(Scenarios.negatives) - val ddlNegatives = crossFmt(Scenarios.ddlNegatives) - val ddlProps = crossFmt(Scenarios.ddlPropsOperations) - val ddlMisc = crossFmt(Scenarios.ddlMiscOperations) - val ddlPolicy = crossFmt(Scenarios.ddlPolicyOperations) - val ddlCtasRtas = crossFmt(Scenarios.ddlCtasRtasOperations) - val ddlTagAcl = crossFmt(Scenarios.ddlTagAclFeatureOperations) - val ddlEncryption = Scenarios.ddlEncryptionOperations.map { case (name, t) => Case(s"$name @ parquet", t.run) } - - // Phase 24 prep multipliers (full DML cross). Ordered prep × all operations; evolved prep × - // delete/update/read only (ADD COLUMN changes INSERT arity, breaking full-column inserts). - val ddlPrepOrdered = for { - layout <- Scenarios.layouts - (name, op) <- Scenarios.operations - } yield Case(s"prep.ordered:$name @ ${layout.label}", Scenarios.createAndSeedOrdered(layout, 3).andThen(op).run) - - // delete/update/read only, and excluding ops that internally INSERT a full-column row - // (delete.byNullCondition seeds a null row) — those hit the arity mismatch on the +1-column table. - val ddlPrepEvolved = for { - layout <- Scenarios.layouts - (name, op) <- Scenarios.operations.filter { case (n, _) => - (n.startsWith("delete.") || n.startsWith("update.") || n.startsWith("read.")) && !n.contains("byNullCondition") } - } yield Case(s"prep.evolved:$name @ ${layout.label}", Scenarios.createAndSeedEvolved(layout, 3).andThen(op).run) - - // T axis — the whole DML catalog routed onto a BRANCH via spark.wap.branch (SURFACE-APPRAISAL - // step 3). Format is vacuous for branches (refs never touch file encoding), so parquet only; - // both partitionings kept (partitioning changes overwrite/dynamic-overwrite semantics on the - // branch). Every op asserts its normal delta — now proving the op works branch-routed AND that - // main is untouched (isolation). ~106 cases. - // Format policy: ORC + Parquet (both), not parquet-only. Avro is intentionally NOT added to these - // ref/metadata-routed blocks (branch/undrop/DDL-consumer) — the additive ask was ORC, and the - // 3-format blocks keep Avro separately. - val branchParquetLayouts = Scenarios.layouts.filter(l => l.label.endsWith("/parquet") || l.label.endsWith("/orc")) - // WAP mega-axis Stage A — branch DML parity with the core CREATE path: all 6 layouts (incl avro) × - // operations, routed onto a branch, asserting branch delta + main isolation. - val branchWap = for { - layout <- Scenarios.layouts - (name, op) <- Scenarios.operations - } yield Case(s"branchWap:$name @ ${layout.label}", - Scenarios.createAndSeedOnBranch(layout, 3).andThen(op).andThen(Scenarios.branchMainIsolation).run) - - // Stage A — partition-only ops routed onto a branch (mirrors the core `partitioned` block). - val branchWapPartitioned = for { - layout <- Scenarios.layouts.filter(_.label.startsWith("partitioned/")) - (name, op) <- Scenarios.partitionedOperations - } yield Case(s"branchWap:$name @ ${layout.label}", - Scenarios.createAndSeedOnBranch(layout, 3).andThen(op).andThen(Scenarios.branchMainIsolation).run) - - // Branch × MoR — mutation ops routed onto a branch of a MoR table (cherry-pick rejects row-delete - // snapshots). 3-format for parity with morLayouts (Stage A). - val branchMorLayout = Scenarios.morLayouts.filter(_.label.startsWith("mor-unpartitioned/")) - val branchWapMor = for { - layout <- branchMorLayout - (name, op) <- Scenarios.mutationOperations - } yield Case(s"branchWap:$name @ ${layout.label}", - Scenarios.createAndSeedOnBranch(layout, 3).andThen(op).andThen(Scenarios.branchMainIsolation).run) - - // P axis (replace-lineage leg) — the whole DML catalog on an RTAS'd table (SURFACE-APPRAISAL - // step 2). ~106 cases. (The undrop leg is gated on the embedded-HTS restructure — see - // REST-FIDELITY-EVAL.md — so only the RTAS leg is runnable now.) - val prepRtas = for { - (label, partitionClause, fmt) <- Scenarios.rtasPrepShapes - (name, op) <- Scenarios.operations - } yield Case(s"prep.rtas:$name @ $label", Scenarios.createAndSeedRtas(partitionClause, 3, fmt).andThen(op).run) - - // RTAS full cross (Phase 28): partition-only ops on the partitioned RTAS shapes — mirrors the core - // `partitioned` block (partitionedOperations × partitioned layouts) but on a replace-lineage base. - val prepRtasPartitioned = for { - (label, partitionClause, fmt) <- Scenarios.rtasPrepShapes.filter(_._1.startsWith("partitioned/")) - (name, op) <- Scenarios.partitionedOperations - } yield Case(s"prep.rtas:$name @ $label", Scenarios.createAndSeedRtas(partitionClause, 3, fmt).andThen(op).run) - - // RTAS × MoR — mutation ops on a replace-lineage MoR table. 3-format for parity with the core MoR - // block (morLayouts = parquet/orc/avro), per the Phase-28 full cross. - val prepRtasMor = for { - fmt <- List("parquet", "orc", "avro") - (name, op) <- Scenarios.mutationOperations - } yield Case(s"prep.rtasMor:$name @ mor-unpartitioned/$fmt", - Scenarios.createAndSeedRtasMor("", 3, fmt).andThen(op).run) - - // P axis (drop→undrop leg) — the whole DML catalog on a table taken through a real HTS soft-delete - // → restore round-trip (SURFACE-APPRAISAL). Requires the embedded real HTS (HARNESS_REAL_HTS=1); - // empty otherwise. This is the surface-DOUBLING leg: every op re-verifies that the restored table - // still behaves identically, i.e. that restore's destruction set does not intersect the feature's - // state-dependency set. Undrop is metadata/ref reconstruction — file encoding is vacuous → parquet - // layouts only (as with RTAS/branch). - val undrop = - if (HtsAdmin.enabled) for { - layout <- branchParquetLayouts - (name, op) <- Scenarios.operations - } yield Case(s"undrop:$name @ ${layout.label}", - Scenarios.createAndSeedUndropped(layout, 3).andThen(op).run) - else Nil - - // Undrop admin-lifecycle block (Phase 5) — soft-delete/list/restore/purge, real HTS only. - val undropAdmin = - if (HtsAdmin.enabled) Scenarios.undropAdminOps.map { case (name, run) => Case(name, run) } - else Nil - - // Block 9 deepening: undrop 3-way compositions (branch/time-travel/schema survival), real HTS only. - val undropInteract = - if (HtsAdmin.enabled) Scenarios.undropInteractOps.map { case (name, run) => Case(name, run) } - else Nil - - // DDL × consumer battery (task #3): each state-changing DDL, then each consumer must still work. - // 4 DDL × 6 consumers × {unpartitioned, partitioned}/parquet = 48. - val ddlConsumerBattery = for { - layout <- branchParquetLayouts - (ddlName, prep) <- Scenarios.ddlPreps - (conName, con) <- Scenarios.ddlConsumers - } yield Case(s"ddlConsume:$ddlName.$conName @ ${layout.label}", prep(layout).andThen(con).run) - - // MoR reads with a live position delete (closes the scan-path gap, step 1). Read/scan ops only — - // they must apply the position delete at read time. Across formats (delete-file encoding differs). - val morReadOps = Scenarios.operations.filter { case (n, _) => n.startsWith("read.") || n == "format.materialization" } - val prepMorRead = for { - layout <- Scenarios.morVerifyLayouts // single-file-friendly MoR layouts, per format - (name, op) <- morReadOps - } yield Case(s"prep.morRead:$name @ ${layout.label}", Scenarios.createAndSeedMorDeleted(layout, 3).andThen(op).run) - - // MoR delete-file COEXISTENCE (task #5 non-vacuous core): ops on a table that already carries a - // live position delete. Format matters (delete-file encoding) → × 3 MoR formats. - val morCoexist = for { - layout <- Scenarios.morVerifyLayouts - (name, op) <- Scenarios.morCoexistOps - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedMorDeleted(layout, 3).andThen(op).run) - - // Block 8 deepening: maintenance × MoR-with-live-delete. The delete-DECODE op (rewrite_data_files - // fold) is format-relevant → × 3 MoR formats; metadata-only maintenance is format-vacuous → × 1. - val maintenanceMorFold = for { - layout <- Scenarios.morVerifyLayouts - (name, op) <- Scenarios.maintenanceMorFoldOps - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedMorDeleted(layout, 3).andThen(op).run) - val morParquetVerify = Scenarios.morVerifyLayouts.filter(l => l.label == "mor-verify/parquet" || l.label == "mor-verify/orc") - val maintenanceMorMeta = for { - layout <- morParquetVerify - (name, op) <- Scenarios.maintenanceMorMetaOps - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedMorDeleted(layout, 3).andThen(op).run) - - // Block 10 deepening: MoR delete-file modality hazards (time-travel / rollback / expire). Snapshot - // logic is format-vacuous → × 1 MoR layout. - val morHazard = for { - layout <- morParquetVerify - (name, op) <- Scenarios.morHazardOps - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedMorDeleted(layout, 3).andThen(op).run) - - // MoR × branch MERGE: position deletes carried across fast_forward / cherry_pick / REPLACE BRANCH. - // Single-file MoR seed so a branch DELETE is a real position delete; merge is format-vacuous → ×1. - val morBranchMerge = for { - layout <- morParquetVerify - (name, op) <- Scenarios.morBranchMergeOps - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeedSingleFile(layout, 3).andThen(op).run) - - // Encryption capability pin (characterization): OSS writes plaintext parquet (encryption un-wired). - val encryptionPin = List(Case("surface.pin.dataPlaintext @ parquet", Scenarios.encryptionPlaintextPin.run)) - - val creates = Scenarios.layouts.map { layout => - Case(s"create.schema @ ${layout.label}", Scenarios.createSchema(layout).run) - } - - // DDL Phase 12: schema-evolution behaviors crossed with every layout. - val ddlSchema = for { - layout <- Scenarios.layouts - (name, op) <- Scenarios.ddlSchemaOperations - } yield Case(s"$name @ ${layout.label}", Scenarios.createAndSeed(layout, 3).andThen(op).run) - - dml ++ partitioned ++ mor ++ morVerify ++ cowVerify ++ nested ++ types ++ partitionTransforms ++ - partitionEvolution ++ timeTravel ++ restoreRollback ++ negatives ++ creates ++ ddlSchema ++ - ddlNegatives ++ ddlProps ++ ddlMisc ++ ddlPolicy ++ ddlCtasRtas ++ ddlTagAcl ++ ddlEncryption ++ - maintenance ++ control ++ branching ++ interactions ++ surface ++ hazards ++ branchWap ++ - branchDdl ++ wapStaged ++ branchWapPartitioned ++ branchWapMor ++ prepRtas ++ prepRtasPartitioned ++ prepRtasMor ++ prepMorRead ++ morCoexist ++ ddlConsumerBattery ++ - readerWriter ++ ddlPrepOrdered ++ ddlPrepEvolved ++ undrop ++ undropAdmin ++ - maintenanceMorFold ++ maintenanceMorMeta ++ undropInteract ++ morHazard ++ morBranchMerge ++ - encryptionPin ++ forkColDefault ++ forkPartitionDist ++ - forkDeleteFileReplication ++ forkFileReplicationFactor ++ forkSplitSize ++ - forkBinPackByLength ++ forkCompactionOrder - } + def cases: List[Case] = + List( + Scenarios.coreDmlCases, + Scenarios.partitionedDmlCases, + Scenarios.morDmlCases, + Scenarios.deleteFileModeCases, + Scenarios.nestedCases, + Scenarios.typesCases, + Scenarios.partitionTransformCases, + Scenarios.partitionEvolutionCases, + Scenarios.timeTravelCases, + Scenarios.restoreRollbackCases, + Scenarios.negativeCases, + Scenarios.createSchemaCases, + Scenarios.ddlSchemaCases, + Scenarios.ddlNegativeCases, + Scenarios.ddlPropertyCases, + Scenarios.ddlMiscellaneousCases, + Scenarios.ddlPolicyCases, + Scenarios.ddlCtasRtasCases, + Scenarios.ddlTagAclFeatureCases, + Scenarios.ddlEncryptionCases, + Scenarios.maintenanceCases, + Scenarios.controlPlaneCases, + Scenarios.branchingCases, + Scenarios.interactionCases, + Scenarios.interactionContextCases, + Scenarios.surfaceCases, + Scenarios.hazardCases, + Scenarios.hazardContextCases, + Scenarios.branchDmlCases, + Scenarios.branchDdlCases, + Scenarios.wapStagedCases, + Scenarios.branchPartitionedDmlCases, + Scenarios.branchMorDmlCases, + Scenarios.rtasDmlCases, + Scenarios.rtasPartitionedDmlCases, + Scenarios.rtasMorDmlCases, + Scenarios.morReadDmlCases, + Scenarios.morCoexistCases, + Scenarios.ddlConsumerCases, + Scenarios.readerWriterCases, + Scenarios.orderedDmlCases, + Scenarios.evolvedDmlCases, + Scenarios.undroppedDmlCases, + Scenarios.undropAdminCases, + Scenarios.maintenanceMorFoldCases, + Scenarios.maintenanceMorMetaCases, + Scenarios.undropInteractionCases, + Scenarios.morHazardCases, + Scenarios.morBranchMergeCases, + Scenarios.encryptionPinCases, + Scenarios.forkCases + ).flatten } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala index 9e52bce4e..2e2f37bbb 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala @@ -80,6 +80,37 @@ trait ScenarioKit { def createAndSeed(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = TableTest(Core).sql("create")(layout.create)().insert(numberOfRows)() + val preparedCoreTables: List[TablePreparation[CoreTable.type]] = + layouts.map(layout => TablePreparation(layout.label, createAndSeed(layout, 3))) + + val preparedMorCoreTables: List[TablePreparation[CoreTable.type]] = + morLayouts.map(layout => TablePreparation(layout.label, createAndSeed(layout, 3))) + + val preparedOrderedCoreTables: List[TablePreparation[CoreTable.type]] = + layouts.map(layout => + TablePreparation( + layout.label, + createAndSeedOrdered(layout, 3), + "prep.ordered:")) + + val preparedEvolvedCoreTables: List[TablePreparation[CoreTable.type]] = + layouts.map(layout => + TablePreparation( + layout.label, + createAndSeedEvolved(layout, 3), + "prep.evolved:")) + + val preparedEmptyCoreTables: List[TablePreparation[CoreTable.type]] = + layouts.map(layout => + TablePreparation(layout.label, TableTest(Core).sql("create")(layout.create)())) + + val preparedCoreFormats: List[TablePreparation[CoreTable.type]] = + layouts + .filter(layout => + layout.label == "unpartitioned/parquet" || layout.label == "unpartitioned/orc") + .map(layout => + TablePreparation(layout.label.stripPrefix("unpartitioned/"), createAndSeed(layout, 3))) + // Preparation for the physical CoW/MoR discriminator: seed all rows into ONE data file. A plain // seed INSERT fans the rows across a couple of files (writer-dependent), so a strict-subset delete // can land on a whole file and be satisfied by file elimination rather than a position delete. The @@ -114,6 +145,37 @@ trait ScenarioKit { spark.conf.set("spark.wap.branch", "b") }() + private def assertBranchMainIsolation(table: PreparedTable[CoreTable.type]): Unit = { + table.spark.conf.unset("spark.wap.branch") + val mainCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + assert( + mainCount == 3, + s"branch operation leaked to main: expected 3 rows, got $mainCount") + } + + val preparedBranchCoreTables: List[TablePreparation[CoreTable.type]] = + layouts.map { layout => + TablePreparation( + layout.label, + createAndSeedOnBranch(layout, 3), + "branchWap:", + assertBranchMainIsolation) + } + + val preparedBranchMorCoreTables: List[TablePreparation[CoreTable.type]] = + morLayouts + .filter(_.label.startsWith("mor-unpartitioned/")) + .map { layout => + TablePreparation( + layout.label, + createAndSeedOnBranch(layout, 3), + "branchWap:", + assertBranchMainIsolation) + } + // RTAS prep prefix (the P axis, replace-lineage leg — SURFACE-APPRAISAL step 2): create + seed, // then CREATE OR REPLACE ... AS SELECT * re-specifying the SAME shape, so the table is // functionally identical but reached via the replace path (the path G9/G10 showed misbehaves). @@ -140,6 +202,14 @@ trait ScenarioKit { assert(deleteFiles == 1, s"MoR prep must leave a live position-delete file, got $deleteFiles") } + val preparedMorReadCoreTables: List[TablePreparation[CoreTable.type]] = + morVerifyLayouts.map { layout => + TablePreparation( + layout.label, + createAndSeedMorDeleted(layout, 3), + "prep.morRead:") + } + // Undrop prep (the P axis, drop→undrop leg — SURFACE-APPRAISAL, requires embedded real HTS). Seed a // plain table, then take it through the FULL soft-delete → restore round-trip on the real HTS, and // hand the RESTORED table to the downstream op. The point is a modality audit: every feature's state @@ -161,6 +231,18 @@ trait ScenarioKit { s"restored table must keep its $numberOfRows rows, got ${view.after.size}") } + val preparedUndroppedCoreTables: List[TablePreparation[CoreTable.type]] = + layouts + .filter(layout => + layout.label.endsWith("/parquet") || + layout.label.endsWith("/orc")) + .map { layout => + TablePreparation( + layout.label, + createAndSeedUndropped(layout, 3), + "undrop:") + } + def createAndSeedRtas(partitionClause: String, numberOfRows: Int, format: String = "parquet"): TableTest[CoreTable.type] = TableTest(Core) .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource $partitionClause " + @@ -181,7 +263,6 @@ trait ScenarioKit { // table. Non-vacuous per the appraisal — replace + MoR is a distinct combination. protected def morPropsFmt(format: String) = s"'write.format.default'='$format', 'format-version'='2', " + "'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read', 'write.merge.mode'='merge-on-read'" - protected val morProps = morPropsFmt("parquet") def createAndSeedRtasMor(partitionClause: String, numberOfRows: Int, format: String = "parquet"): TableTest[CoreTable.type] = TableTest(Core) @@ -194,6 +275,22 @@ trait ScenarioKit { // OpenHouse catalog's stale-pointer divergence (filed as a product bug). .sql("prep.rtasMor.refresh")(t => s"REFRESH TABLE $t")() + val preparedRtasCoreTables: List[TablePreparation[CoreTable.type]] = + rtasPrepShapes.map { + case (label, partitionClause, format) => + TablePreparation( + label, + createAndSeedRtas(partitionClause, 3, format), + "prep.rtas:") + } + + val preparedRtasMorCoreTables: List[TablePreparation[CoreTable.type]] = + List("parquet", "orc", "avro").map { format => + TablePreparation( + s"mor-unpartitioned/$format", + createAndSeedRtasMor("", 3, format), + "prep.rtasMor:") + } // ── hoisted shared helpers (used across domain traits) ── protected def coreTwoSnapshots(fmt: String): TableTest[CoreTable.type] = @@ -234,15 +331,8 @@ trait ScenarioKit { // before it builds `Plan.cases`. The emitted SQL is otherwise byte-identical across environments. var dataSource: String = "iceberg" - // "should be format-independent" is a hypothesis this harness must verify, not assume (see G8/G10, and - // the fork carries patched ORC paths). Only table-LESS ops (no CREATE) have no format axis. - protected val seedFmtTL = new ThreadLocal[String]() - def seedFmt: String = Option(seedFmtTL.get).getOrElse("parquet") - def withSeedFmt[A](fmt: String)(body: => A): A = { - seedFmtTL.set(fmt); try body finally seedFmtTL.remove() - } protected def coreCreateParquet(table: String): String = - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$seedFmt')" + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='parquet')" protected def undropSeed(ctx: Ctx, name: String): (String, String, String) = { val table = s"${ctx.namespace}.$name" @@ -265,10 +355,6 @@ trait ScenarioKit { protected val extraColInsert9 = "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01', 42)" protected val extraColInsert10 = "(CAST(10 AS BIGINT), 10, 'row-10', 10.5, true, '2024-01-10-01', 43)" - protected def rtasPrep: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("enableReplace")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('replace.enabled'='true')")() - protected def countOf(spark: SparkSession, sql: String): String = spark.sql(sql).collect()(0).getLong(0).toString diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala index 915425f3e..5348f8918 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala @@ -24,536 +24,1038 @@ trait SurfaceScenarios extends ScenarioKit { assert(!m.startsWith("java.lang.NullPointerException"), s"$context: bare NPE surfaced: ${m.take(160)}") } - val surfaceMsgReadabilityGuard: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.msg.readabilityGuard") { (spark, table) => + private def runConcurrently(functions: Seq[() => Unit]): Seq[Throwable] = { + val errors = new java.util.concurrent.ConcurrentLinkedQueue[Throwable]() + val threads = functions.map(function => + new Thread(() => + try function() + catch { case throwable: Throwable => errors.add(throwable) })) + threads.foreach(_.start()) + threads.foreach(_.join(180000)) + errors.toArray(Array.empty[Throwable]).toSeq + } + + private def isTypedCommitConflict(throwable: Throwable): Boolean = + Exceptions.causeChain(throwable).exists { cause => + val className = cause.getClass.getName + className.contains("CommitFailed") || + className.contains("CommitStateUnknown") || + className.contains("Validation") || + className.contains("BadRequest") || + className.contains("WebClientResponse") + } + + private def surfaceBranchCases(format: String): List[Plan.Case] = { + val basePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + val twoSnapshotPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("insertMore")(table => + s"INSERT INTO $table VALUES " + + "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + + "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) + val wapPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableWap")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")()) + + List( + twoSnapshotPreparation.test( + "surface.maint.compactWithBranch") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH cb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_cb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + val compactionResult = table.spark + .sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('min-input-files', '2'))") + .collect()(0) + + println( + "DIAG compactWithBranch: " + + s"mainCompaction rewritten=${compactionResult.get(0)} " + + s"added=${compactionResult.get(1)}") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "6", + "main compaction should preserve 6 rows") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") == "6", + "main compaction should preserve the branch") + + table.spark.conf.set("spark.wap.branch", "cb") + val branchRoutedOutcome = + try { + val result = table.spark + .sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}')") + .collect()(0) + s"RAN (rewritten=${result.get(0)}, added=${result.get(1)})" + } catch { + case exception: Throwable => + s"THREW ${exception.getClass.getSimpleName} :: " + + Option(exception.getMessage).getOrElse("").take(140) + } finally { + table.spark.conf.unset("spark.wap.branch") + } + println(s"DIAG compactUnderWapConf: $branchRoutedOutcome") + + table.spark.sql(s"REFRESH TABLE ${table.name}") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "6", + "branch-routed compaction attempt should preserve main") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") == "6", + "branch-routed compaction attempt should preserve the branch") + }, + basePreparation.test("surface.msg.readabilityGuard") { table => assertReadableMessage("dropColumn")( - Check.intercept[Exception](spark.sql(s"ALTER TABLE $table DROP COLUMN ${Core.int0.columnName}"))) + Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} " + + s"DROP COLUMN ${Core.int0.columnName}"))) assertReadableMessage("reservedProp")( - Check.intercept[Exception](spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('openhouse.tableUUID'='x')"))) + Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('openhouse.tableUUID'='x')"))) assertReadableMessage("rtasDisabled")( - Check.intercept[Exception](spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table"))) + Check.intercept[Exception]( + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name}"))) assertReadableMessage("createNamespace")( - Check.intercept[Exception](spark.sql("CREATE NAMESPACE openhouse.nope_ns"))) - }() - - // ── G8 legs: the other main-affecting DDLs leak from a branch to main ──────────────────────── - val surfaceBranchLeakSetProps: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("branch.leak.setProps") { (spark, table) => - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") - spark.sql(s"ALTER TABLE $table CREATE BRANCH lb2") - spark.conf.set("spark.wap.branch", "lb2") - try spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('user.leaked'='yes')") - finally spark.conf.unset("spark.wap.branch") - assert(tableProps(spark, table).get("user.leaked").contains("yes"), - "G8 appears FIXED for SET TBLPROPERTIES — props no longer leak from branch to main; update AUDIT-FINDINGS G8") - }() - - val surfaceBranchLeakWriteOrdered: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("branch.leak.writeOrderedBy") { (spark, table) => - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") - spark.sql(s"ALTER TABLE $table CREATE BRANCH lb3") - spark.conf.set("spark.wap.branch", "lb3") - try spark.sql(s"ALTER TABLE $table WRITE ORDERED BY ${Core.long0.columnName}") - finally spark.conf.unset("spark.wap.branch") - assert(tableProps(spark, table).get("write.distribution-mode").contains("range"), - "G8 appears FIXED for WRITE ORDERED BY — sort order no longer leaks from branch to main; update AUDIT-FINDINGS G8") - }() - - // ── G4 pin: toggling WAP off while staged snapshots exist is NOT guarded ───────────────────── - val surfaceWapToggleNoGuard: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("branch.wapToggle.noGuard") { (spark, table) => - spark.conf.set("spark.wap.id", "w9") - try spark.sql(s"INSERT INTO $table VALUES (CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - finally spark.conf.unset("spark.wap.id") - val staged = countOf(spark, s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'w9'") - assert(staged == "1", s"staging failed: $staged staged snapshots") - // G4 pin: the toggle is ACCEPTED with a staged snapshot outstanding (no guard exists). - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='false')") - val stagedAfter = countOf(spark, s"SELECT count(*) FROM $table.snapshots WHERE summary['wap.id'] = 'w9'") - println(s"DIAG wapToggle: stagedAfterToggle=$stagedAfter") - }() - - // ── WAP negatives (B2 follow-ups) ──────────────────────────────────────────────────────────── - val surfaceWapDoubleCherrypick: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("wap.neg.doubleCherrypick") { (spark, table) => - spark.conf.set("spark.wap.id", "w1") - try spark.sql(s"INSERT INTO $table VALUES (CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - finally spark.conf.unset("spark.wap.id") - val sid = spark.sql(s"SELECT snapshot_id FROM $table.snapshots WHERE summary['wap.id'] = 'w1'").collect()(0).getLong(0) - spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${sid}L)") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "4", "first publish failed") - val e = Check.intercept[Exception]( - spark.sql(s"CALL openhouse.system.cherrypick_snapshot('${catalogRelative(table)}', ${sid}L)")) - println(s"DIAG doubleCherrypick: ${e.getClass.getName} :: ${Option(e.getMessage).getOrElse("").take(180)}") - assert(Option(e.getMessage).exists(m => m.toLowerCase.contains("duplicate") || m.toLowerCase.contains("already")), - s"double cherry-pick should be rejected as a duplicate WAP commit: ${e.getMessage.take(180)}") - }() - - val surfaceWapExpireRefTarget: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("wap.neg.expireRefTarget") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH eb2") - val headId = spark.sql(s"SELECT snapshot_id FROM $table.refs WHERE name = 'eb2'").collect()(0).getLong(0) - val e = Check.intercept[Exception](spark.sql( - s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', snapshot_ids => ARRAY(${headId}L))")) - println(s"DIAG expireRefTarget: ${e.getClass.getName} :: ${Option(e.getMessage).getOrElse("").take(180)}") - }() - - // ── Branch lifecycle tail: fast_forward IS the merge; replace branch ──────────────────────── - val surfaceBranchFastForwardMerge: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("branch.fastForward.merge") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH fb") - spark.sql(s"INSERT INTO $table.branch_fb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - spark.sql(s"INSERT INTO $table.branch_fb VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "main advanced unexpectedly") - spark.sql(s"CALL openhouse.system.fast_forward('${catalogRelative(table)}', 'main', 'fb')") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "5", - "fast_forward must merge the branch into main (main == branch head)") - }() - - val surfaceBranchFastForwardDivergent: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("branch.fastForward.divergent") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH db") - spark.sql(s"INSERT INTO $table.branch_db VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - spark.sql(s"INSERT INTO $table VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") // diverge main - val e = Check.intercept[Exception]( - spark.sql(s"CALL openhouse.system.fast_forward('${catalogRelative(table)}', 'main', 'db')")) - println(s"DIAG ffDivergent: ${e.getClass.getName} :: ${Option(e.getMessage).getOrElse("").take(180)}") - assert(Option(e.getMessage).exists(m => m.toLowerCase.contains("ancestor") || m.toLowerCase.contains("fast-forward")), - s"divergent fast_forward should be rejected with an ancestry error: ${e.getMessage.take(180)}") - }() - - val surfaceBranchReplaceBranch: TableTest[CoreTable.type] = - coreTwoSnapshots.step("branch.replaceBranch") { (spark, table) => - val snaps = snapshotIds(spark, table) - spark.sql(s"ALTER TABLE $table CREATE BRANCH rb2") - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'rb2'") == "5", "branch at head") - spark.sql(s"ALTER TABLE $table REPLACE BRANCH rb2 AS OF VERSION ${snaps.head}") - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'rb2'") == "3", - "REPLACE BRANCH must retarget the ref to the older snapshot") - }() - - // ── Streaming (structured streaming read + write) ──────────────────────────────────────────── - val surfaceStreamRead: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.stream.read") { (spark, table) => - val ckpt = java.nio.file.Files.createTempDirectory("ck-read").toString + Check.intercept[Exception]( + table.spark.sql("CREATE NAMESPACE openhouse.nope_ns"))) + }, + basePreparation.test("branch.leak.setProps") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH lb2") + table.spark.conf.set("spark.wap.branch", "lb2") + try { + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('user.leaked'='yes')") + } finally { + table.spark.conf.unset("spark.wap.branch") + } + + assert( + tableProps(table.spark, table.name) + .get("user.leaked") + .contains("yes"), + "branch-routed property update should change table-global metadata") + }, + basePreparation.test("branch.leak.writeOrderedBy") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH lb3") + table.spark.conf.set("spark.wap.branch", "lb3") + try { + table.spark.sql( + s"ALTER TABLE ${table.name} " + + s"WRITE ORDERED BY ${Core.long0.columnName}") + } finally { + table.spark.conf.unset("spark.wap.branch") + } + + assert( + tableProps(table.spark, table.name) + .get("write.distribution-mode") + .contains("range"), + "branch-routed ordering should change table-global metadata") + }, + wapPreparation.test("branch.wapToggle.noGuard") { table => + table.spark.conf.set("spark.wap.id", "w9") + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + } finally { + table.spark.conf.unset("spark.wap.id") + } + val stagedSnapshotCount = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w9'") + assert( + stagedSnapshotCount == "1", + s"expected one staged snapshot, got $stagedSnapshotCount") + + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='false')") + val stagedAfterToggle = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w9'") + + println(s"DIAG wapToggle: stagedAfterToggle=$stagedAfterToggle") + }, + wapPreparation.test("wap.neg.doubleCherrypick") { table => + table.spark.conf.set("spark.wap.id", "w1") + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + } finally { + table.spark.conf.unset("spark.wap.id") + } + val stagedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w1'") + .collect()(0) + .getLong(0) + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', ${stagedSnapshotId}L)") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "4", + "first cherry-pick should publish the staged row") + + val exception = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', ${stagedSnapshotId}L)")) + println( + "DIAG doubleCherrypick: " + + s"${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(180)) + assert( + Option(exception.getMessage).exists(message => + message.toLowerCase.contains("duplicate") || + message.toLowerCase.contains("already")), + "second cherry-pick should reject the duplicate WAP commit") + }, + basePreparation.test("wap.neg.expireRefTarget") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH eb2") + val branchHeadSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.refs " + + "WHERE name = 'eb2'") + .collect()(0) + .getLong(0) + val exception = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + s"snapshot_ids => ARRAY(${branchHeadSnapshotId}L))")) + + println( + "DIAG expireRefTarget: " + + s"${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(180)) + }, + basePreparation.test("branch.fastForward.merge") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH fb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_fb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_fb VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "branch writes should not advance main") + + table.spark.sql( + "CALL openhouse.system.fast_forward(" + + s"'${catalogRelative(table.name)}', 'main', 'fb')") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "5", + "fast_forward should move main to the branch head") + }, + basePreparation.test("branch.fastForward.divergent") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH db") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_db VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + val exception = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.fast_forward(" + + s"'${catalogRelative(table.name)}', 'main', 'db')")) + + println( + "DIAG ffDivergent: " + + s"${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(180)) + assert( + Option(exception.getMessage).exists(message => + message.toLowerCase.contains("ancestor") || + message.toLowerCase.contains("fast-forward")), + "divergent fast_forward should report an ancestry error") + }, + twoSnapshotPreparation.test("branch.replaceBranch") { table => + val snapshots = snapshotIds(table.spark, table.name) + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH rb2") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb2'") == "5", + "new branch should point at the current head") + + table.spark.sql( + s"ALTER TABLE ${table.name} REPLACE BRANCH rb2 " + + s"AS OF VERSION ${snapshots.head}") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb2'") == "3", + "REPLACE BRANCH should retarget the branch to the older snapshot") + }) + } + + private def surfaceReaderProcedureCases( + format: String): List[Plan.Case] = { + val basePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + val twoSnapshotPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("insertMore")(table => + s"INSERT INTO $table VALUES " + + "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + + "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) + val emptyPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")()) + val morPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', " + + "'write.delete.mode'='merge-on-read')")() + .sql("seed")(table => + s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM " + + s"(${RowGenerator.valuesClause(Core, 3)}) AS seed")()) + val wapPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableWap")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")()) + + List( + basePreparation.test("surface.stream.read") { table => + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-read").toString val sink = s"memsink_${System.nanoTime}" - val q = spark.readStream.table(table) - .writeStream.format("memory").queryName(sink) + val query = table.spark.readStream + .table(table.name) + .writeStream + .format("memory") + .queryName(sink) .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", ckpt) + .option("checkpointLocation", checkpoint) .start() - assert(q.awaitTermination(120000), "streaming read did not finish in 120s") - assert(countOf(spark, s"SELECT count(*) FROM $sink") == "3", - "streaming read must deliver the seeded rows") - }() - - val surfaceStreamWrite: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.stream.write") { (spark, table) => - import spark.implicits._ - implicit val sqlc: org.apache.spark.sql.SQLContext = spark.sqlContext - val ms = org.apache.spark.sql.execution.streaming.MemoryStream[Long] - ms.addData(100L, 101L) - val df = ms.toDF().selectExpr( + + assert( + query.awaitTermination(120000), + "streaming read did not finish in 120 seconds") + assert( + countOf(table.spark, s"SELECT count(*) FROM $sink") == "3", + "streaming read should deliver the three seed rows") + }, + basePreparation.test("surface.stream.write") { table => + import table.spark.implicits._ + implicit val sqlContext: org.apache.spark.sql.SQLContext = + table.spark.sqlContext + val memoryStream = + org.apache.spark.sql.execution.streaming.MemoryStream[Long] + memoryStream.addData(100L, 101L) + val rows = memoryStream.toDF().selectExpr( s"value AS ${Core.long0.columnName}", s"CAST(value AS INT) AS ${Core.int0.columnName}", s"concat('row-', value) AS ${Core.string0.columnName}", s"CAST(value AS DOUBLE) AS ${Core.double0.columnName}", s"true AS ${Core.boolean0.columnName}", s"'2024-01-01-00' AS ${Core.datePartition.columnName}") - val ckpt = java.nio.file.Files.createTempDirectory("ck-write").toString - val q = df.writeStream.format("iceberg").outputMode("append") - .option("checkpointLocation", ckpt) - .toTable(table) - q.processAllAvailable() - q.stop() - assert(countOf(spark, s"SELECT count(*) FROM $table") == "5", - "streaming write must append the 2 streamed rows") - }() - - // ── CDC: changelog view procedure ───────────────────────────────────────────────────────────── - val surfaceCdcChangelogView: TableTest[CoreTable.type] = - coreTwoSnapshots.step("surface.cdc.changelogView") { (spark, table) => - val viewName = spark.sql( - s"CALL openhouse.system.create_changelog_view(table => '${catalogRelative(table)}')").collect()(0).getString(0) - val changes = spark.sql(s"SELECT count(*) FROM $viewName").collect()(0).getLong(0) - assert(changes == 5, s"changelog must contain one INSERT change per seeded row: $changes") - val types = spark.sql(s"SELECT DISTINCT _change_type FROM $viewName").collect().toSeq.map(_.getString(0)).toSet - assert(types == Set("INSERT"), s"append-only history must yield INSERT changes only: $types") - }() - - // ── Procedures not yet exercised ───────────────────────────────────────────────────────────── - // Manifest compaction must actually DO ITS JOB — reduce the manifest count — not merely preserve data. - // Five separate appends produce ~5 manifests (one per commit); rewrite_manifests must coalesce them. - val surfaceProcRewriteManifests: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)() - .step("surface.proc.rewriteManifests") { (spark, table) => - (1 to 5).foreach(i => spark.sql(s"INSERT INTO $table VALUES ${coreRow(i, s"r$i")}")) - val before = spark.sql(s"SELECT count(*) FROM $table.manifests").collect()(0).getLong(0) - spark.sql(s"CALL openhouse.system.rewrite_manifests(table => '${catalogRelative(table)}', use_caching => false)") - val after = spark.sql(s"SELECT count(*) FROM $table.manifests").collect()(0).getLong(0) - println(s"DIAG surface.proc.rewriteManifests: manifests before=$before after=$after") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "5", "rewrite_manifests changed the live row set") - assert(before >= 2 && after < before, - s"rewrite_manifests did not COMPACT the manifests (before=$before after=$after) — it should coalesce them") - }() - - val surfaceProcRewritePositionDeletes: TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$seedFmt', 'write.delete.mode'='merge-on-read')")() - .sql("seed(3, one-file)")(t => - s"INSERT INTO $t SELECT /*+ COALESCE(1) */ * FROM (${RowGenerator.valuesClause(Core, 3)}) AS seed")() - .step("surface.proc.rewritePositionDeletes") { (spark, table) => - spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") - assert(countOf(spark, s"SELECT count(*) FROM $table.all_delete_files") == "1", "MoR delete file missing") - spark.sql(s"CALL openhouse.system.rewrite_position_delete_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "2", "rewrite_position_delete_files changed data") - }() - - val surfaceProcPublishChanges: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .sql("enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("surface.proc.publishChanges") { (spark, table) => - spark.conf.set("spark.wap.id", "pw1") - try spark.sql(s"INSERT INTO $table VALUES (CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - finally spark.conf.unset("spark.wap.id") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "staged write must not be visible") - spark.sql(s"CALL openhouse.system.publish_changes(table => '${catalogRelative(table)}', wap_id => 'pw1')") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "4", - "publish_changes (the wap_id publish path beside cherrypick) must publish the staged write") - }() - - val surfaceProcAncestorsOf: TableTest[CoreTable.type] = - coreTwoSnapshots.step("surface.proc.ancestorsOf") { (spark, table) => - val n = spark.sql(s"CALL openhouse.system.ancestors_of(table => '${catalogRelative(table)}')").collect().length - assert(n == 2, s"ancestors_of must list main's full ancestry (2 snapshots): $n") - }() - - val surfaceProcRemoveOrphanReal: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.proc.removeOrphanReal") { (spark, table) => - val dataFile = spark.sql(s"SELECT file_path FROM $table.files LIMIT 1").collect()(0).getString(0).stripPrefix("file:") - val orphan = java.nio.file.Paths.get(dataFile).getParent.resolve("zz_orphan_plant.parquet") - java.nio.file.Files.write(orphan, "not-a-real-parquet".getBytes) - java.nio.file.Files.setLastModifiedTime(orphan, - java.nio.file.attribute.FileTime.fromMillis(1546300800000L)) // 2019-01-01 - spark.sql(s"CALL openhouse.system.remove_orphan_files(table => '${catalogRelative(table)}', older_than => TIMESTAMP '2020-01-01 00:00:00')") - assert(java.nio.file.Files.notExists(orphan), "planted orphan file must be removed") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "live data must survive orphan removal") - }() - - // ── Metadata surface: hidden columns + full metadata-table sweep ───────────────────────────── - val surfaceMetaHiddenColumns: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.meta.hiddenColumns") { (spark, table) => - val rows = spark.sql(s"SELECT _file, _pos, _spec_id, _partition FROM $table").collect().toSeq - assert(rows.size == 3, s"hidden metadata columns must be selectable per row: ${rows.size}") - assert(rows.forall(r => r.getString(0) != null && r.getString(0).nonEmpty), "_file must be populated") - assert(rows.forall(r => r.getLong(1) >= 0), "_pos must be populated") - }() - - val surfaceMetaTableSweep: TableTest[CoreTable.type] = - coreTwoSnapshots.step("surface.meta.tableSweep") { (spark, table) => - val metaTables = Seq("entries", "files", "manifests", "snapshots", "history", "refs", "partitions", - "metadata_log_entries", "data_files", "all_data_files", "all_manifests", "all_entries", "all_files") - metaTables.foreach { m => - val n = spark.sql(s"SELECT count(*) FROM $table.`$m`").collect()(0).getLong(0) - assert(n >= 0, s"metadata table $m unreadable") // queryability is the assertion; count is a bonus - } - assert(countOf(spark, s"SELECT count(*) FROM $table.snapshots") == "2", "snapshots count sanity") - }() - - val surfaceMetaPositionDeletes: TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$seedFmt', 'write.delete.mode'='merge-on-read')")() - .sql("seed(3, one-file)")(t => - s"INSERT INTO $t SELECT /*+ COALESCE(1) */ * FROM (${RowGenerator.valuesClause(Core, 3)}) AS seed")() - .step("surface.meta.positionDeletes") { (spark, table) => - spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") - assert(countOf(spark, s"SELECT count(*) FROM $table.position_deletes") == "1", - "position_deletes metadata table must expose the position delete") - }() - - // ── Concurrency: invariant-based (no torn state; failures must be typed) ───────────────────── - private def runConcurrently(fs: Seq[() => Unit]): Seq[Throwable] = { - val errors = new java.util.concurrent.ConcurrentLinkedQueue[Throwable]() - val threads = fs.map(f => new Thread(() => try f() catch { case t: Throwable => errors.add(t) })) - threads.foreach(_.start()) - threads.foreach(_.join(180000)) - errors.toArray(Array.empty[Throwable]).toSeq + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-write").toString + val query = rows.writeStream + .format("iceberg") + .outputMode("append") + .option("checkpointLocation", checkpoint) + .toTable(table.name) + + query.processAllAvailable() + query.stop() + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "5", + "streaming write should append two rows") + }, + twoSnapshotPreparation.test("surface.cdc.changelogView") { table => + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}')") + .collect()(0) + .getString(0) + val changeCount = table.spark + .sql(s"SELECT count(*) FROM $view") + .collect()(0) + .getLong(0) + val changeTypes = table.spark + .sql(s"SELECT DISTINCT _change_type FROM $view") + .collect() + .map(_.getString(0)) + .toSet + + assert( + changeCount == 5, + s"append-only changelog should contain 5 changes, got $changeCount") + assert( + changeTypes == Set("INSERT"), + s"append-only changelog should contain only INSERT: $changeTypes") + }, + emptyPreparation.test("surface.proc.rewriteManifests") { table => + (1 to 5).foreach(index => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + coreRow(index, s"r$index"))) + val manifestCountBefore = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.manifests") + .collect()(0) + .getLong(0) + table.spark.sql( + "CALL openhouse.system.rewrite_manifests(" + + s"table => '${catalogRelative(table.name)}', " + + "use_caching => false)") + val manifestCountAfter = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.manifests") + .collect()(0) + .getLong(0) + + println( + "DIAG surface.proc.rewriteManifests: " + + s"manifests before=$manifestCountBefore after=$manifestCountAfter") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "5", + "rewrite_manifests should preserve the five rows") + assert( + manifestCountBefore >= 2 && + manifestCountAfter < manifestCountBefore, + "rewrite_manifests should compact the manifest set") + }, + morPreparation.test( + "surface.proc.rewritePositionDeletes") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.all_delete_files") == "1", + "MoR delete should create one position-delete file") + + table.spark.sql( + "CALL openhouse.system.rewrite_position_delete_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "2", + "rewrite_position_delete_files should preserve live rows") + }, + wapPreparation.test("surface.proc.publishChanges") { table => + table.spark.conf.set("spark.wap.id", "pw1") + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + } finally { + table.spark.conf.unset("spark.wap.id") + } + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "staged write should not be visible before publish") + + table.spark.sql( + "CALL openhouse.system.publish_changes(" + + s"table => '${catalogRelative(table.name)}', wap_id => 'pw1')") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "4", + "publish_changes should publish the staged row") + }, + twoSnapshotPreparation.test("surface.proc.ancestorsOf") { table => + val ancestorCount = table.spark + .sql( + "CALL openhouse.system.ancestors_of(" + + s"table => '${catalogRelative(table.name)}')") + .collect() + .length + + assert( + ancestorCount == 2, + s"ancestors_of should list two snapshots, got $ancestorCount") + }, + basePreparation.test("surface.proc.removeOrphanReal") { table => + val dataFile = table.spark + .sql(s"SELECT file_path FROM ${table.name}.files LIMIT 1") + .collect()(0) + .getString(0) + .stripPrefix("file:") + val orphanFile = java.nio.file.Paths + .get(dataFile) + .getParent + .resolve("zz_orphan_plant.parquet") + java.nio.file.Files.write( + orphanFile, + "not-a-real-parquet".getBytes) + java.nio.file.Files.setLastModifiedTime( + orphanFile, + java.nio.file.attribute.FileTime.fromMillis(1546300800000L)) + + table.spark.sql( + "CALL openhouse.system.remove_orphan_files(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2020-01-01 00:00:00')") + assert( + java.nio.file.Files.notExists(orphanFile), + "remove_orphan_files should delete the planted orphan") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "remove_orphan_files should preserve live data") + }, + basePreparation.test("surface.meta.hiddenColumns") { table => + val rows = table.spark + .sql( + s"SELECT _file, _pos, _spec_id, _partition FROM ${table.name}") + .collect() + .toSeq + + assert( + rows.size == 3, + s"hidden metadata columns should return 3 rows, got ${rows.size}") + assert( + rows.forall(row => + Option(row.getString(0)).exists(_.nonEmpty)), + "_file should be populated for every row") + assert( + rows.forall(_.getLong(1) >= 0), + "_pos should be non-negative for every row") + }, + twoSnapshotPreparation.test("surface.meta.tableSweep") { table => + val metadataTables = Seq( + "entries", + "files", + "manifests", + "snapshots", + "history", + "refs", + "partitions", + "metadata_log_entries", + "data_files", + "all_data_files", + "all_manifests", + "all_entries", + "all_files") + metadataTables.foreach { metadataTable => + val rowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.`$metadataTable`") + .collect()(0) + .getLong(0) + assert( + rowCount >= 0, + s"metadata table $metadataTable should be queryable") + } + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots") == "2", + "snapshot metadata should contain two snapshots") + }, + morPreparation.test("surface.meta.positionDeletes") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.position_deletes") == "1", + "position_deletes should expose the MoR position delete") + }) } - private def isTypedCommitConflict(t: Throwable): Boolean = - Exceptions.causeChain(t).exists { c => - val n = c.getClass.getName - n.contains("CommitFailed") || n.contains("CommitStateUnknown") || n.contains("Validation") || - n.contains("BadRequest") || n.contains("WebClientResponse") - } + private def surfaceRemainingCases(format: String): List[Plan.Case] = { + val basePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + val replacePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableReplace")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")()) + val hashPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"PARTITIONED BY (${Core.datePartition.columnName}) " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', " + + "'write.distribution-mode'='hash')")() + .insert(3)()) + val targetSizePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', " + + "'write.target-file-size-bytes'='1048576')")() + .insert(3)()) - val surfaceConcAppendAppend: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.conc.appendAppend") { (spark, table) => - val failures = new java.util.concurrent.atomic.AtomicInteger(0) - def writer(base: Int): () => Unit = () => (0 until 3).foreach { i => - try spark.sql(s"INSERT INTO $table VALUES (CAST(${base + i} AS BIGINT), ${base + i}, 'row-c', 1.5, true, '2024-01-09-01')") - catch { case t: Throwable => - assert(isTypedCommitConflict(t), s"concurrent append failed with an UNTYPED error: ${t.getClass.getName} ${Option(t.getMessage).getOrElse("").take(160)}") - failures.incrementAndGet() + List( + basePreparation.test("surface.conc.appendAppend") { table => + val failureCount = + new java.util.concurrent.atomic.AtomicInteger(0) + def writer(base: Int): () => Unit = () => + (0 until 3).foreach { offset => + val value = base + offset + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + s"(CAST($value AS BIGINT), $value, 'row-c', 1.5, " + + "true, '2024-01-09-01')") + } catch { + case exception: Throwable => + assert( + isTypedCommitConflict(exception), + "concurrent append failed with an untyped error: " + + s"${exception.getClass.getName}") + failureCount.incrementAndGet() + } } - } - val errs = runConcurrently(Seq(writer(100), writer(200))) - assert(errs.isEmpty, s"writer thread died outside the insert loop: ${errs.headOption.map(_.toString)}") - val expected = 3 + 6 - failures.get - assert(countOf(spark, s"SELECT count(*) FROM $table") == expected.toString, - s"row count must equal successful appends (3 seed + ${6 - failures.get} landed)") - println(s"DIAG conc.appendAppend: ${failures.get}/6 inserts hit a typed commit conflict") - }() - - val surfaceConcUpdateUpdate: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.conc.updateUpdate") { (spark, table) => - val col = Core.string0.columnName - def updater(v: String): () => Unit = () => - try spark.sql(s"UPDATE $table SET $col = '$v' WHERE ${Core.long0.columnName} = 2") - catch { case t: Throwable => - assert(isTypedCommitConflict(t), s"concurrent update failed with an UNTYPED error: ${t.getClass.getName} ${Option(t.getMessage).getOrElse("").take(160)}") } - val errs = runConcurrently(Seq(updater("AAA"), updater("BBB"))) - assert(errs.isEmpty, s"updater thread died with a non-conflict error: ${errs.headOption.map(_.toString)}") - val v = spark.sql(s"SELECT $col FROM $table WHERE ${Core.long0.columnName} = 2").collect()(0).getString(0) - assert(v == "AAA" || v == "BBB" || v == "row-2", s"row must hold one writer's value or the original, not torn state: $v") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "row count must be unchanged") - }() - - val surfaceConcRtasVsAppend: TableTest[CoreTable.type] = - rtasPrep.step("surface.conc.rtasVsAppend") { (spark, table) => - def rtas(): Unit = - try spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - catch { case t: Throwable => assert(isTypedCommitConflict(t), s"RTAS race failed UNTYPED: ${t.getClass.getName}") } - def append(): Unit = - try spark.sql(s"INSERT INTO $table VALUES (CAST(30 AS BIGINT), 30, 'row-30', 30.5, true, '2024-01-09-01')") - catch { case t: Throwable => assert(isTypedCommitConflict(t), s"append race failed UNTYPED: ${t.getClass.getName}") } - val errs = runConcurrently(Seq(() => rtas(), () => append())) - assert(errs.isEmpty, s"racing thread died with a non-conflict error: ${errs.headOption.map(_.toString)}") - spark.sql(s"REFRESH TABLE $table") - val n = countOf(spark, s"SELECT count(*) FROM $table").toLong - assert(n == 2 || n == 3, s"RTAS-vs-append must settle to a consistent state (2 or 3 rows), got $n") - println(s"DIAG conc.rtasVsAppend: settled at $n rows") - }() - - // ── Schema-evolution edges ─────────────────────────────────────────────────────────────────── - val surfaceSchemaRelaxNotNull: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.schema.relaxNotNull") { (spark, table) => - val side = s"${table}_nn" - spark.sql(s"DROP TABLE IF EXISTS $side") + val threadErrors = + runConcurrently(Seq(writer(100), writer(200))) + val expectedRowCount = 3 + 6 - failureCount.get + val actualRowCount = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") + + assert( + threadErrors.isEmpty, + s"writer thread failed outside the insert loop: $threadErrors") + assert( + actualRowCount == expectedRowCount.toString, + s"expected $expectedRowCount rows, got $actualRowCount") + println( + s"DIAG conc.appendAppend: ${failureCount.get}/6 inserts " + + "hit a typed commit conflict") + }, + basePreparation.test("surface.conc.updateUpdate") { table => + val column = Core.string0.columnName + def updater(value: String): () => Unit = () => + try { + table.spark.sql( + s"UPDATE ${table.name} SET $column = '$value' " + + s"WHERE ${Core.long0.columnName} = 2") + } catch { + case exception: Throwable => + assert( + isTypedCommitConflict(exception), + "concurrent update failed with an untyped error: " + + s"${exception.getClass.getName}") + } + val threadErrors = + runConcurrently(Seq(updater("AAA"), updater("BBB"))) + val finalValue = table.spark + .sql( + s"SELECT $column FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 2") + .collect()(0) + .getString(0) + + assert( + threadErrors.isEmpty, + s"updater thread failed with a non-conflict error: $threadErrors") + assert( + finalValue == "AAA" || + finalValue == "BBB" || + finalValue == "row-2", + s"concurrent updates produced a torn value: $finalValue") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "concurrent updates should not change row count") + }, + replacePreparation.test("surface.conc.rtasVsAppend") { table => + def replaceTable(): Unit = + try { + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + } catch { + case exception: Throwable => + assert( + isTypedCommitConflict(exception), + s"RTAS race failed with ${exception.getClass.getName}") + } + def appendRow(): Unit = + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(30 AS BIGINT), 30, 'row-30', 30.5, " + + "true, '2024-01-09-01')") + } catch { + case exception: Throwable => + assert( + isTypedCommitConflict(exception), + s"append race failed with ${exception.getClass.getName}") + } + val threadErrors = + runConcurrently(Seq(() => replaceTable(), () => appendRow())) + + assert( + threadErrors.isEmpty, + s"racing thread failed with a non-conflict error: $threadErrors") + table.spark.sql(s"REFRESH TABLE ${table.name}") + val rowCount = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}").toLong + assert( + rowCount == 2 || rowCount == 3, + s"RTAS and append race settled at $rowCount rows") + println(s"DIAG conc.rtasVsAppend: settled at $rowCount rows") + }, + basePreparation.test("surface.schema.relaxNotNull") { table => + val sideTable = s"${table.name}_nn" + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") try { - spark.sql(s"CREATE TABLE $side (id BIGINT, req INT NOT NULL) USING $dataSource") - spark.sql(s"ALTER TABLE $side ALTER COLUMN req DROP NOT NULL") - spark.sql(s"INSERT INTO $side VALUES (CAST(1 AS BIGINT), NULL)") - assert(spark.sql(s"SELECT count(*) FROM $side WHERE req IS NULL").collect()(0).getLong(0) == 1, - "relaxing NOT NULL must allow null writes (the inverse of the pinned-rejected tighten)") - } finally spark.sql(s"DROP TABLE IF EXISTS $side") - }() - - val surfaceSchemaDecimalWiden: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.schema.decimalWiden") { (spark, table) => - val side = s"${table}_dec" - spark.sql(s"DROP TABLE IF EXISTS $side") + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, req INT NOT NULL) USING $dataSource") + table.spark.sql( + s"ALTER TABLE $sideTable ALTER COLUMN req DROP NOT NULL") + table.spark.sql( + s"INSERT INTO $sideTable VALUES (CAST(1 AS BIGINT), NULL)") + assert( + table.spark + .sql(s"SELECT count(*) FROM $sideTable WHERE req IS NULL") + .collect()(0) + .getLong(0) == 1, + "relaxing NOT NULL should allow a null write") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + } + }, + basePreparation.test("surface.schema.decimalWiden") { table => + val sideTable = s"${table.name}_dec" + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") try { - spark.sql(s"CREATE TABLE $side (id BIGINT, dec DECIMAL(10,2)) USING $dataSource") - spark.sql(s"INSERT INTO $side VALUES (CAST(1 AS BIGINT), CAST(12345678.99 AS DECIMAL(10,2)))") - spark.sql(s"ALTER TABLE $side ALTER COLUMN dec TYPE DECIMAL(12,2)") - spark.sql(s"INSERT INTO $side VALUES (CAST(2 AS BIGINT), CAST(1234567890.99 AS DECIMAL(12,2)))") - assert(spark.sql(s"SELECT count(*) FROM $side").collect()(0).getLong(0) == 2, - "decimal precision widen must keep old data readable and accept wider values") - } finally spark.sql(s"DROP TABLE IF EXISTS $side") - }() - - val surfaceSchemaNestedAddField: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.schema.nestedAddField") { (spark, table) => - val side = s"${table}_nst" - spark.sql(s"DROP TABLE IF EXISTS $side") + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, dec DECIMAL(10,2)) USING $dataSource") + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(1 AS BIGINT), CAST(12345678.99 AS DECIMAL(10,2)))") + table.spark.sql( + s"ALTER TABLE $sideTable ALTER COLUMN dec TYPE DECIMAL(12,2)") + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(2 AS BIGINT), CAST(1234567890.99 AS DECIMAL(12,2)))") + assert( + table.spark + .sql(s"SELECT count(*) FROM $sideTable") + .collect()(0) + .getLong(0) == 2, + "decimal widening should preserve old and new values") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + } + }, + basePreparation.test("surface.schema.nestedAddField") { table => + val sideTable = s"${table.name}_nst" + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") try { - spark.sql(s"CREATE TABLE $side (id BIGINT, s STRUCT) USING $dataSource") - spark.sql(s"INSERT INTO $side VALUES (CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") - spark.sql(s"ALTER TABLE $side ADD COLUMN s.w INT") - assert(spark.sql(s"SELECT count(*) FROM $side WHERE s.w IS NULL").collect()(0).getLong(0) == 1, - "adding a nested struct field must null-fill existing rows") - spark.sql(s"INSERT INTO $side VALUES (CAST(2 AS BIGINT), named_struct('x', 2, 'y', 'b', 'w', 9))") - assert(spark.sql(s"SELECT count(*) FROM $side WHERE s.w = 9").collect()(0).getLong(0) == 1, - "the new nested field must be writable") - } finally spark.sql(s"DROP TABLE IF EXISTS $side") - }() - - val surfaceSchemaNestedDropField: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.schema.nestedDropField") { (spark, table) => - val side = s"${table}_nsd" - spark.sql(s"DROP TABLE IF EXISTS $side") + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, s STRUCT) USING $dataSource") + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") + table.spark.sql( + s"ALTER TABLE $sideTable ADD COLUMN s.w INT") + assert( + table.spark + .sql(s"SELECT count(*) FROM $sideTable WHERE s.w IS NULL") + .collect()(0) + .getLong(0) == 1, + "new nested field should null-fill the existing row") + + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(2 AS BIGINT), " + + "named_struct('x', 2, 'y', 'b', 'w', 9))") + assert( + table.spark + .sql(s"SELECT count(*) FROM $sideTable WHERE s.w = 9") + .collect()(0) + .getLong(0) == 1, + "new nested field should be writable") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + } + }, + basePreparation.test("surface.schema.nestedDropField") { table => + val sideTable = s"${table.name}_nsd" + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") try { - spark.sql(s"CREATE TABLE $side (id BIGINT, s STRUCT) USING $dataSource") - spark.sql(s"INSERT INTO $side VALUES (CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") - val e = Check.intercept[Exception](spark.sql(s"ALTER TABLE $side DROP COLUMN s.x")) - println(s"DIAG nestedDropField: ${e.getClass.getName} :: ${Option(e.getMessage).getOrElse("").take(180)}") - assert(spark.sql(s"SELECT s.x FROM $side").collect()(0).getInt(0) == 1, - "rejected nested drop must leave the field readable") - } finally spark.sql(s"DROP TABLE IF EXISTS $side") - }() - - val surfaceSchemaReorderExisting: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.schema.reorderExisting") { (spark, table) => - spark.sql(s"ALTER TABLE $table ALTER COLUMN ${Core.string0.columnName} FIRST") - val cols = spark.sql(s"SELECT * FROM $table LIMIT 1").columns.toSeq - assert(cols.head == Core.string0.columnName, s"column reorder (FIRST) must change projection order: $cols") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "reorder must not affect data") - }() - - // ── Write-path configs ─────────────────────────────────────────────────────────────────────── - val surfaceWriteDistributionHash: TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource PARTITIONED BY (${Core.datePartition.columnName}) " + - s"TBLPROPERTIES ('write.format.default'='$seedFmt', 'write.distribution-mode'='hash')")() - .insert(3)() - .check("surface.write.distributionHash") { view => - assert(tableProps(view.spark, view.table).get("write.distribution-mode").contains("hash"), "hash mode not honored") - assert(view.after.size == 3, "hash-distributed write failed") - } - - val surfaceWriteTargetFileSize: TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$seedFmt', 'write.target-file-size-bytes'='1048576')")() - .insert(3)() - .check("surface.write.targetFileSize") { view => - assert(tableProps(view.spark, view.table).get("write.target-file-size-bytes").contains("1048576"), "target size not honored") - assert(view.after.size == 3, "write under custom target file size failed") - } - - val surfaceWriteDfToBranch: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.write.dfToBranch") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH wb") - val df = spark.sql(s"SELECT CAST(50 AS BIGINT) AS ${Core.long0.columnName}, 50 AS ${Core.int0.columnName}, " + - s"'row-50' AS ${Core.string0.columnName}, 50.5 AS ${Core.double0.columnName}, " + - s"true AS ${Core.boolean0.columnName}, '2024-01-09-01' AS ${Core.datePartition.columnName}") - df.writeTo(s"$table.branch_wb").append() - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'wb'") == "4", - "DataFrame-API write must land on the branch") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "3", "main must be untouched by the branch DF write") - }() - - // ── Pins: import/migration procedures, views, ANALYZE (expected-unsupported tripwires) ─────── - // The bogus-input probes showed these procedures fail on INPUT (NotFound/NoSuchTable), not on an - // OpenHouse catalog block — so settle register_table with a REAL metadata file: is importing a - // table into the managed catalog (bypassing normal creation) actually possible? - val surfacePinImportProcs: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.pin.importProcs") { (spark, table) => - val metadataFile = spark.sql( - s"SELECT file FROM $table.metadata_log_entries ORDER BY timestamp DESC LIMIT 1").collect()(0).getString(0) - val regOutcome = + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, s STRUCT) USING $dataSource") + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") + val exception = Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE $sideTable DROP COLUMN s.x")) + + println( + "DIAG nestedDropField: " + + s"${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(180)) + assert( + table.spark + .sql(s"SELECT s.x FROM $sideTable") + .collect()(0) + .getInt(0) == 1, + "rejected nested drop should leave the field readable") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + } + }, + basePreparation.test("surface.schema.reorderExisting") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} " + + s"ALTER COLUMN ${Core.string0.columnName} FIRST") + val columns = table.spark + .sql(s"SELECT * FROM ${table.name} LIMIT 1") + .columns + .toSeq + + assert( + columns.head == Core.string0.columnName, + s"FIRST should move the column to the front: $columns") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "column reorder should preserve the rows") + }, + hashPreparation.test("surface.write.distributionHash") { table => + val properties = tableProps(table.spark, table.name) + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + properties.get("write.distribution-mode").contains("hash"), + "hash distribution mode should be retained") + assert( + rowCount == 3, + s"hash-distributed seed should contain 3 rows, got $rowCount") + }, + targetSizePreparation.test("surface.write.targetFileSize") { table => + val properties = tableProps(table.spark, table.name) + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + properties + .get("write.target-file-size-bytes") + .contains("1048576"), + "target file size should be retained") + assert( + rowCount == 3, + s"custom target-size seed should contain 3 rows, got $rowCount") + }, + basePreparation.test("surface.write.dfToBranch") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH wb") + val row = table.spark.sql( + s"SELECT CAST(50 AS BIGINT) AS ${Core.long0.columnName}, " + + s"50 AS ${Core.int0.columnName}, " + + s"'row-50' AS ${Core.string0.columnName}, " + + s"50.5 AS ${Core.double0.columnName}, " + + s"true AS ${Core.boolean0.columnName}, " + + s"'2024-01-09-01' AS ${Core.datePartition.columnName}") + row.writeTo(s"${table.name}.branch_wb").append() + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'wb'") == "4", + "DataFrame writer should append to the branch") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "DataFrame branch write should leave main unchanged") + }, + basePreparation.test("surface.pin.importProcs") { table => + val metadataFile = table.spark + .sql( + s"SELECT file FROM ${table.name}.metadata_log_entries " + + "ORDER BY timestamp DESC LIMIT 1") + .collect()(0) + .getString(0) + val registerOutcome = try { - spark.sql(s"CALL openhouse.system.register_table(table => 'dbMatrix.zz_reg', metadata_file => '$metadataFile')") - val n = countOf(spark, "SELECT count(*) FROM openhouse.dbMatrix.zz_reg") - spark.sql("DROP TABLE IF EXISTS openhouse.dbMatrix.zz_reg") - s"REGISTERED (readable, $n rows) — import into the managed catalog is NOT blocked" - } catch { case t: Throwable => - s"REJECTED ${t.getClass.getName} :: ${Option(t.getMessage).getOrElse("").take(160)}" } - println(s"DIAG pin.register_table(real): $regOutcome") - val snap = Check.intercept[Exception](spark.sql( - s"CALL openhouse.system.snapshot(source_table => '${catalogRelative(table)}', table => 'dbMatrix.zz_snap')")) - println(s"DIAG pin.snapshot: ${snap.getClass.getName} :: ${Option(snap.getMessage).getOrElse("").take(160)}") - val add = Check.intercept[Exception](spark.sql( - s"CALL openhouse.system.add_files(table => '${catalogRelative(table)}', source_table => '`parquet`.`/tmp/zz_nope_dir`')")) - println(s"DIAG pin.add_files: ${add.getClass.getName} :: ${Option(add.getMessage).getOrElse("").take(160)}") - }() - - val surfacePinViewsAnalyze: TableTest[CoreTable.type] = - TableTest(Core).sql("create")(coreCreateParquet)().insert(3)() - .step("surface.pin.viewsAnalyze") { (spark, table) => - val view = Check.intercept[Exception](spark.sql(s"CREATE VIEW openhouse.dbMatrix.zz_v1 AS SELECT 1 AS one")) - println(s"DIAG pin.createView: ${view.getClass.getName} :: ${Option(view.getMessage).getOrElse("").take(160)}") - val analyze = Check.intercept[Exception](spark.sql(s"ANALYZE TABLE $table COMPUTE STATISTICS")) - println(s"DIAG pin.analyze: ${analyze.getClass.getName} :: ${Option(analyze.getMessage).getOrElse("").take(160)}") - }() - - // Compaction × branch: does rewrite_data_files touch/break branch state, and where does it land - // when spark.wap.branch is set? (Untested cell flagged in the surface appraisal.) - val surfaceMaintCompactWithBranch: TableTest[CoreTable.type] = - coreTwoSnapshots.step("surface.maint.compactWithBranch") { (spark, table) => - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')") - spark.sql(s"ALTER TABLE $table CREATE BRANCH cb") - spark.sql(s"INSERT INTO $table.branch_cb VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - spark.sql(s"INSERT INTO $table VALUES (CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - val r = spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('min-input-files', '2'))").collect()(0) - println(s"DIAG compactWithBranch: mainCompaction rewritten=${r.get(0)} added=${r.get(1)}") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "6", "main data preserved by compaction") - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'cb'") == "6", - "branch data preserved and readable after main compaction") - spark.conf.set("spark.wap.branch", "cb") - val confOutcome = try { - val rc = spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}')").collect()(0) - s"RAN (rewritten=${rc.get(0)}, added=${rc.get(1)})" - } catch { case t: Throwable => s"THREW ${t.getClass.getSimpleName} :: ${Option(t.getMessage).getOrElse("").take(140)}" } - finally spark.conf.unset("spark.wap.branch") - println(s"DIAG compactUnderWapConf: $confOutcome") - spark.sql(s"REFRESH TABLE $table") - assert(countOf(spark, s"SELECT count(*) FROM $table") == "6", "main intact after conf-routed compaction attempt") - assert(countOf(spark, s"SELECT count(*) FROM $table VERSION AS OF 'cb'") == "6", "branch intact after conf-routed compaction attempt") - }() - - val surfaceOps: List[(String, TableTest[CoreTable.type])] = List( - "surface.maint.compactWithBranch" -> surfaceMaintCompactWithBranch, - "surface.msg.readabilityGuard" -> surfaceMsgReadabilityGuard, - "branch.leak.setProps" -> surfaceBranchLeakSetProps, - "branch.leak.writeOrderedBy" -> surfaceBranchLeakWriteOrdered, - "branch.wapToggle.noGuard" -> surfaceWapToggleNoGuard, - "wap.neg.doubleCherrypick" -> surfaceWapDoubleCherrypick, - "wap.neg.expireRefTarget" -> surfaceWapExpireRefTarget, - "branch.fastForward.merge" -> surfaceBranchFastForwardMerge, - "branch.fastForward.divergent" -> surfaceBranchFastForwardDivergent, - "branch.replaceBranch" -> surfaceBranchReplaceBranch, - "surface.stream.read" -> surfaceStreamRead, - "surface.stream.write" -> surfaceStreamWrite, - "surface.cdc.changelogView" -> surfaceCdcChangelogView, - "surface.proc.rewriteManifests" -> surfaceProcRewriteManifests, - "surface.proc.rewritePositionDeletes" -> surfaceProcRewritePositionDeletes, - "surface.proc.publishChanges" -> surfaceProcPublishChanges, - "surface.proc.ancestorsOf" -> surfaceProcAncestorsOf, - "surface.proc.removeOrphanReal" -> surfaceProcRemoveOrphanReal, - "surface.meta.hiddenColumns" -> surfaceMetaHiddenColumns, - "surface.meta.tableSweep" -> surfaceMetaTableSweep, - "surface.meta.positionDeletes" -> surfaceMetaPositionDeletes, - "surface.conc.appendAppend" -> surfaceConcAppendAppend, - "surface.conc.updateUpdate" -> surfaceConcUpdateUpdate, - "surface.conc.rtasVsAppend" -> surfaceConcRtasVsAppend, - "surface.schema.relaxNotNull" -> surfaceSchemaRelaxNotNull, - "surface.schema.decimalWiden" -> surfaceSchemaDecimalWiden, - "surface.schema.nestedAddField" -> surfaceSchemaNestedAddField, - "surface.schema.nestedDropField" -> surfaceSchemaNestedDropField, - "surface.schema.reorderExisting" -> surfaceSchemaReorderExisting, - "surface.write.distributionHash" -> surfaceWriteDistributionHash, - "surface.write.targetFileSize" -> surfaceWriteTargetFileSize, - "surface.write.dfToBranch" -> surfaceWriteDfToBranch, - "surface.pin.importProcs" -> surfacePinImportProcs, - "surface.pin.viewsAnalyze" -> surfacePinViewsAnalyze - ) + table.spark.sql( + "CALL openhouse.system.register_table(" + + "table => 'dbMatrix.zz_reg', " + + s"metadata_file => '$metadataFile')") + val rowCount = countOf( + table.spark, + "SELECT count(*) FROM openhouse.dbMatrix.zz_reg") + table.spark.sql( + "DROP TABLE IF EXISTS openhouse.dbMatrix.zz_reg") + s"REGISTERED (readable, $rowCount rows)" + } catch { + case exception: Throwable => + s"REJECTED ${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(160) + } + println(s"DIAG pin.register_table(real): $registerOutcome") + + val snapshotException = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.snapshot(" + + s"source_table => '${catalogRelative(table.name)}', " + + "table => 'dbMatrix.zz_snap')")) + println( + "DIAG pin.snapshot: " + + s"${snapshotException.getClass.getName} :: " + + Option(snapshotException.getMessage).getOrElse("").take(160)) + + val addFilesException = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.add_files(" + + s"table => '${catalogRelative(table.name)}', " + + "source_table => '`parquet`.`/tmp/zz_nope_dir`')")) + println( + "DIAG pin.add_files: " + + s"${addFilesException.getClass.getName} :: " + + Option(addFilesException.getMessage).getOrElse("").take(160)) + }, + basePreparation.test("surface.pin.viewsAnalyze") { table => + val viewException = Check.intercept[Exception]( + table.spark.sql( + "CREATE VIEW openhouse.dbMatrix.zz_v1 AS SELECT 1 AS one")) + println( + "DIAG pin.createView: " + + s"${viewException.getClass.getName} :: " + + Option(viewException.getMessage).getOrElse("").take(160)) + + val analyzeException = Check.intercept[Exception]( + table.spark.sql( + s"ANALYZE TABLE ${table.name} COMPUTE STATISTICS")) + println( + "DIAG pin.analyze: " + + s"${analyzeException.getClass.getName} :: " + + Option(analyzeException.getMessage).getOrElse("").take(160)) + }) + } + + val surfaceCases: List[Plan.Case] = + List("parquet", "orc").flatMap { format => + surfaceBranchCases(format) ++ + surfaceReaderProcedureCases(format) ++ + surfaceRemainingCases(format) + } // ═══ Hazard demonstrations H1-H8 (MODALITY-RECON.md; gates cleared per FEATURE-ANALYSIS-PLAN) ══ // Each was PREDICTED by the state-flow model, verified in code/bytecode, and is demonstrated diff --git a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala new file mode 100644 index 000000000..3fdb41dc3 --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala @@ -0,0 +1,41 @@ +package harness + +import java.nio.charset.StandardCharsets +import java.security.MessageDigest + +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} +import org.junit.jupiter.api.Test + +final class CaseCatalogTest { + private val expectedCaseCount = 2574 + private val expectedCatalogSha256 = + "9e5ec513f2bbc775469154c8d1cf45e14654af2fca0e0f29b4bba6acae286a0a" + + @Test + def orderedCaseCatalogMatchesBaseline(): Unit = { + val caseIds = Plan.caseIds + val actualCatalogSha256 = sha256(caseIds.mkString("\n")) + val duplicateCaseIds = caseIds.groupBy(identity).collect { + case (caseId, occurrences) if occurrences.size > 1 => caseId + }.toList.sorted + + assertTrue( + duplicateCaseIds.isEmpty, + s"case IDs must be unique; duplicates=${duplicateCaseIds.mkString(", ")}") + assertEquals( + expectedCaseCount, + caseIds.size, + s"ordered case catalog changed; count=${caseIds.size}, sha256=$actualCatalogSha256") + assertEquals( + expectedCatalogSha256, + actualCatalogSha256, + s"ordered case catalog changed; count=${caseIds.size}, sha256=$actualCatalogSha256") + } + + private def sha256(value: String): String = + MessageDigest + .getInstance("SHA-256") + .digest(value.getBytes(StandardCharsets.UTF_8)) + .map(byte => f"$byte%02x") + .mkString +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala new file mode 100644 index 000000000..c024ac223 --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala @@ -0,0 +1,20 @@ +package harness + +import org.junit.jupiter.api.Assertions.assertEquals +import org.junit.jupiter.api.Test + +final class TablePreparationTest { + @Test + def formatsCaseIdFromPrefixNameAndLabel(): Unit = { + val preparation = TablePreparation( + "partitioned/orc", + TableTest(CoreTable), + "prep.evolved:") + + val testCase = preparation.test("delete.byPredicate")(_ => ()) + + assertEquals( + "prep.evolved:delete.byPredicate @ partitioned/orc", + testCase.id) + } +} From 5190f3ac24a635d1e4da0329dc72330aebaca259 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Mon, 24 Aug 2026 16:33:20 -0700 Subject: [PATCH 04/24] docs(delta-harness): update test guide Explain the scenario-owned test structure, immutable preparations, and fresh-table isolation used by the localized test cases. Describe the matrix as living documentation for the current harness architecture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../spark/delta-harness/HARNESS-GUIDE.md | 169 +++++++++--------- .../spark/delta-harness/TESTING-MATRIX.md | 46 +++-- 2 files changed, 113 insertions(+), 102 deletions(-) diff --git a/integrations/spark/delta-harness/HARNESS-GUIDE.md b/integrations/spark/delta-harness/HARNESS-GUIDE.md index e90fe3068..21a82d6ba 100644 --- a/integrations/spark/delta-harness/HARNESS-GUIDE.md +++ b/integrations/spark/delta-harness/HARNESS-GUIDE.md @@ -1,4 +1,4 @@ -# delta-harness — a guide to grokking the tests +# delta-harness: a guide to grokking the tests This is the single document to read to understand what this harness is, how it is built, why it is built that way, and what it found. It is written for a person picking the harness up cold. If you read only one @@ -22,15 +22,14 @@ A few facts set expectations before you read further. every case passes with no divergence between the ORC and Parquet encodings. The guide deliberately does not quote an exact case count, because that number changes every time a case is added; the exact figure is whatever the final line of a full run prints. -- A test is written as a typed pipeline (`TableTest[S <: Schema]`). A preparation prefix (create and - seed, or RTAS, or drop and undrop) is composed with an operation suffix (the thing under test), and - every step asserts a delta against the observed pre-state rather than an absolute row set. -- The suite scales by crossing one authored operation against many substrates (file format, partitioning, - copy-on-write versus merge-on-read, replace-lineage, branch, and restored-from-undrop). File format is - a per-case parameter, so most blocks run on both Parquet and ORC automatically. -- The purpose is to find broken feature interactions, not to accumulate green cases. The findings — the +- A test is a localized `Plan.Case`. A reusable `TablePreparation` creates a fresh table in a known + state, then the case body performs the action and assertions together. +- The suite scales by constructing localized cases from shared preparation collections for file + format, partitioning, copy-on-write versus merge-on-read, replace lineage, branch, and restored + tables. Every case materializes its own table from the selected preparation. +- The purpose is to find broken feature interactions, not to accumulate green cases. The findings, the `G`-series product-behavior notes, the `WAP1` note, the fork behaviors, and an error-message - readability audit — are the real output, and the green count only tells you that the tripwires are + readability audit, are the real output, and the green count only tells you that the tripwires are still where they were left. --- @@ -69,8 +68,8 @@ one or less runs sequentially. ### What the script does `run-openhouse.sh` performs three steps. First, it resolves the OpenHouse classpath through a system -Gradle — the Gradle wrapper cannot download behind the proxy, as noted in the pitfalls below — and caches -the result. Second, it compiles every `.scala` file under `src/main/scala/harness/openhouse/` with +Gradle. The Gradle wrapper cannot download behind the proxy, as noted in the pitfalls below. The script +caches the result. Second, it compiles every `.scala` file under `src/main/scala/harness/openhouse/` with `scalac`. Third, it runs `harness.Main` on JDK 17 with the `--add-opens` flags that Spark 3.5 needs. Gradle is used only to produce the classpath and OpenHouse's own jars; it does not build the harness. @@ -78,45 +77,49 @@ Gradle is used only to produce the classpath and OpenHouse's own jars; it does n ## 3. The mental model, and why a test looks the way it does -A test is a typed pipeline, `TableTest[S <: Schema]`. The type parameter `S` names the table -implementation the test depends on, and every step references that schema's columns through typed handles -such as `row.get(CoreTable.long0): Long`. The compiler therefore forbids mixing schemas or naming a column -the schema does not declare, so a whole class of "the test drifted from the table shape" bug is impossible -by construction. +A test is a `Plan.Case` owned by one scenario trait. The case is usually created through +`TablePreparation.test`, which makes the preparation, action, and assertions readable in one +continuous block: -Four ideas do all of the work. +```scala +preparedCoreTables.flatMap { preparation => + List( + preparation.test("delete.byPredicate") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") -1. **A schema is columns only.** `CoreTable` has one column per common type plus a `datepartition` string - in the form `YYYY-MM-DD-HH`, while `NestedTable` and `TypesTable` cover struct and complex types and - type-edge coverage respectively. Each `Column[T]` carries its Scala type and a deterministic + assert(table.rows == table.preparedRows.filterNot(_.get(Core.long0) < 2)) + }) +} +``` + +Four ideas define this structure. + +1. **A schema is columns only.** `CoreTable` has one column per common type plus a `datepartition` + string in the form `YYYY-MM-DD-HH`, while `NestedTable` and `TypesTable` cover nested structures + and type-edge coverage. Each `Column[T]` carries its Scala type and a deterministic `literalAt(rowIndex)` generator, so seeding is reproducible and schema-checked. -2. **A preparation prefix and an operation suffix compose with `andThen`.** An operation — the thing under - test, such as a `DELETE`, a `MERGE`, or an `ADD COLUMN` — is authored headless, meaning it assumes a - seeded table and does not create one. The run composes a preparation before it. Because the preparation - and the operation are the same kind of object, you can swap the preparation without touching the - operation, and that is the entire trick that lets the whole DML catalog be re-run on an RTAS'd table, a - branch-routed table, or a table that has been through a real drop-then-undrop round trip. The operation - set is authored once, and the substrate set multiplies it. - -3. **The layout axis is file format crossed with partitioning.** A `Layout` is expressed as a literal - `CREATE` statement. There are six base layouts — the two partitionings crossed with Parquet, ORC, and - Avro — plus merge-on-read variants, plus dedicated single-data-file layouts used as a physical - copy-on-write versus merge-on-read discriminator. On such a layout, a strict-subset delete on one data - file must produce a position-delete file under merge-on-read and must not produce one under - copy-on-write, and the harness asserts exactly that against the `.delete_files` metadata table. - -4. **Assertions are deltas, never absolutes.** Each step's validation thunk receives a `StepView` that - carries `before` and `after` row snapshots along with `snapshotsBefore` and `snapshotsAfter` commit - counts. Every operation asserts a change — two rows fewer, one new snapshot, this key now excluded — so - the identical assertion holds under any layout, any seed size, and any substrate. This is what makes a - single authored operation valid across the whole substrate cross. +2. **A preparation is an immutable recipe.** `TableTest[S]` remains the typed pipeline used to define + reusable setup such as create and seed, ordered, evolved, RTAS, merge-on-read, branch, or undrop. + `TablePreparation[S]` gives that recipe a case label and optional post-case assertion. A preparation + object stores instructions, not a live table. + +3. **Every case receives a fresh prepared table.** `TablePreparation.test` runs its recipe against a + unique table name, captures the prepared rows and snapshot count in `PreparedTable[S]`, executes the + localized case body, runs any preparation-level postcondition, and drops the table in `finally`. + Reusing a preparation across a family therefore preserves isolation. + +4. **The action and assertions stay together.** The case body issues the SQL or API call and immediately + asserts the resulting rows, snapshots, metadata, or error. `PreparedTable.preparedRows` and + `preparedSnapshotCount` support relative assertions, while `rows` and `snapshotCount` read the live + state. A reviewer can follow a test from setup choice through action to expected result without + jumping through a separate operation catalog or central assembly file. The parallel runner, `harness.Main`, runs cases on a worker pool, and each worker gets its own -`spark.newSession()` with a separate `SQLConf`, so the session-global state that some tests mutate — such -as `spark.wap.branch`, `spark.wap.id`, and changelog temp views — never leaks between cases. Results are -collected and printed in the original case order, so the output is identical to a sequential run. Each -case owns its own table through an atomic counter, so the cases are independent. +`spark.newSession()` with a separate `SQLConf`. Session state such as `spark.wap.branch`, +`spark.wap.id`, and changelog temp views is scoped to one worker session. Results are collected and +printed in catalog order. Fresh table names and per-case teardown keep table state isolated. Known product bugs are tagged rather than skipped into silence. `Plan.knownBugs` maps a case-id substring to a reason, and a matching case is reported as `SKIP (bug: …)`. This is how a genuine defect is deferred @@ -131,9 +134,9 @@ not affect the package. Open the file whose concern matches what you are after. | File | What it holds | |---|---| -| `Framework.scala` | This file holds the DSL and the plumbing: `Ctx`, the REST and `HtsAdmin` clients, `Outcome` and `Check`, the `Column`/`Schema`/`Rows` vocabulary, the three tables, `RowGenerator`, `StepView` and `Step`, and `TableTest` itself. Read it first to learn the vocabulary. | -| `ScenarioKit.scala` | This is the shared kit that every test group builds on. It holds `Layout` and the layout lists, all of the `createAndSeed*` preparations, the format-multiplex hooks (`seedFmt` and `withSeedFmt`), and the cross-cutting helpers. Every `*Scenarios` trait extends it, and any helper used by more than one trait belongs here. | -| `DmlScenarios.scala` | This is the core DML surface. It holds the read, delete, update, merge, and insert/append/overwrite operation catalog; the `operations`, `partitionedOperations`, and `mutationOperations` lists; the DDL-by-consumer battery; the ADD COLUMN family; and the physical copy-on-write versus merge-on-read discriminator. | +| `Framework.scala` | This file holds the DSL and plumbing: `Ctx`, the REST and `HtsAdmin` clients, `Outcome` and `Check`, the typed schema vocabulary, `TableTest` preparation pipelines, `TablePreparation`, and `PreparedTable`. Read it first to learn the lifecycle. | +| `ScenarioKit.scala` | This is the shared kit that every test group builds on. It holds `Layout`, reusable preparation collections, the format-multiplex hooks used by context-only cases, and cross-cutting helpers. Every `*Scenarios` trait extends it. | +| `DmlScenarios.scala` | This is the core DML surface. It owns localized read, delete, update, merge, insert, append, overwrite, DDL-consumer, schema-DDL, and copy-on-write versus merge-on-read discriminator cases. | | `NestedTypesScenarios.scala` | This holds nested and complex-type coverage, type-edge coverage, and partition transforms together with partition-evolution rejections. | | `MorMaintScenarios.scala` | This holds merge-on-read delete-file coexistence (operations on a table that already carries a live position delete), merge-on-read maintenance folds, merge-on-read modality hazards, and merge-on-read crossed with branch merge. | | `MaintControlScenarios.scala` | This holds time travel, restore and rollback, the maintenance procedures such as `expire_snapshots` and `rewrite_data_files`, the REST control-plane operations for lock and unlock, and the undrop admin lifecycle. | @@ -143,7 +146,7 @@ not affect the package. Open the file whose concern matches what you are after. | `InteractionScenarios.scala` | This holds the three-way compositions where the interesting behavior lives: DDL crossed with history, RTAS crossed with history, lineage, and property-merge, branch crossed with history and maintenance, and the composite branch-expiration-merge defect. | | `SurfaceScenarios.scala` | This holds surface completion: the error-message readability guard, branch leaks, WAP negatives, streaming and CDC, procedures, metadata tables, concurrency invariants, schema-evolution edges, write-path configs, and expected-unsupported pins. | | `HazardReaderWriterScenarios.scala` | This holds the hazard and modality interactions (expired checkpoints, RTAS wiping tags, rename breaking consumers) and the reader-by-writer-class battery (changelog, incremental, and streaming over both copy-on-write and merge-on-read). | -| `Plan.scala` | This is the assembly. `object Plan` is where substrates crossed with operations become the actual `Case` list, where `crossFmt` doubles a block across Parquet and ORC, and where `knownBugs` lives. If you want to know what actually runs, read `Plan.cases`. | +| `Plan.scala` | This is the ordered index. `object Plan` concatenates scenario-owned case lists and holds `knownBugs`. It contains no test behavior or matrix construction. | | `OpenHouseMatrix.scala` | This mixes the domain traits into `object Scenarios`. The `extends` clause here is the authoritative order in which the traits' `val`s initialize, as explained in section 6. | | `Env.scala` | This handles boot and run: the embedded OpenHouse server wiring in `OpenHouseEnv`, the embedded real HTS in `HtsEnv` and `HtsBootApp`, the retrying `Runner`, and `Main`. | @@ -151,16 +154,17 @@ not affect the package. Open the file whose concern matches what you are after. ## 5. The axes, and why the honest target is well below the naive product -You can think of the suite as substrates crossed with operations crossed with consumers. +You can think of the suite as preparations crossed with localized behaviors and consumers. -- The operations are the DML catalog (authored once as explicit literals in `DmlScenarios.operations`), - together with the DDL operations and the procedures. -- The substrates are the preparations — plain create-and-seed, RTAS'd (replace-lineage), branch-routed - through `spark.wap.branch`, restored-from-drop on the real HTS, schema-evolved, sort-ordered, and - merge-on-read — and each of them multiplies the operation catalog. -- The consumers answer a question: after a state-changing DDL, does each reader — plain scan, time - travel, changelog, incremental, and streaming — still work? -- File format is a per-case parameter, described below, so blocks double across Parquet and ORC for free. +- The scenario traits define the behaviors: DML, DDL, procedures, branch operations, streaming, and + feature interactions. +- The preparation collections define the starting states: plain create and seed, RTAS replace lineage, + branch routed through `spark.wap.branch`, restored from drop on the real HTS, schema evolved, sort + ordered, and merge-on-read. +- Each family constructs one local case body per applicable preparation. The body contains the action + and assertions for that exact combination. +- The consumers answer a question: after a state-changing DDL, does each reader, such as a plain scan, + time travel, changelog, incremental, or streaming read, still work? The naive product is much larger than what actually runs, because a large fraction of the cells would be vacuous, and the harness refuses to inflate its count with them. Three arguments carry most of that @@ -179,14 +183,11 @@ section explains. ### Format multiplex, and why "format-inert" is a hypothesis rather than an assumption -Every table-creating block reads a per-case thread-local seed format, `seedFmt`, and `Plan.crossFmt` wraps -a block so that it runs once per format in `dataFormats` (Parquet and ORC), setting `seedFmt` around each -case. The mechanism is safe because cases run sequentially per worker. The point is a philosophical one: -you do not bake a file format into a test. Whether a behavior is format-independent is something this -harness verifies rather than assumes, because the fork carries patched ORC paths and the replace-path -findings showed metadata surprises. Only table-less operations, which issue no `CREATE`, have no format -axis. This is why the summary above says there is no divergence between ORC and Parquet: that is a checked -result, not a design assumption. +Most table-creating families iterate a preparation collection whose labels and `CREATE TABLE` recipes +already carry the format. Context-only families either create a fixed Parquet table when encoding is +irrelevant or construct one explicit case per selected format. Whether a behavior is format-independent +is something the harness verifies rather than assumes, because the fork carries patched ORC paths and +replace-path findings have exposed metadata differences. Only table-less operations have no format axis. --- @@ -274,8 +275,8 @@ The second group is behavior and limitation findings. - **G13 is that CDC changelog is unsupported over a merge-on-read table after an UPDATE or MERGE**, which fails with "Delete files are currently not supported in changelog scans". Merge-on-read delete-only and - all copy-on-write cases work, but merge-on-read update and merge — the shapes a merge-on-read table - exists to optimize — break CDC silently. This is a stock Iceberg 1.5 limitation, and it is demonstrated + all copy-on-write cases work, but merge-on-read update and merge, the shapes a merge-on-read table + exists to optimize, break CDC silently. This is a stock Iceberg 1.5 limitation, and it is demonstrated by `readerWriter.changelog.{update,merge}.mor`. - **G14 is that `rewrite_data_files` leaves a dangling position delete on a merge-on-read table.** Compaction applies the delete, so the row set is correct, but it does not fold out the now-dangling @@ -300,10 +301,10 @@ Finally, the tagged and deferred defects are the ones that appear in `Plan.known silent no-op (a genuine OpenHouse regression traced to server commit #558), and encryption that writes plaintext because the KMS plugin is out of the repository. -> The exhaustive ledgers behind this section — the findings with code citations, the fork-commit audit, -> the tagged-defect ledger, and the dated run log — live alongside the harness in the pull request that -> developed it, and not necessarily in this tree. You do not need them to grok the tests, so reach for -> them only when you want the evidence behind a specific claim made here. +> The exhaustive ledgers behind this section include the findings with code citations, the fork-commit +> audit, the tagged-defect ledger, and the dated run log. They live alongside the harness in the pull +> request that developed it, and not necessarily in this tree. You do not need them to grok the tests, +> so reach for them only when you want the evidence behind a specific claim made here. --- @@ -328,8 +329,8 @@ overclaim. `#251` backports column defaults to the API and core, but there is no no Spark wiring in the open fork, because `SparkTable` does not implement `SupportsColumnDefaultValue`. As a result, over OSS Spark, `ADD COLUMN … DEFAULT 5` parses, but the default is not written into the Iceberg schema, old rows read NULL, and an INSERT that omits the column is rejected. The serialization does round -trip on a branch build. The harness pins exactly that — the observable OSS-Spark DDL behavior and the -serialization — and it explicitly does not claim the feature is broken, because read-application may exist +trip on a branch build. The harness pins exactly that: the observable OSS-Spark DDL behavior and the +serialization. It explicitly does not claim the feature is broken, because read-application may exist in LinkedIn's private Spark, which this harness cannot see. A whole-suite branch-versus-release run, performed through `ICEBERG_RUNTIME_JAR`, showed no correctness deltas. @@ -337,16 +338,18 @@ performed through `ICEBERG_RUNTIME_JAR`, showed no correctness deltas. ## 9. Adding a test -Adding a test follows a short recipe. First, pick the schema, which is `CoreTable` unless you need nesting -or type edges. Second, author the operation headless, as a `TableTest` step that assumes a seeded table -and asserts a delta through its `StepView`, using `view.before` and `view.after`, `snapshotsBefore` and -`snapshotsAfter`, and the metadata tables such as `.delete_files` and `.snapshots`. Third, put it in the -trait whose concern matches, as described in section 4, and if it needs a helper used by another trait, -add that helper to `ScenarioKit`. Fourth, wire it into `Plan` by adding it to the relevant list, and use -`crossFmt(...)` if it creates a table, so that it runs on both Parquet and ORC; do not bake a single -format into it. Fifth, if it exercises a real product bug that you are deferring, tag it in -`Plan.knownBugs` with a reason, and never let it pass or skip silently. Finally, run the slice and then the -full gate, and confirm that the count moved by what you expect and that nothing else regressed. +Adding a test follows a short recipe. First, pick the schema, which is `CoreTable` unless the behavior +needs nested or type-edge columns. Second, select or add the smallest reusable preparation that produces +the required starting state. Third, add a `preparation.test("caseName") { table => ... }` body in the +scenario trait whose concern matches. Keep the action and every assertion in that body. Use +`table.preparedRows` and `table.preparedSnapshotCount` for the starting state, and use `table.rows`, +`table.snapshotCount`, and metadata queries for the result. + +Construct the family across each applicable preparation or format in the scenario trait, then add the +scenario-owned case list to `Plan.cases` in the intended catalog position. If the case characterizes a +deferred product bug, tag it in `Plan.knownBugs` with a reason. Run the catalog regression test to verify +the intended count and ordering change, run the narrow local slice, and then run the broader validation +gate. --- diff --git a/integrations/spark/delta-harness/TESTING-MATRIX.md b/integrations/spark/delta-harness/TESTING-MATRIX.md index 4713c16bf..cabe0bf03 100644 --- a/integrations/spark/delta-harness/TESTING-MATRIX.md +++ b/integrations/spark/delta-harness/TESTING-MATRIX.md @@ -2,9 +2,14 @@ This document describes how the harness is organized. Every case in the suite is one point in a cross product of independent axes, so the suite is best understood as a matrix rather than a flat -list of tests. `Plan.cases` (in `Plan.scala`) assembles the matrix by crossing an operation list with -a layout or format axis for each family, and `Scenarios` (in `OpenHouseMatrix.scala`) supplies the -operations by mixing in the per-domain traits. +list of tests. Each scenario trait owns the complete cases for its domain. Shared +`TablePreparation` recipes create common starting states, and each +`preparation.test("caseName") { table => ... }` body keeps the action and assertions together. +`Plan.cases` preserves the catalog order by concatenating those scenario-owned case lists. + +Every prepared case receives a fresh table. The preparation creates and seeds that table, the case +body performs the behavior under test, and teardown drops the table after the body completes. A +shared preparation is therefore a reusable immutable recipe, not a table shared by multiple cases. ## How to read a case id @@ -104,7 +109,8 @@ reads directly in the case id. ## Preparation lineages Preparation determines what the base table has already been through when the operation runs. The -same operation list is reused across lineages so a behavior can be checked on each base. +same preparation recipes are reused across cases so a behavior can be checked on each base without +moving its action or assertions away from the case body. | Lineage | Explanation | |---------|-------------| @@ -119,9 +125,10 @@ same operation list is reused across lineages so a behavior can be checked on ea ## Operation families -Each family is an operation list that `Plan.cases` crosses with a layout or format axis. The tables -below name the family and describe what it exercises. Representative operation names are included so -the family is recognizable in the case ids. +Each family owns a list of localized cases. The family constructs those cases from the applicable +preparation and format collections, while each case body contains the behavior and its assertions. +The tables below name each family and describe what it exercises. Representative operation names +are included so the family is recognizable in the case ids. ### DML @@ -139,17 +146,17 @@ branch lineages, because those lineages are about the mutation write path. ### DDL DDL is split into sub-families so each area of the OpenHouse table surface is exercised on its own. -Every sub-family crosses its operation list with the six copy-on-write layouts unless noted. +Every sub-family constructs cases for the six copy-on-write layouts unless noted. | Sub-family | Operations | Explanation | |------------|-----------|-------------| -| Schema evolution (`ddlSchemaOperations`) | `ddl.addColumn.single`, `ddl.addColumn.multiple`, `ddl.addColumn.comment`, `ddl.addColumn.position`, `ddl.alterColumn.typeWiden`, `ddl.renameColumn` | Column additions in each position and with comments, safe type widening, and column rename. These exercise how the server validates and applies a schema change. | -| Table properties (`ddlPropsOperations`) | `ddl.props.userRoundTrip`, `ddl.props.reservedOpenhouse`, `ddl.props.formatVersionForced`, `ddl.props.previousVersionsHonored` | User property round-tripping, the handling of reserved OpenHouse properties, forced format version, and honoring previously set versions. | -| Miscellaneous (`ddlMiscOperations`) | `ddl.sortOrder.orderedBy`, `ddl.sortOrder.orderedByMulti`, `ddl.renameTable`, `ddl.renameTable.conflict`, `ddl.ns.createRejected`, `ddl.ns.dropRejected` | Setting a sort order, renaming a table and the name-conflict case, and the namespace create and drop rejections. | -| Policy (`ddlPolicyOperations`) | `ddl.policy.sharing`, `ddl.policy.history`, `ddl.policy.replication`, `ddl.policy.retention`, `ddl.policy.neg.historyMaxAge`, `ddl.policy.neg.historyVersions` | `SET POLICY` for sharing, history, replication, and retention, plus the negative cases where a policy bound is out of range. | -| CTAS and RTAS (`ddlCtasRtasOperations`) | `ddl.ctas`, `ddl.rtas.enabled`, `ddl.rtas.disabled`, `ddl.rtas.replicationConflict` | Create-table-as-select, replace-table-as-select with replace enabled and disabled, and the replace-under-replication conflict. | -| Tagging, ACL, and features (`ddlTagAclFeatureOperations`) | `ddl.colTag`, `ddl.acl.grantUnshared`, `ddl.acl.grantShared`, `ddl.featureFlag.distributionMode`, `ddl.repl.tableTypeImmutable`, `ddl.encryption.active` | Column tagging, ACL grants on shared and unshared tables, the distribution-mode feature flag, replica-table-type immutability, and the encryption-active property. | -| Encryption (`ddlEncryptionOperations`) | `ddl.encryption` | The encryption capability, pinned on Parquet. | +| Schema evolution (`ddlSchemaCases`) | `ddl.addColumn.single`, `ddl.addColumn.multiple`, `ddl.addColumn.comment`, `ddl.addColumn.position`, `ddl.alterColumn.typeWiden`, `ddl.renameColumn` | Column additions in each position and with comments, safe type widening, and column rename. These exercise how the server validates and applies a schema change. | +| Table properties (`ddlPropertyCases`) | `ddl.props.userRoundTrip`, `ddl.props.reservedOpenhouse`, `ddl.props.formatVersionForced`, `ddl.props.previousVersionsHonored` | User property round-tripping, the handling of reserved OpenHouse properties, forced format version, and honoring previously set versions. | +| Miscellaneous (`ddlMiscellaneousCases`) | `ddl.sortOrder.orderedBy`, `ddl.sortOrder.orderedByMulti`, `ddl.renameTable`, `ddl.renameTable.conflict`, `ddl.ns.createRejected`, `ddl.ns.dropRejected` | Setting a sort order, renaming a table and the name-conflict case, and the namespace create and drop rejections. | +| Policy (`ddlPolicyCases`) | `ddl.policy.sharing`, `ddl.policy.history`, `ddl.policy.replication`, `ddl.policy.retention`, `ddl.policy.neg.historyMaxAge`, `ddl.policy.neg.historyVersions` | `SET POLICY` for sharing, history, replication, and retention, plus the negative cases where a policy bound is out of range. | +| CTAS and RTAS (`ddlCtasRtasCases`) | `ddl.ctas`, `ddl.rtas.enabled`, `ddl.rtas.disabled`, `ddl.rtas.replicationConflict` | Create-table-as-select, replace-table-as-select with replace enabled and disabled, and the replace-under-replication conflict. | +| Tagging, ACL, and features (`ddlTagAclFeatureCases`) | `ddl.colTag`, `ddl.acl.grantUnshared`, `ddl.acl.grantShared`, `ddl.featureFlag.distributionMode`, `ddl.repl.tableTypeImmutable`, `ddl.encryption.active` | Column tagging, ACL grants on shared and unshared tables, the distribution-mode feature flag, replica-table-type immutability, and the encryption-active property. | +| Encryption (`ddlEncryptionCases`) | `ddl.encryption` | The encryption capability, pinned on Parquet. | The schema-evolution operations are also crossed with every layout as a separate `ddlSchema` block, and there is a DDL-then-consumer battery (`ddlConsumeBattery`) that applies each state-changing DDL @@ -224,10 +231,11 @@ They characterize the fork surface at the API and table-property level. ## How assertions are framed -Every case asserts a delta against the pre-state it observed, meaning the change in rows or in the -commit count, rather than an absolute row set. Framing assertions as deltas is what lets one -operation hold under any layout, format, and lineage, which is what makes the cross product -meaningful. +Each `PreparedTable` exposes the rows and snapshot count produced by its preparation, plus methods +that read the live rows and snapshot count after the action. Cases use those values to assert the +relevant row, commit, metadata, or error outcome. Delta assertions remain useful when the expected +behavior is relative to the starting state, while cases with a fixed contract can state that +contract directly in the same body. ## Known bugs From 40642512e60b83b510f1532bd64d95346c787286 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Mon, 24 Aug 2026 17:01:21 -0700 Subject: [PATCH 05/24] docs(delta-harness): move guides to stack Remove the harness guide and testing matrix from the implementation PR so the documentation can be reviewed in a separate stacked change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../spark/delta-harness/HARNESS-GUIDE.md | 363 ------------------ .../spark/delta-harness/TESTING-MATRIX.md | 244 ------------ 2 files changed, 607 deletions(-) delete mode 100644 integrations/spark/delta-harness/HARNESS-GUIDE.md delete mode 100644 integrations/spark/delta-harness/TESTING-MATRIX.md diff --git a/integrations/spark/delta-harness/HARNESS-GUIDE.md b/integrations/spark/delta-harness/HARNESS-GUIDE.md deleted file mode 100644 index 21a82d6ba..000000000 --- a/integrations/spark/delta-harness/HARNESS-GUIDE.md +++ /dev/null @@ -1,363 +0,0 @@ -# delta-harness: a guide to grokking the tests - -This is the single document to read to understand what this harness is, how it is built, why it is built -that way, and what it found. It is written for a person picking the harness up cold. If you read only one -file, read this one; `run-openhouse.sh` and the `*.scala` sources are the ground truth beneath it. - ---- - -## 1. What it is, in brief - -`delta-harness` is a self-contained Scala test rig that drives real, customer-facing Spark SQL against a -real embedded OpenHouse catalog and asserts what actually happened to the table. It is not a unit test of -OpenHouse internals. It is a behavioral matrix over the surface a data engineer actually touches: -`DELETE`, `UPDATE`, `MERGE`, `INSERT`, and `OVERWRITE`; copy-on-write versus merge-on-read; DDL; -branching and Write-Audit-Publish; time travel; restore; maintenance procedures; streaming and CDC -readers; the drop-then-undrop lifecycle; and the behaviors specific to LinkedIn's `com.linkedin.iceberg` -1.5.2 fork. - -A few facts set expectations before you read further. - -- The suite runs a few thousand cases in each mode (the in-memory-stub mode and the real-HTS mode), and - every case passes with no divergence between the ORC and Parquet encodings. The guide deliberately does - not quote an exact case count, because that number changes every time a case is added; the exact figure - is whatever the final line of a full run prints. -- A test is a localized `Plan.Case`. A reusable `TablePreparation` creates a fresh table in a known - state, then the case body performs the action and assertions together. -- The suite scales by constructing localized cases from shared preparation collections for file - format, partitioning, copy-on-write versus merge-on-read, replace lineage, branch, and restored - tables. Every case materializes its own table from the selected preparation. -- The purpose is to find broken feature interactions, not to accumulate green cases. The findings, the - `G`-series product-behavior notes, the `WAP1` note, the fork behaviors, and an error-message - readability audit, are the real output, and the green count only tells you that the tripwires are - still where they were left. - ---- - -## 2. How to run it - -The harness requires JDK 17, because the repository pins Lombok 1.18.20, which does not compile on JDK 21 -or newer. Point the script at a 17 through `JAVA17_HOME`; it also accepts `JAVA_HOME` when that already -points at a 17. - -```bash -export JAVA17_HOME=/usr/lib/jvm/java-17-openjdk-amd64 # or wherever your 17 lives - -./run-openhouse.sh # the full matrix; the last printed line is the case count -./run-openhouse.sh delete parquet # a fast slice (~25s): delete tests, Parquet only -./run-openhouse.sh merge parquet # merge tests on Parquet -./run-openhouse.sh delete.byPredicate # one operation across its layouts -``` - -Each positional argument is an AND-substring filter on the case id, so a case runs only if its id -contains all of the arguments. The match is a substring rather than an exact token, which means -`partitioned` also matches `unpartitioned`. A narrow slice takes roughly 25 seconds end to end, because -the embedded-server and Spark startup dominate while the assertions themselves take milliseconds. You -should iterate on a slice and run the whole matrix only as a final gate. - -Two environment variables change what is exercised. - -| Variable | What it does | -|---|---| -| `HARNESS_REAL_HTS=1` | This boots the real House Table Service as a second in-JVM Spring context and runs the drop-then-undrop blocks (`undrop:*`, `undropAdmin.*`, and the `undropInteract` three-way compositions) against it. When the variable is unset, the harness uses an in-memory stub and skips those blocks, which is why the real-HTS run has more cases than the default run. | -| `ICEBERG_RUNTIME_JAR=` | This is branch-testing mode. It swaps the shaded Iceberg runtime jar on the classpath for a locally built fork-branch-HEAD jar, so the whole suite runs against un-released fork bytecode. The swap is reversible, and it hard-fails when the jar it is asked to replace is not found, so a typo cannot silently leave the release jar in place. | - -`HARNESS_PARALLELISM=N` overrides the worker count, which otherwise defaults to the CPU count; a value of -one or less runs sequentially. - -### What the script does - -`run-openhouse.sh` performs three steps. First, it resolves the OpenHouse classpath through a system -Gradle. The Gradle wrapper cannot download behind the proxy, as noted in the pitfalls below. The script -caches the result. Second, it compiles every `.scala` file under `src/main/scala/harness/openhouse/` with -`scalac`. Third, it runs `harness.Main` on JDK 17 with the `--add-opens` flags that Spark 3.5 needs. -Gradle is used only to produce the classpath and OpenHouse's own jars; it does not build the harness. - ---- - -## 3. The mental model, and why a test looks the way it does - -A test is a `Plan.Case` owned by one scenario trait. The case is usually created through -`TablePreparation.test`, which makes the preparation, action, and assertions readable in one -continuous block: - -```scala -preparedCoreTables.flatMap { preparation => - List( - preparation.test("delete.byPredicate") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") - - assert(table.rows == table.preparedRows.filterNot(_.get(Core.long0) < 2)) - }) -} -``` - -Four ideas define this structure. - -1. **A schema is columns only.** `CoreTable` has one column per common type plus a `datepartition` - string in the form `YYYY-MM-DD-HH`, while `NestedTable` and `TypesTable` cover nested structures - and type-edge coverage. Each `Column[T]` carries its Scala type and a deterministic - `literalAt(rowIndex)` generator, so seeding is reproducible and schema-checked. - -2. **A preparation is an immutable recipe.** `TableTest[S]` remains the typed pipeline used to define - reusable setup such as create and seed, ordered, evolved, RTAS, merge-on-read, branch, or undrop. - `TablePreparation[S]` gives that recipe a case label and optional post-case assertion. A preparation - object stores instructions, not a live table. - -3. **Every case receives a fresh prepared table.** `TablePreparation.test` runs its recipe against a - unique table name, captures the prepared rows and snapshot count in `PreparedTable[S]`, executes the - localized case body, runs any preparation-level postcondition, and drops the table in `finally`. - Reusing a preparation across a family therefore preserves isolation. - -4. **The action and assertions stay together.** The case body issues the SQL or API call and immediately - asserts the resulting rows, snapshots, metadata, or error. `PreparedTable.preparedRows` and - `preparedSnapshotCount` support relative assertions, while `rows` and `snapshotCount` read the live - state. A reviewer can follow a test from setup choice through action to expected result without - jumping through a separate operation catalog or central assembly file. - -The parallel runner, `harness.Main`, runs cases on a worker pool, and each worker gets its own -`spark.newSession()` with a separate `SQLConf`. Session state such as `spark.wap.branch`, -`spark.wap.id`, and changelog temp views is scoped to one worker session. Results are collected and -printed in catalog order. Fresh table names and per-case teardown keep table state isolated. - -Known product bugs are tagged rather than skipped into silence. `Plan.knownBugs` maps a case-id substring -to a reason, and a matching case is reported as `SKIP (bug: …)`. This is how a genuine defect is deferred -without either failing the suite or silently pretending it passed. - ---- - -## 4. Where things live - -The harness is split by concern, and every file declares `package harness`, so the directory name does -not affect the package. Open the file whose concern matches what you are after. - -| File | What it holds | -|---|---| -| `Framework.scala` | This file holds the DSL and plumbing: `Ctx`, the REST and `HtsAdmin` clients, `Outcome` and `Check`, the typed schema vocabulary, `TableTest` preparation pipelines, `TablePreparation`, and `PreparedTable`. Read it first to learn the lifecycle. | -| `ScenarioKit.scala` | This is the shared kit that every test group builds on. It holds `Layout`, reusable preparation collections, the format-multiplex hooks used by context-only cases, and cross-cutting helpers. Every `*Scenarios` trait extends it. | -| `DmlScenarios.scala` | This is the core DML surface. It owns localized read, delete, update, merge, insert, append, overwrite, DDL-consumer, schema-DDL, and copy-on-write versus merge-on-read discriminator cases. | -| `NestedTypesScenarios.scala` | This holds nested and complex-type coverage, type-edge coverage, and partition transforms together with partition-evolution rejections. | -| `MorMaintScenarios.scala` | This holds merge-on-read delete-file coexistence (operations on a table that already carries a live position delete), merge-on-read maintenance folds, merge-on-read modality hazards, and merge-on-read crossed with branch merge. | -| `MaintControlScenarios.scala` | This holds time travel, restore and rollback, the maintenance procedures such as `expire_snapshots` and `rewrite_data_files`, the REST control-plane operations for lock and unlock, and the undrop admin lifecycle. | -| `ForkScenarios.scala` | This holds the `com.linkedin.iceberg` fork-behavior pins; the fork commits themselves are tabulated in section 8. | -| `BranchWapScenarios.scala` | This holds branching and Write-Audit-Publish: the undrop three-way compositions, the direct-branch operations, and the branch and WAP battery, which covers staged-write publish visibility and the systematic branch-DDL leak. | -| `NegativeDdlScenarios.scala` | This holds the typed negatives and contract pins together with the DDL phases: properties, sort order, rename, namespace, policy, CTAS and RTAS, column tags and ACL, and encryption. | -| `InteractionScenarios.scala` | This holds the three-way compositions where the interesting behavior lives: DDL crossed with history, RTAS crossed with history, lineage, and property-merge, branch crossed with history and maintenance, and the composite branch-expiration-merge defect. | -| `SurfaceScenarios.scala` | This holds surface completion: the error-message readability guard, branch leaks, WAP negatives, streaming and CDC, procedures, metadata tables, concurrency invariants, schema-evolution edges, write-path configs, and expected-unsupported pins. | -| `HazardReaderWriterScenarios.scala` | This holds the hazard and modality interactions (expired checkpoints, RTAS wiping tags, rename breaking consumers) and the reader-by-writer-class battery (changelog, incremental, and streaming over both copy-on-write and merge-on-read). | -| `Plan.scala` | This is the ordered index. `object Plan` concatenates scenario-owned case lists and holds `knownBugs`. It contains no test behavior or matrix construction. | -| `OpenHouseMatrix.scala` | This mixes the domain traits into `object Scenarios`. The `extends` clause here is the authoritative order in which the traits' `val`s initialize, as explained in section 6. | -| `Env.scala` | This handles boot and run: the embedded OpenHouse server wiring in `OpenHouseEnv`, the embedded real HTS in `HtsEnv` and `HtsBootApp`, the retrying `Runner`, and `Main`. | - ---- - -## 5. The axes, and why the honest target is well below the naive product - -You can think of the suite as preparations crossed with localized behaviors and consumers. - -- The scenario traits define the behaviors: DML, DDL, procedures, branch operations, streaming, and - feature interactions. -- The preparation collections define the starting states: plain create and seed, RTAS replace lineage, - branch routed through `spark.wap.branch`, restored from drop on the real HTS, schema evolved, sort - ordered, and merge-on-read. -- Each family constructs one local case body per applicable preparation. The body contains the action - and assertions for that exact combination. -- The consumers answer a question: after a state-changing DDL, does each reader, such as a plain scan, - time travel, changelog, incremental, or streaming read, still work? - -The naive product is much larger than what actually runs, because a large fraction of the cells would be -vacuous, and the harness refuses to inflate its count with them. Three arguments carry most of that -reduction. First, a read or insert on a delete-free merge-on-read table is byte-identical to -copy-on-write, because there are no delete files to apply and append is mode-independent, so the real -merge-on-read surface is mutation operations crossed with merge-on-read, plus delete-file coexistence, -plus reads with live deletes, rather than the whole operation catalog crossed with merge-on-read. Second, -RTAS and branch commute with file format, because refs and metadata never touch file encoding, so those -legs run on Parquet only rather than across all three formats. Third, a DDL-by-consumer cross over a -rejected or one-shot DDL has no post-state to consume, so only state-changing DDL crossed with real -consumers is non-vacuous. - -When an estimate turns out to be inflated by vacuous cells, the honest move is to correct the estimate in -the open rather than to chase the vacuous number. File format, however, is not a vacuity axis, as the next -section explains. - -### Format multiplex, and why "format-inert" is a hypothesis rather than an assumption - -Most table-creating families iterate a preparation collection whose labels and `CREATE TABLE` recipes -already carry the format. Context-only families either create a fixed Parquet table when encoding is -irrelevant or construct one explicit case per selected format. Whether a behavior is format-independent -is something the harness verifies rather than assumes, because the fork carries patched ORC paths and -replace-path findings have exposed metadata differences. Only table-less operations have no format axis. - ---- - -## 6. Design decisions and pitfalls - -The catalog wiring is copied rather than extended. `OpenHouseEnv` composes an embedded -`OpenHouseLocalServer` together with Spark-catalog configuration lifted from `OpenHouseLocalServer` and -`TestSparkSessionUtil` as components, so no OpenHouse test class is subclassed and no existing test is -altered. The harness is a bolt-on observer. - -The undrop leg drives a real HTS through a single backward-compatible production change. A customer `DROP` -hard-codes `purge=true`, so a customer can never populate the soft-deleted store, and the embedded -server's default `HouseTableRepository` is an in-memory stub, so an undrop test against it would test the -stub rather than production. For that reason, `HARNESS_REAL_HTS=1` boots the genuine House Table Service -as a second in-JVM Spring context and points the tables server at it. The only production-code change is -one `@ConditionalOnProperty` on `HouseTablesH2Repository`, with `havingValue="true"` and -`matchIfMissing=true`, so that the stub can be switched off. The change is fully backward compatible, -because an absent property leaves the stub in place exactly as before, and everything else is on the -harness side. - -Assertions are deltas, and rejections are pins. A negative test asserts a rejection-message substring and, -following the readability audit in section 7, also asserts that the message is not a raw stacktrace, an -`[INTERNAL_ERROR]`, or a bare NullPointerException. These rejections are tripwires rather than contracts, -which means that if OpenHouse later supports the operation, the pinned test is meant to flip and be -updated rather than to keep passing silently. The goal is to catch a change in behavior in either -direction. - -The trait layout determines the initialization order. `object Scenarios`, in `OpenHouseMatrix.scala`, is -assembled by mixing the domain traits on top of `ScenarioKit` through an explicit `extends … with …` -clause, and that clause is the authoritative order. `ScenarioKit` linearizes first, so its shared `val`s -initialize before any domain trait references them, and the domain traits then initialize in the order -written. A helper used by more than one trait must live in `ScenarioKit`, because a reference to a sibling -trait's member will not resolve and the compiler will tell you. As long as the `extends` clause and the -member order within each trait stay stable, initialization stays deterministic. - -Several pitfalls are specific to this harness. The first is that only JDK 17 works, because Lombok 1.18.20 -in the repository does not compile on 21 or newer. The second is that the Gradle wrapper cannot download -behind the proxy and returns a 403, so you must use a system Gradle through `GRADLE_BIN`; the script -caches the resolved classpath after the first run. The third is that Avro required a classpath fix, -because a duplicate shaded and unshaded Iceberg on the classpath broke Avro until a dependency exclusion -was added in `scripts/print-cp.init.gradle`. The fourth is that file format is a hypothesis and the format -policy is additive: you should not optimize a block down to Parquet only on the grounds that it should be -format-inert, because that is precisely the assumption the harness exists to check, and every -table-creating block covers at least Parquet and ORC while the three-format blocks keep Avro. Adding -coverage is additive and never removes an existing format. - ---- - -## 7. What the harness found - -The following are product-behavior findings, and each is demonstrated live by named cases. - -The first group is guard gaps, where an operation that can corrupt or mislead is not blocked. - -- **G2 is that RTAS on a locked table succeeds.** The lock rejects an `UPDATE`, and then `CREATE OR - REPLACE` replaces the locked table, taking it from three rows to two, because the replace path never - reaches the lock check. This is a data-loss-class gap with the cleanest one-line fix, and it is - demonstrated by `interact.rtas.onLockedTable`. -- **G8 is that table-global DDL "on a branch" silently mutates main.** With `spark.wap.branch` set, `ADD - COLUMN`, `SET TBLPROPERTIES`, and `WRITE ORDERED BY` change main's schema, properties, and sort order, - because there is no branch dimension anywhere in the metadata commit path. It is demonstrated by - `branch.ddlLeak.*`. -- **G9 and G10 are that the replace path dodges the update-path guards.** RTAS can change the partition - spec and drop columns that `ALTER` rejects (G9), and RTAS silently wipes the `policies` plane, so that - retention, sharing, and PII column tags are gone after a replace while user properties survive (G10). - G10 is the highest-severity member of the replace-path cluster, and both are demonstrated by - `interact.rtas.*` and `hazard.rtas.wipesColumnTags`. -- **G11 is that a routine snapshot expiration destroys merge connectivity between live refs.** Expiration - retention is per-ref and head-anchored, so nothing protects the ancestry between live refs. The - consequences are all demonstrated: a `fast_forward` merge is spuriously rejected with "main is not an - ancestor" even though main never moved; a cherry-pick silently loses the expired intermediate commit, - which is a partial merge that presents as success and is the worst variant; the branch becomes - permanently unmergeable; and staged WAP snapshots are expired before publish. OpenHouse's default - three-day expiration makes all of this automatic, and it is demonstrated by - `interact.branch.expireMerge.*`. -- **G12 is that a lock starves maintenance for its whole lifetime while not stopping RTAS**, which makes - it the mirror of G2. Scheduled expiration and compaction hit the lock gate and fail every cycle, so - snapshots and files accrete unboundedly. It is demonstrated by `hazard.lock.starvesMaintenance`. -- **G3 through G7 are the lower-severity gaps**: replica-path spec divergence, free WAP and replace - toggling, ref preservation, format-version on update, and the all-or-nothing `skipEligibilityCheck` on - the replica path. G1 was investigated and then withdrawn, because the replication snapshot-walk turned - out to be sound. - -The second group is behavior and limitation findings. - -- **G13 is that CDC changelog is unsupported over a merge-on-read table after an UPDATE or MERGE**, which - fails with "Delete files are currently not supported in changelog scans". Merge-on-read delete-only and - all copy-on-write cases work, but merge-on-read update and merge, the shapes a merge-on-read table - exists to optimize, break CDC silently. This is a stock Iceberg 1.5 limitation, and it is demonstrated - by `readerWriter.changelog.{update,merge}.mor`. -- **G14 is that `rewrite_data_files` leaves a dangling position delete on a merge-on-read table.** - Compaction applies the delete, so the row set is correct, but it does not fold out the now-dangling - delete file until `rewrite_position_delete_files` runs. This is stock Iceberg 1.5, which has no - `remove-dangling-deletes` yet. It is classified as a pin rather than a bug, because the recovery path is - verified to work by `maint.mor.rewritePositionDeleteFolds` across the merge-on-read formats. The - operational takeaway is that, on merge-on-read under 1.5, you should pair `rewrite_data_files` with - `rewrite_position_delete_files`. -- **WAP1 is that a staged DELETE (with `spark.wap.id` set) is not honored by WAP and publishes to main - immediately.** In the same block, staged `INSERT`, `OVERWRITE`, `UPDATE`, and `MERGE` all stage - correctly. The consequence is that an operator relying on WAP to stage and review a deletion gets an - immediate, un-reviewed publish. It is demonstrated by `wapStaged.delete.bypassesWap`. - -There is also an error-message readability finding. A separate sweep grades rejection messages as good, -acceptable, or bad for a non-expert SQL user. The systemic result is that the client drags the entire -error body, including a stacktrace, into the message, so that even a good server sentence reaches the user -as `400 , {json + java frames}`; surfacing only `ErrorResponseBody.message` would upgrade nearly every 4xx -path at once. - -Finally, the tagged and deferred defects are the ones that appear in `Plan.knownBugs` and are reported as -`SKIP (bug: …)`. They are a nested-field DELETE optimizer NullPointerException, a RENAME COLUMN that is a -silent no-op (a genuine OpenHouse regression traced to server commit #558), and encryption that writes -plaintext because the KMS plugin is out of the repository. - -> The exhaustive ledgers behind this section include the findings with code citations, the fork-commit -> audit, the tagged-defect ledger, and the dated run log. They live alongside the harness in the pull -> request that developed it, and not necessarily in this tree. You do not need them to grok the tests, -> so reach for them only when you want the evidence behind a specific claim made here. - ---- - -## 8. The `com.linkedin.iceberg` fork - -The harness runs against fork bytecode, namely `com.linkedin.iceberg:iceberg-spark-runtime-3.5_2.12`, -rather than against Apache Iceberg. The tested behaviors are listed below. Each one is pinned by a -`fork.*` case, and each is keyed to the fork's own commit number or the upstream-Iceberg issue number. - -| Commit | The behavior the fork changes | Pinned by | -|---|---|---| -| `#249` | The partitioned default write distribution becomes NONE, where Apache uses HASH, which produces more and smaller files. | `fork.partitionDist.default` | -| `#229` | A `write.delete-file-replication` toggle is added for merge-on-read delete files. | `fork.deleteFileReplication` | -| `#219` | A per-output-file replication factor is stamped by the delete-file write path. | `fork.fileReplicationFactor` | -| `#228` | A `spark.sql.iceberg.split-size` read split-size property is added. | `fork.splitSize` | -| `#233` | Compaction bin-pack weight is computed by data-file length and ignores delete size. | `fork.binPackByLength` | -| `#189` | A budgeted rewrite is ordered by file-sequence-number. | `fork.compactionOrder` | -| `#251` | Column-default APIs and `SchemaParser` serialization are added; this exists on the branch HEAD only and is tabled. | `fork.colDefault.*` | - -The `#251` story is worth understanding, because it is a good example of the harness resisting an -overclaim. `#251` backports column defaults to the API and core, but there is no read-application code and -no Spark wiring in the open fork, because `SparkTable` does not implement `SupportsColumnDefaultValue`. As -a result, over OSS Spark, `ADD COLUMN … DEFAULT 5` parses, but the default is not written into the Iceberg -schema, old rows read NULL, and an INSERT that omits the column is rejected. The serialization does round -trip on a branch build. The harness pins exactly that: the observable OSS-Spark DDL behavior and the -serialization. It explicitly does not claim the feature is broken, because read-application may exist -in LinkedIn's private Spark, which this harness cannot see. A whole-suite branch-versus-release run, -performed through `ICEBERG_RUNTIME_JAR`, showed no correctness deltas. - ---- - -## 9. Adding a test - -Adding a test follows a short recipe. First, pick the schema, which is `CoreTable` unless the behavior -needs nested or type-edge columns. Second, select or add the smallest reusable preparation that produces -the required starting state. Third, add a `preparation.test("caseName") { table => ... }` body in the -scenario trait whose concern matches. Keep the action and every assertion in that body. Use -`table.preparedRows` and `table.preparedSnapshotCount` for the starting state, and use `table.rows`, -`table.snapshotCount`, and metadata queries for the result. - -Construct the family across each applicable preparation or format in the scenario trait, then add the -scenario-owned case list to `Plan.cases` in the intended catalog position. If the case characterizes a -deferred product bug, tag it in `Plan.knownBugs` with a reason. Run the catalog regression test to verify -the intended count and ordering change, run the narrow local slice, and then run the broader validation -gate. - ---- - -## 10. Decisions worth knowing - -File format is a per-case parameter rather than a baked-in constant, because un-baking the format is what -lets a test multiplex and compose; whether a behavior is format-inert is verified rather than assumed. The -dangling merge-on-read delete described in G14 is a pin rather than a bug, because -`rewrite_position_delete_files` is verified to recover it, and merge-on-read under 1.5 simply requires that -extra maintenance step. Encryption and KMS support is deferred, because the plugin is out of the -repository, so the plaintext behavior is pinned and the intended-behavior assertion waits for the plugin. diff --git a/integrations/spark/delta-harness/TESTING-MATRIX.md b/integrations/spark/delta-harness/TESTING-MATRIX.md deleted file mode 100644 index cabe0bf03..000000000 --- a/integrations/spark/delta-harness/TESTING-MATRIX.md +++ /dev/null @@ -1,244 +0,0 @@ -# Delta-harness testing matrix - -This document describes how the harness is organized. Every case in the suite is one point in a -cross product of independent axes, so the suite is best understood as a matrix rather than a flat -list of tests. Each scenario trait owns the complete cases for its domain. Shared -`TablePreparation` recipes create common starting states, and each -`preparation.test("caseName") { table => ... }` body keeps the action and assertions together. -`Plan.cases` preserves the catalog order by concatenating those scenario-owned case lists. - -Every prepared case receives a fresh table. The preparation creates and seeds that table, the case -body performs the behavior under test, and teardown drops the table after the body completes. A -shared preparation is therefore a reusable immutable recipe, not a table shared by multiple cases. - -## How to read a case id - -A case id has the shape ` @ `, and some families add a preparation prefix. - -| Part | Meaning | -|------|---------| -| `` | The behavior under test, for example `delete.byPredicate`, `ddl.addColumn.single`, or `merge.upsert`. | -| `@ ` | The table shape or environment the operation ran against, for example `partitioned/orc`, `mor-unpartitioned/avro`, `@ parquet`, or `@ embedded`. | -| `prep.rtas:`, `prep.ordered:`, `prep.evolved:`, `branchWap:`, `undrop:` | A prefix that names the preparation lineage the base table was taken through before the operation ran. | - -For example, `prep.ordered:update.byPredicate @ partitioned/parquet` is the `update.byPredicate` -operation, run on a partitioned Parquet table that was created with a `WRITE ORDERED BY` clause. - -## The axes - -The matrix is the product of the following axes. Not every family uses every axis, because some axes -are vacuous for some operations. A branch reference, for instance, never touches file encoding, so -branch-routed families do not multiply across all three file formats. - -| Axis | Values | Notes | -|------|--------|-------| -| Operation | The families listed below | The behavior being asserted. | -| Data file format | `parquet`, `orc`, `avro` | Applied through `write.format.default` and, for table-creating operations, through a per-case seed format. Format independence is treated as a hypothesis the harness verifies, not an assumption. | -| Partitioning | `unpartitioned`, `partitioned` | Partitioned tables partition by the `datepartition` string column. | -| Write mode | copy-on-write, merge-on-read | Merge-on-read tables set `format-version=2` and the merge-on-read delete, update, and merge modes, so mutations write position-delete files instead of rewriting data files. | -| Schema | `CoreTable`, `NestedTypesTable`, `TypesTable` | The column set the operation reads and writes. | -| Preparation lineage | base, ordered, evolved, replace (RTAS), branch, merge-on-read-deleted, undropped | How the base table was created and seeded before the operation ran. | -| Reference routing | main, WAP branch | Whether the operation was applied to the table directly or routed onto a write-audit-publish branch. | - -## Data file formats - -| Format | Explanation | -|--------|-------------| -| `parquet` | The default columnar format and the seed format when no other is set. | -| `orc` | Exercised because the fork carries patched ORC paths, so ORC coverage is not assumed to match Parquet. | -| `avro` | Exercised for the row-oriented write path on the create and merge-on-read families. | - -## Schemas and data types - -The harness pins one representative table per type concern. Column value generators are pure -functions of the row index, so a seed of N rows is reproducible. - -### CoreTable - -`CoreTable` carries one column per common primitive type plus a string date-partition column, and it -is the schema for the DML, DDL, maintenance, branching, and negative families. - -| Column | SQL type | -|--------|----------| -| `foo_col_long` | `bigint` | -| `foo_col_int` | `int` | -| `foo_col_string` | `string` | -| `foo_col_double` | `double` | -| `foo_col_boolean` | `boolean` | -| `datepartition` | `string` | - -### NestedTypesTable - -`NestedTypesTable` covers complex and nested types, and it is the schema for the nested family. - -| Column | SQL type | -|--------|----------| -| `id` | `bigint` | -| `s` | `struct` | -| `arr` | `array` | -| `m` | `map` | -| `nested` | `struct>` | - -### TypesTable - -`TypesTable` covers type-edge cases such as decimal and binary, and it is the schema for the type -family. - -| Column | SQL type | -|--------|----------| -| `id` | `bigint` | -| `n` | `int` | -| `x` | `double` | -| `dec` | `decimal(10,2)` | -| `str` | `string` | -| `bin` | `binary` | - -## Table layouts - -A layout is a labeled `CREATE TABLE` recipe. The label encodes the partitioning and format so it -reads directly in the case id. - -| Layout family | Labels | Explanation | -|---------------|--------|-------------| -| `layouts` | `{unpartitioned,partitioned}/{parquet,orc,avro}` | The six copy-on-write CoreTable shapes that back the DML, DDL, and negative families. | -| `morLayouts` | `mor-{unpartitioned,partitioned}/{parquet,orc,avro}` | The six merge-on-read CoreTable shapes that back the mutation families. | -| `morVerifyLayouts`, `cowVerifyLayouts` | `mor-verify/{format}`, `cow-verify/{format}` | Single-data-file shapes with `write.distribution-mode=none` so a subset delete is a partial-file match, which makes the physical outcome deterministic for the merge-on-read versus copy-on-write discriminator. | -| `nestedLayouts` | `nested-unpartitioned/{format}` | Unpartitioned shapes on `NestedTypesTable`. | -| `typesLayouts` | `types-unpartitioned/{format}` | Unpartitioned shapes on `TypesTable`. | - -## Preparation lineages - -Preparation determines what the base table has already been through when the operation runs. The -same preparation recipes are reused across cases so a behavior can be checked on each base without -moving its action or assertions away from the case body. - -| Lineage | Explanation | -|---------|-------------| -| base (`createAndSeed`) | Create under the layout and seed a fixed number of deterministic rows. | -| ordered (`createAndSeedOrdered`) | The base plus `ALTER TABLE ... WRITE ORDERED BY`, so the operation runs on a table with a declared sort order. | -| evolved (`createAndSeedEvolved`) | The base plus an added column, so the operation runs against a schema-evolved table. | -| replace (`createAndSeedRtas`, `createAndSeedRtasMor`) | The base is rebuilt through `CREATE OR REPLACE TABLE ... AS SELECT`, so the operation runs on a replace-lineage table. | -| branch (`createAndSeedOnBranch`) | The seed and the operation are routed onto a write-audit-publish branch, and the case also asserts that main is untouched. | -| merge-on-read-deleted (`createAndSeedMorDeleted`) | The base carries a live position delete, so read and maintenance operations must apply the delete at read time. | -| undropped (`createAndSeedUndropped`) | The base is taken through a real House Table Service soft-delete and restore. These cases run only when the embedded real House Table Service is enabled. | -| single-file (`createAndSeedSingleFile`) | The seed lands all rows in one data file, which is required by the copy-on-write versus merge-on-read physical discriminator. | - -## Operation families - -Each family owns a list of localized cases. The family constructs those cases from the applicable -preparation and format collections, while each case body contains the behavior and its assertions. -The tables below name each family and describe what it exercises. Representative operation names -are included so the family is recognizable in the case ids. - -### DML - -| Family | Explanation | -|--------|-------------| -| Reads (`read.projection`, `read.filter`, `format.materialization`) | Read-path and scan behavior, including projection, predicate pushdown, and materialization. | -| Deletes (`delete.byPredicate`, `delete.byInList`, `delete.byInSubquery`, `delete.byPartitionPredicate`, `delete.all`, `delete.truncate`, and more) | Row-level deletes across the full range of predicate shapes, including in-list, correlated and scalar subqueries, null conditions, partition predicates, and whole-table truncation. | -| Updates (`update.byPredicate`, `update.multipleColumns`, `update.byExpression`, `update.movePartition`, and more) | Row-level updates across predicate shapes, multi-column assignments, expression assignments, and partition-moving updates. | -| Merges (`merge.upsert`, `merge.insertNotMatched`, `merge.deleteMatched`, `merge.multipleMatchedClauses`, `merge.resolveByName`, and more) | `MERGE INTO` across matched and not-matched clauses, conditional clauses, upserts, source common table expressions, set operations, and by-name resolution. | -| Inserts and overwrites (`insert.into`, `insert.explicitColumns`, `append.dataFrame`, `insert.overwrite`, `insert.dynamicOverwrite`, `overwrite.dataFrame`) | The append and overwrite write paths through both SQL and the DataFrame API, including dynamic partition overwrite. | - -The mutation subset (`delete.*`, `update.*`, `merge.*`) is reused on the merge-on-read, replace, and -branch lineages, because those lineages are about the mutation write path. - -### DDL - -DDL is split into sub-families so each area of the OpenHouse table surface is exercised on its own. -Every sub-family constructs cases for the six copy-on-write layouts unless noted. - -| Sub-family | Operations | Explanation | -|------------|-----------|-------------| -| Schema evolution (`ddlSchemaCases`) | `ddl.addColumn.single`, `ddl.addColumn.multiple`, `ddl.addColumn.comment`, `ddl.addColumn.position`, `ddl.alterColumn.typeWiden`, `ddl.renameColumn` | Column additions in each position and with comments, safe type widening, and column rename. These exercise how the server validates and applies a schema change. | -| Table properties (`ddlPropertyCases`) | `ddl.props.userRoundTrip`, `ddl.props.reservedOpenhouse`, `ddl.props.formatVersionForced`, `ddl.props.previousVersionsHonored` | User property round-tripping, the handling of reserved OpenHouse properties, forced format version, and honoring previously set versions. | -| Miscellaneous (`ddlMiscellaneousCases`) | `ddl.sortOrder.orderedBy`, `ddl.sortOrder.orderedByMulti`, `ddl.renameTable`, `ddl.renameTable.conflict`, `ddl.ns.createRejected`, `ddl.ns.dropRejected` | Setting a sort order, renaming a table and the name-conflict case, and the namespace create and drop rejections. | -| Policy (`ddlPolicyCases`) | `ddl.policy.sharing`, `ddl.policy.history`, `ddl.policy.replication`, `ddl.policy.retention`, `ddl.policy.neg.historyMaxAge`, `ddl.policy.neg.historyVersions` | `SET POLICY` for sharing, history, replication, and retention, plus the negative cases where a policy bound is out of range. | -| CTAS and RTAS (`ddlCtasRtasCases`) | `ddl.ctas`, `ddl.rtas.enabled`, `ddl.rtas.disabled`, `ddl.rtas.replicationConflict` | Create-table-as-select, replace-table-as-select with replace enabled and disabled, and the replace-under-replication conflict. | -| Tagging, ACL, and features (`ddlTagAclFeatureCases`) | `ddl.colTag`, `ddl.acl.grantUnshared`, `ddl.acl.grantShared`, `ddl.featureFlag.distributionMode`, `ddl.repl.tableTypeImmutable`, `ddl.encryption.active` | Column tagging, ACL grants on shared and unshared tables, the distribution-mode feature flag, replica-table-type immutability, and the encryption-active property. | -| Encryption (`ddlEncryptionCases`) | `ddl.encryption` | The encryption capability, pinned on Parquet. | - -The schema-evolution operations are also crossed with every layout as a separate `ddlSchema` block, -and there is a DDL-then-consumer battery (`ddlConsumeBattery`) that applies each state-changing DDL -and then runs each consumer to confirm the table still reads and writes. - -### Maintenance - -| Operations | Explanation | -|-----------|-------------| -| `maintenance.expireSnapshots`, `maintenance.rewriteDataFiles`, `maintenance.removeOrphanFiles` | The table-maintenance procedures, including the locked variants, crossed with both file formats. | - -### Merge-on-read verification - -| Family | Explanation | -|--------|-------------| -| `mor.writesDeleteFiles`, `cow.writesNoDeleteFiles` | The physical discriminator that proves merge-on-read wrote a position delete and copy-on-write did not. | -| merge-on-read read (`prep.morRead`), coexistence (`morCoexist`), maintenance fold and meta, hazards, and branch merge | Reads and maintenance over a table that already carries a live position delete, and the survival of position deletes across time travel, rollback, expiration, and branch merges. | - -### Branching and write-audit-publish - -| Family | Explanation | -|--------|-------------| -| `branching` | Branch creation and the basic branch operations. | -| `branchWap:` blocks | The DML catalog routed onto a branch, asserting both the branch delta and that main stays isolated. | -| `branchDdl`, `wapStaged` | The DDL-on-branch axis and the staged-then-publish write-audit-publish flow. | - -### Interactions, surface, and hazards - -| Family | Explanation | -|--------|-------------| -| `interactions` | Cross-feature cases where one feature is exercised in the presence of another. | -| `surface` | The read and write surface, including streaming read and write and the plaintext data pin. | -| `hazards`, `readerWriter` | Reader and writer hazard scenarios, such as a streaming checkpoint crossed with snapshot expiration, change-data-capture over an expired range, and replace-table-as-select wiping column tags. | - -### Nested and type-edge coverage - -| Family | Explanation | -|--------|-------------| -| nested (`nestedOperations` on `NestedTypesTable`) | Operations over struct, array, map, and doubly-nested struct columns. | -| types (`typesOperations` on `TypesTable`) | Operations over type-edge columns such as decimal and binary. | - -### Time travel, restore, and rollback - -| Family | Explanation | -|--------|-------------| -| `timeTravel` | Reads at an earlier snapshot, crossed with both file formats. | -| `restoreRollback` | `RESTORE` and rollback to an earlier snapshot, crossed with both file formats. | - -### Fork behavior pins - -These families pin behaviors specific to the `com.linkedin.iceberg` fork the harness runs against. -They characterize the fork surface at the API and table-property level. - -| Family | Explanation | -|--------|-------------| -| `forkColDefault` | Column-default serialization through `SchemaParser`. | -| `forkPartitionDist` | Partition distribution behavior. | -| `forkDeleteFileReplication`, `forkFileReplicationFactor` | Delete-file replication and the output-file replication factor. | -| `forkSplitSize`, `forkBinPackByLength`, `forkCompactionOrder` | Split size, bin-pack by length, and compaction ordering. | - -### Negatives - -| Operations | Explanation | -|-----------|-------------| -| `negative.nonExistentColumn`, `negative.nonDeterministicDelete`, `negative.nonDeterministicUpdate`, `negative.insertArity`, `negative.mergeConflictingUpdates`, `negative.mergeCardinalityViolation`, `negative.partitionByNonExistent` | Cases that must be rejected. Each asserts that the operation fails, so a silent acceptance is itself a failure. | - -### Control plane - -| Family | Explanation | -|--------|-------------| -| `control`, `undropAdmin`, `undropInteract` | Control-plane cases such as lock and unlock, and the soft-delete, list, restore, and purge lifecycle. The undrop lifecycle cases run only when the embedded real House Table Service is enabled, and are otherwise empty. | - -## How assertions are framed - -Each `PreparedTable` exposes the rows and snapshot count produced by its preparation, plus methods -that read the live rows and snapshot count after the action. Cases use those values to assert the -relevant row, commit, metadata, or error outcome. Delta assertions remain useful when the expected -behavior is relative to the starting state, while cases with a fixed contract can state that -contract directly in the same body. - -## Known bugs - -A genuine product or upstream bug is tagged in `Plan.knownBugs` by a substring of the case id, along -with a prose explanation. A tagged case is reported as skipped with its reason rather than failing -the suite, which keeps the suite green while keeping the defect visible and documented. From e83da3b0d00351d6a5669cbc2d750d4f5d72dad0 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Wed, 26 Aug 2026 12:57:19 -0700 Subject: [PATCH 06/24] refactor(delta-harness): make tests readable Separate table preparations from DML operations so each case shows its starting state, mutation, and relative assertions in one place. Keep feature-owned scenarios in removable RTAS, merge-on-read, and branch layers while preserving the exact ordered 2,572-case catalog. Run the same published sources through the local Gradle task and the acceptance adapter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- build.gradle | 15 +- integrations/spark/delta-harness/build.gradle | 70 +- .../spark/delta-harness/run-openhouse.sh | 114 +- .../scripts/print-cp.init.gradle | 37 - .../openhouse/BranchDmlScenarios.scala | 54 + .../openhouse/BranchHazardScenarios.scala | 148 ++ .../BranchInteractionScenarios.scala | 453 ++++ .../openhouse/BranchMorScenarios.scala | 163 ++ .../harness/openhouse/BranchScenarioKit.scala | 80 + .../openhouse/BranchSurfaceScenarios.scala | 422 ++++ .../openhouse/BranchWapScenarios.scala | 211 +- .../harness/openhouse/DmlScenarios.scala | 1943 ++++++++++------- .../main/scala/harness/openhouse/Env.scala | 311 ++- .../harness/openhouse/ForkScenarios.scala | 407 ++-- .../scala/harness/openhouse/Framework.scala | 211 +- .../HazardReaderWriterScenarios.scala | 932 +++----- .../ImplementationPinScenarios.scala | 58 + .../openhouse/InteractionScenarios.scala | 796 +------ .../openhouse/MaintControlScenarios.scala | 154 +- .../harness/openhouse/MorDmlScenarios.scala | 107 + .../harness/openhouse/MorForkScenarios.scala | 72 + .../openhouse/MorInteractionScenarios.scala | 59 + .../harness/openhouse/MorMaintScenarios.scala | 343 +-- .../openhouse/MorReaderWriterScenarios.scala | 194 ++ .../harness/openhouse/MorScenarioKit.scala | 137 ++ .../openhouse/MorSurfaceScenarios.scala | 76 + .../openhouse/NegativeDdlScenarios.scala | 299 +-- .../openhouse/NestedTypesScenarios.scala | 185 +- .../harness/openhouse/OpenHouseMatrix.scala | 29 +- .../main/scala/harness/openhouse/Plan.scala | 176 +- .../harness/openhouse/RtasDdlScenarios.scala | 76 + .../harness/openhouse/RtasDmlScenarios.scala | 16 + .../openhouse/RtasHazardScenarios.scala | 57 + .../openhouse/RtasInteractionScenarios.scala | 390 ++++ .../harness/openhouse/RtasScenarioKit.scala | 56 + .../openhouse/RtasSurfaceScenarios.scala | 120 + .../scala/harness/openhouse/ScenarioKit.scala | 429 ++-- .../harness/openhouse/SurfaceScenarios.scala | 798 ++----- .../harness/BranchDmlCaseCatalogTest.scala | 83 + .../test/scala/harness/CaseCatalogTest.scala | 10 +- .../scala/harness/DmlCaseCatalogTest.scala | 238 ++ .../scala/harness/MorDmlCaseCatalogTest.scala | 96 + .../harness/RtasDmlCaseCatalogTest.scala | 72 + .../scala/harness/TablePreparationTest.scala | 33 +- 44 files changed, 6490 insertions(+), 4240 deletions(-) delete mode 100644 integrations/spark/delta-harness/scripts/print-cp.init.gradle create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchDmlScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchHazardScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchInteractionScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchMorScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchScenarioKit.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchSurfaceScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorDmlScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorForkScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorInteractionScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorReaderWriterScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorScenarioKit.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorSurfaceScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDdlScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDmlScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasHazardScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasInteractionScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasScenarioKit.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasSurfaceScenarios.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/BranchDmlCaseCatalogTest.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/MorDmlCaseCatalogTest.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/RtasDmlCaseCatalogTest.scala diff --git a/build.gradle b/build.gradle index 1f4cbbb0b..85eb88b4f 100644 --- a/build.gradle +++ b/build.gradle @@ -153,19 +153,22 @@ allprojects { } afterEvaluate { - for (def task in it.tasks) { - if (task != rootProject.tasks.CopyGitHooksTask) { + it.tasks + .findAll { task -> task != rootProject.tasks.CopyGitHooksTask } + .each { task -> task.dependsOn rootProject.tasks.CopyGitHooksTask } - } } } -// Local Git Hooks cannot be shared, as .git directory is gitignore'd. +def gitHooksDirectory = providers.exec { + commandLine 'git', 'rev-parse', '--git-path', 'hooks' +}.standardOutput.asText.map { it.trim() } + tasks.register('CopyGitHooksTask', Copy) { - println 'Make the git hook available in .git/hooks directory.' + println 'Make the Git hooks available in the repository hooks directory.' from file('scripts/git-hooks') - into file('.git/hooks/') + into gitHooksDirectory } // ============================================================================= diff --git a/integrations/spark/delta-harness/build.gradle b/integrations/spark/delta-harness/build.gradle index cf1da2f16..55cc11717 100644 --- a/integrations/spark/delta-harness/build.gradle +++ b/integrations/spark/delta-harness/build.gradle @@ -4,15 +4,12 @@ plugins { id 'scala' } -// The delta-harness behavioral matrix, published as a portable Scala library so it can be authored -// and run ONCE and consumed in two homes: -// 1. LOCALLY in this repo against the embedded catalog (see Env.scala + run-openhouse.sh), and -// 2. As an acceptance test in a downstream environment against a real cluster, which depends on this -// published artifact and supplies only its own environment adapter. +// The delta-harness behavioral matrix is published as a portable Scala library. This repository runs +// it against the embedded catalog, and the LinkedIn acceptance tests supply a remote environment. // // Only the portable scenario/framework sources are published. Env.scala boots the embedded OpenHouse -// server + House Table Service and pulls in Spring/housetables/tables-test-fixtures, so it is EXCLUDED -// from the published library and compiled separately for the local run (it depends on this library). +// server and pulls in its test fixtures, so it is excluded from the published library and compiled in +// the local source set. ext { icebergVersion = rootProject.ext.iceberg_1_5_version @@ -24,17 +21,25 @@ sourceSets { main { scala { srcDirs = ['src/main/scala'] - // Embedded-only boot/run wiring — not part of the portable, publishable library. + // Embedded-only boot and run wiring is not part of the portable library. exclude 'harness/openhouse/Env.scala' } } + local { + scala { + srcDirs = ['src/main/scala'] + include 'harness/openhouse/Env.scala' + } + compileClasspath += sourceSets.main.output + runtimeClasspath += sourceSets.main.output + } } dependencies { implementation "org.scala-lang:scala-library:${scalaLibVersion}" - // Compile-only: the consumer (Env.scala locally, or a downstream environment adapter) provides the - // actual Spark, Iceberg and OpenHouse client runtime. The library jar carries only the harness classes. + // The consumer provides the Spark, Iceberg, and OpenHouse runtime. The library jar carries only the + // harness classes. compileOnly("org.apache.spark:spark-sql_2.12:${sparkVersion}") { exclude group: 'io.netty' } @@ -53,6 +58,51 @@ dependencies { exclude group: 'io.netty' } testImplementation(project(path: ':integrations:spark:spark-3.5:openhouse-spark-3.5-runtime_2.12', configuration: 'shadow')) + + localImplementation sourceSets.main.output + localImplementation "org.scala-lang:scala-library:${scalaLibVersion}" + localImplementation("org.apache.spark:spark-sql_2.12:${sparkVersion}") { + exclude group: 'io.netty' + } + localImplementation(project(path: ':integrations:spark:spark-3.5:openhouse-spark-3.5-runtime_2.12', configuration: 'shadow')) { + exclude group: 'org.apache.commons', module: 'commons-lang3' + } + localImplementation project(':tables-test-fixtures:tables-test-fixtures-iceberg-1.5_2.12') +} + +configurations.localRuntimeClasspath { + exclude group: 'com.linkedin.iceberg', module: 'iceberg-core' + exclude group: 'com.linkedin.iceberg', module: 'iceberg-api' + exclude group: 'com.linkedin.iceberg', module: 'iceberg-common' + exclude group: 'com.linkedin.iceberg', module: 'iceberg-data' + resolutionStrategy.force( + 'com.fasterxml.jackson.core:jackson-annotations:2.15.2', + 'com.fasterxml.jackson.core:jackson-core:2.15.2', + 'com.fasterxml.jackson.core:jackson-databind:2.15.2', + 'com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.15.2', + 'com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.15.2', + 'com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.15.2', + 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.15.2', + 'com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.15.2', + 'com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.15.2', + 'com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.15.2', + 'com.fasterxml.jackson.module:jackson-module-parameter-names:2.15.2', + 'com.fasterxml.jackson.module:jackson-module-scala_2.12:2.15.2') +} + +tasks.register('runOpenHouse', JavaExec) { + group = 'verification' + description = 'Runs the delta harness against an embedded OpenHouse catalog.' + dependsOn localClasses + classpath = sourceSets.local.runtimeClasspath + mainClass = 'harness.Main' + if (JavaVersion.current() >= JavaVersion.VERSION_1_9) { + jvmArgs( + '--add-opens=java.base/java.nio=ALL-UNNAMED', + '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED', + '--add-opens=java.base/sun.util.calendar=ALL-UNNAMED', + '--add-exports=java.base/sun.util.calendar=ALL-UNNAMED') + } } jar.enabled = true diff --git a/integrations/spark/delta-harness/run-openhouse.sh b/integrations/spark/delta-harness/run-openhouse.sh index 6432e98a5..6cda85599 100755 --- a/integrations/spark/delta-harness/run-openhouse.sh +++ b/integrations/spark/delta-harness/run-openhouse.sh @@ -1,102 +1,28 @@ #!/usr/bin/env bash -# Build + run the delta-harness DELETE slice against the REAL OpenHouse catalog -# (embedded OpenHouseLocalServer + OpenHouseCatalog). -# -# Requirements: -# - JDK 17 (the OpenHouse build pins Lombok 1.18.20, which is incompatible with JDK 21+). -# Set JAVA17_HOME, or the script uses $JAVA_HOME if it is a 17. -# - A Gradle able to build the repo (system gradle 8.x works; the pinned 7.6.2 wrapper -# may be blocked from downloading in restricted networks). -# - Scala 2.12.18 compiler jars in the local Maven cache (~/.m2), or adjust SCALAC_CP. -# -# Real-HTS mode (HARNESS_REAL_HTS=1): boots the REAL embedded House Table Service as a 2nd Spring -# context and points the tables server at it (replacing the in-memory stub), and enables the undrop -# preparation axis + undropAdmin lifecycle cases (soft-delete/restore/purge). Requires the housetables -# classes on the classpath — run once with FORCE_CP=1 after adding them (print-cp.init.gradle already -# pulls :services:housetables). See HTS-EMBED-PLAN.md / HTS-EMBED-IMPL.md. Default (unset) uses the stub. set -euo pipefail -cd "$(dirname "$0")" -REPO_ROOT="$(cd ../../.. && pwd)" -HERE="$(pwd)" -WORK="${TMPDIR:-/tmp}/delta-harness-oh" -mkdir -p "$WORK" -JDK17="${JAVA17_HOME:-${JAVA_HOME:?set JAVA17_HOME to a JDK 17}}" -GRADLE="${GRADLE_BIN:-gradle}" -M2="${HOME}/.m2/repository/org/scala-lang" -SCALAC_CP="$M2/scala-compiler/2.12.18/scala-compiler-2.12.18.jar:$M2/scala-reflect/2.12.18/scala-reflect-2.12.18.jar:$M2/scala-library/2.12.18/scala-library-2.12.18.jar" - -# Classpath resolution is the slow part (~82s of gradle). It only changes when OpenHouse deps -# change, so we cache it in $WORK/oh-cp.txt and reuse it for fast inner-loop iteration. Force a -# fresh resolve with FORCE_CP=1 (do this after pulling dep changes or the first run in a session). -if [[ "${FORCE_CP:-0}" != "1" && -s "$WORK/oh-cp.txt" ]]; then - echo ">> reusing cached OpenHouse classpath ($WORK/oh-cp.txt) — set FORCE_CP=1 to re-resolve" -else - echo ">> resolving OpenHouse itest runtime classpath (builds the runtime uber jar + fixtures)" - ( cd "$REPO_ROOT" && "$GRADLE" -Dorg.gradle.java.home="$JDK17" -DcpOut="$WORK/oh-cp.txt" \ - --init-script "$HERE/scripts/print-cp.init.gradle" \ - :integrations:spark:spark-3.5:openhouse-spark-3.5-itest:printHarnessCp \ - -x CopyGitHooksTask --console=plain ) +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)" +JAVA_HOME="${JAVA17_HOME:-${JAVA_HOME:-}}" +if [[ -z "$JAVA_HOME" ]]; then + echo "Set JAVA17_HOME or JAVA_HOME to a JDK 17 installation." >&2 + exit 2 fi -OHCP="$(cat "$WORK/oh-cp.txt")" +export JAVA_HOME -# ── Test-the-BRANCH override ──────────────────────────────────────────────────────────────────── -# The harness normally resolves the PUBLISHED com.linkedin.iceberg:iceberg-spark-runtime-3.5_2.12 -# (e.g. 1.5.2.15) — a Maven-Central snapshot that can LAG the openhouse-1.5.2 branch HEAD (it predates -# #251 column-defaults, etc.). To test the actual BRANCH, build the shaded runtime jar from branch HEAD -# (`gradle :iceberg-spark:iceberg-spark-runtime-3.5_2.12:shadowJar`) and point this at it: -# ICEBERG_RUNTIME_JAR=/workspace/iceberg/spark/v3.5/spark-runtime/build/libs/ ./run-openhouse.sh -# That single shaded jar carries all of iceberg api+core+spark, so swapping it makes the whole harness -# JVM (Spark side + embedded server) run the branch. Unset → back to the published release. Reversible. -if [[ -n "${ICEBERG_RUNTIME_JAR:-}" ]]; then - [[ -f "$ICEBERG_RUNTIME_JAR" ]] || { echo "!! ICEBERG_RUNTIME_JAR not found: $ICEBERG_RUNTIME_JAR" >&2; exit 1; } - # How many spark-runtime-3.5 entries does the resolved cp actually have? If zero, the pattern no longer - # matches (module/version rename, jar absent) and swapping would SILENTLY leave the published jar in place - # — so fail loudly instead of pretending we tested the branch. - matches="$(printf '%s' "$OHCP" | tr ':' '\n' | grep -cE '/iceberg-spark-runtime-3\.5_2\.12-[^/]*\.jar' || true)" - if [[ "$matches" -eq 0 ]]; then - echo "!! ICEBERG_RUNTIME_JAR set but no iceberg-spark-runtime-3.5_2.12 jar found on the resolved classpath" >&2 - echo "!! (pattern changed, or cp cache is stale — re-run with FORCE_CP=1). Refusing to run the PUBLISHED jar." >&2 - exit 1 - fi - # Replace the resolved spark-runtime-3.5 jar path (any version) with the override. Use a `|` sed delimiter - # and a literal-ized replacement so `&`/`#`/`/` in the path are not interpreted. - repl="$(printf '%s' "$ICEBERG_RUNTIME_JAR" | sed -e 's/[&|\\]/\\&/g')" - OHCP="$(printf '%s' "$OHCP" | tr ':' '\n' \ - | sed -E "s|.*/iceberg-spark-runtime-3\.5_2\.12-[^/]*\.jar|$repl|" \ - | paste -sd ':' -)" - inserted="$(printf '%s' "$OHCP" | tr ':' '\n' | grep -Fc "$ICEBERG_RUNTIME_JAR" || true)" - [[ "$inserted" -ge 1 ]] || { echo "!! branch-mode swap produced 0 override entries — aborting" >&2; exit 1; } - echo ">> [BRANCH MODE] iceberg-spark-runtime swapped ($matches slot(s)) -> $ICEBERG_RUNTIME_JAR" - echo ">> [BRANCH MODE] override entries on cp: $inserted" +JAVA_VERSION="$("$JAVA_HOME/bin/java" -version 2>&1 | sed -n '1s/.*version "\([0-9][0-9]*\).*/\1/p')" +if [[ "$JAVA_VERSION" != "17" ]]; then + echo "The delta harness requires JDK 17; $JAVA_HOME reports Java $JAVA_VERSION." >&2 + exit 2 fi -echo ">> compiling harness (scala 2.12) against the OpenHouse classpath" -mkdir -p "$WORK/classes" -# The harness is split across several .scala files (Framework / Scenario traits / Plan / Env), -# all in `package harness`. Compile every source under src/main/scala together so cross-file -# references resolve (order is irrelevant to scalac — it compiles the whole compilation unit set). -mapfile -t SCALA_SRCS < <(find "$HERE/src/main/scala/harness/openhouse" -name '*.scala' | sort) -echo ">> ${#SCALA_SRCS[@]} source files" -"$JDK17/bin/java" -cp "$SCALAC_CP" scala.tools.nsc.Main \ - -classpath "$OHCP" -d "$WORK/classes" \ - "${SCALA_SRCS[@]}" +cd "$REPO_ROOT" +if (( $# == 0 )); then + exec ./gradlew --no-daemon \ + :integrations:spark:openhouse-spark-delta-harness_2.12:runOpenHouse +fi -echo ">> running on JDK 17 (embedded OpenHouse server + OpenHouse catalog)" -OPENS=( - --add-opens=java.base/java.lang=ALL-UNNAMED - --add-opens=java.base/java.lang.invoke=ALL-UNNAMED - --add-opens=java.base/java.io=ALL-UNNAMED - --add-opens=java.base/java.net=ALL-UNNAMED - --add-opens=java.base/java.nio=ALL-UNNAMED - --add-opens=java.base/java.util=ALL-UNNAMED - --add-opens=java.base/java.util.concurrent=ALL-UNNAMED - --add-opens=java.base/sun.nio.ch=ALL-UNNAMED - --add-opens=java.base/sun.security.action=ALL-UNNAMED - --add-opens=java.base/sun.util.calendar=ALL-UNNAMED -) -SCALA_LIB="$M2/scala-library/2.12.18/scala-library-2.12.18.jar" -# Args are passed through as case-id filters (AND). E.g. `run-openhouse.sh delete parquet` -# runs just the delete tests on parquet — a ~25s inner loop. No args runs the full matrix. -exec "$JDK17/bin/java" "${OPENS[@]}" -Dio.netty.tryReflectionSetAccessible=true \ - -cp "$WORK/classes:$SCALA_LIB:$OHCP" harness.Main "$@" +printf -v FILTERS ' %q' "$@" +exec ./gradlew --no-daemon \ + :integrations:spark:openhouse-spark-delta-harness_2.12:runOpenHouse \ + --args="${FILTERS# }" diff --git a/integrations/spark/delta-harness/scripts/print-cp.init.gradle b/integrations/spark/delta-harness/scripts/print-cp.init.gradle deleted file mode 100644 index c876c2dc4..000000000 --- a/integrations/spark/delta-harness/scripts/print-cp.init.gradle +++ /dev/null @@ -1,37 +0,0 @@ -// Resolves the harness runtime classpath from the OpenHouse spark itest module and writes it to -// the file named by -DcpOut. -// -// The itest classpath legitimately pulls two copies of Iceberg into one JVM: the shaded -// iceberg-spark-runtime fat jar (client side) and the unshaded iceberg-{api,common,core,data} -// jars (embedded server side). On the Avro data path those two Avro namespaces collide -// (ClassCastException, see FINDINGS.md F1). We resolve that the proper way — a dependency -// exclusion so the graph carries a single Iceberg — rather than filtering resolved jars by hand. -// The shaded fat jar provides all org.apache.iceberg.* classes, so excluding the unshaded modules -// is safe. This exclusion applies only to this classpath-extraction invocation. -allprojects { - if (path == ':integrations:spark:spark-3.5:openhouse-spark-3.5-itest') { - afterEvaluate { - configurations.testRuntimeClasspath { - exclude group: 'com.linkedin.iceberg', module: 'iceberg-core' - exclude group: 'com.linkedin.iceberg', module: 'iceberg-api' - exclude group: 'com.linkedin.iceberg', module: 'iceberg-common' - exclude group: 'com.linkedin.iceberg', module: 'iceberg-data' - } - // HTS-embed (Option A): pull the REAL House Table Service classes - // (UserTablesServiceImpl, controllers, JDBC repos, api-spec model) onto the harness - // classpath so the harness can boot a real HTS as a 2nd Spring context and point the - // embedded tables server's HouseTableRepositoryImpl at it. Only needed for the real-HTS - // mode; the default stub path does not use these classes. Same single-Iceberg exclusion - // above covers housetables' transitive unshaded iceberg. - dependencies.add('testImplementation', project(':services:housetables')) - } - tasks.register('printHarnessCp') { - dependsOn configurations.testRuntimeClasspath - doLast { - def cp = configurations.testRuntimeClasspath.resolve().collect { it.absolutePath } - new File(System.getProperty('cpOut')).text = cp.join(':') - println "WROTE ${cp.size()} classpath entries" - } - } - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchDmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchDmlScenarios.scala new file mode 100644 index 000000000..a8bfc9796 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchDmlScenarios.scala @@ -0,0 +1,54 @@ +package harness + +// The branch DML buckets. Each bucket is a branch-routed preparation list crossed with one of the +// shared DML test-case lists that DmlScenarios names. A case captures its before state from the +// branch it is routed at, so the same body holds on a branch and on main, and the preparation's own +// isolation check confirms main kept its three seed rows. +trait BranchDmlScenarios extends BranchScenarioKit { this: DmlScenarios => + + lazy val branchDmlCases: List[Plan.Case] = + preparedBranchCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) ++ + preparedNullStringBranchCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) + + lazy val branchPartitionedDmlCases: List[Plan.Case] = + preparedPartitionedBranchCoreTables.flatMap(preparation => + partitionedTableTestCases.map(_.runOn(preparation))) + + lazy val branchMorDmlCases: List[Plan.Case] = + preparedBranchMorCoreTables.flatMap(preparation => + rowMutationTestCases.map(_.runOn(preparation))) ++ + preparedNullStringBranchMorCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) + + // The branch a consumer creates after the DDL. It runs against each of the standard DDL-consumer + // preparations, so Plan places it inside that walk. + def branchDdlConsumerCases( + preparation: TablePreparation[CoreTable.type]): List[Plan.Case] = + List( + preparation.test( + "branch", + "A write to a branch created after the DDL takes the branch to four rows and leaves " + + "main on its three rows.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH cb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_cb " + + s"SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") + .collect()(0) + .getLong(0) == 4, + "branch write failed after DDL") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 3, + "branch write changed the main table") + }) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchHazardScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchHazardScenarios.scala new file mode 100644 index 000000000..8a11bd8d8 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchHazardScenarios.scala @@ -0,0 +1,148 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// The branch hazard families. Each case creates a named branch and then runs an operation that could +// disturb it: a retention policy, a table rename, or turning write.wap.enabled off and on again. The +// cases run on parquet and orc. +trait BranchHazardScenarios extends BranchScenarioKit { this: HazardReaderWriterScenarios => + import Rows._ + + def hazardBranchCases(format: String): List[Plan.Case] = { + val partitionedPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"PARTITIONED BY (${Core.datePartition.columnName}) " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)(), + description = s"Three seed rows in a $format table partitioned by datepartition.") + val twoSnapshotPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => cowCreate(table, format))() + .insert(3)() + .sql("insertMore")(table => + s"INSERT INTO $table VALUES " + + "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + + "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")(), + description = s"Five seed rows across two snapshots in a copy-on-write $format table.") + val wapPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => cowCreate(table, format))() + .insert(3)() + .sql("enableWap")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")(), + description = s"Three seed rows in a $format table with write.wap.enabled set to true.") + + List( + partitionedPreparation.test( + "hazard.retentionBranch.defended", + "After a branch is created, main is trimmed by DELETE, and snapshot expiration plus " + + "orphan-file removal run, the branch still reads its 3 rows and main reflects the " + + "trimmed row count.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH rbb") + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} <= 2") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + table.spark.sql( + "CALL openhouse.system.remove_orphan_files(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2020-01-01 00:00:00')") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rbb'") == "3", + "branch should remain readable after retention cleanup") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "1", + "main should reflect the retention-shaped delete") + }, + twoSnapshotPreparation.test( + "hazard.rename.consumers", + "After ALTER TABLE RENAME, both a pre-existing branch and pre-existing time travel to an " + + "old snapshot remain readable under the new name, and the renamed table still accepts " + + "writes.") { table => + val snapshots = snapshotIds(table.spark, table.name) + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH rnb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_rnb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val renamedTable = s"${table.name}_rn" + table.spark.sql( + s"ALTER TABLE ${table.name} RENAME TO $renamedTable") + try { + assert( + countOf( + table.spark, + s"SELECT count(*) FROM $renamedTable " + + "VERSION AS OF 'rnb'") == "6", + "branch should survive table rename") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM $renamedTable " + + s"VERSION AS OF ${snapshots.head}") == "3", + "time travel should survive table rename") + + table.spark.sql( + s"INSERT INTO $renamedTable VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM $renamedTable") == "6", + "renamed table should remain writable") + } finally { + table.spark.sql( + s"ALTER TABLE $renamedTable RENAME TO ${table.name}") + } + }, + wapPreparation.test( + "hazard.wapToggle.branchesSurvive", + "A named branch keeps accumulating its own rows across write.wap.enabled being turned " + + "off, while main stays at its original 3 rows.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH wtb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_wtb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='false')") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_wtb VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'wtb'") == "5", + "named branch should survive disabling WAP") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "branch writes should leave main unchanged") + }) + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchInteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchInteractionScenarios.scala new file mode 100644 index 000000000..44174a98c --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchInteractionScenarios.scala @@ -0,0 +1,453 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// The branch interaction family. Each case composes a branch or a write-audit-publish staged commit +// with another table state or another operation, so the cases show how branch routing behaves +// alongside DDL, snapshot references and maintenance. The cases run on parquet and orc. +trait BranchInteractionScenarios extends BranchScenarioKit { + import Rows._ + + def interactionBranchCases(format: String): List[Plan.Case] = { + val basePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)(), + description = s"Three seed rows in a $format table.") + val twoSnapshotPreparation = TablePreparation( + format, + coreTwoSnapshots(format), + description = s"Five seed rows across two snapshots in a $format table.") + val wapPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableWap")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")(), + description = s"Three seed rows in a $format table with write.wap.enabled set to true.") + + List( + twoSnapshotPreparation.test( + "interact.branch.ttBeforeBranchPoint", + "After branching and writing to the branch, a snapshot ID or timestamp from before the " + + "branch point still resolves to the pre-branch 3 rows, both on main and while " + + "spark.wap.branch selects the branch.") { table => + val snapshots = snapshotIds(table.spark, table.name) + val firstCommitTimestamp = table.spark + .sql( + s"SELECT CAST(committed_at AS STRING) FROM ${table.name}.snapshots " + + "ORDER BY committed_at LIMIT 1") + .collect()(0) + .getString(0) + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH tb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_tb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'tb'") + .collect()(0) + .getLong(0) == 6, + "branch head should contain 6 rows") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF ${snapshots.head}") + .collect()(0) + .getLong(0) == 3, + "snapshot ID should resolve before the branch point") + + table.spark.conf.set("spark.wap.branch", "tb") + try { + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"TIMESTAMP AS OF '$firstCommitTimestamp'") + .collect()(0) + .getLong(0) == 3, + "explicit timestamp should override spark.wap.branch") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF ${snapshots.head}") + .collect()(0) + .getLong(0) == 3, + "explicit snapshot ID should override spark.wap.branch") + } finally { + table.spark.conf.unset("spark.wap.branch") + } + }, + basePreparation.test( + "interact.branch.mainDdlImmediate", + "ALTER TABLE ADD COLUMN changes the schema seen from a branch immediately, an old-arity " + + "insert into the branch fails afterward, and a new-arity insert matching the added " + + "column succeeds.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_mb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + val branchColumns = table.spark + .sql( + s"SELECT * FROM ${table.name} VERSION AS OF 'mb' LIMIT 1") + .columns + .toSeq + + assert( + branchColumns.contains("extra_col"), + s"main DDL should change the table-global schema: $branchColumns") + + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"INSERT INTO ${table.name}.branch_mb VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')")) + assert( + exception.getMessage.toLowerCase.contains("not enough data columns"), + "old-arity branch writer should fail after main DDL") + + table.spark.sql( + s"INSERT INTO ${table.name}.branch_mb VALUES " + + "(CAST(8 AS BIGINT), 8, 'row-8', 8.5, true, " + + "'2024-01-08-07', 44)") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") + .collect()(0) + .getLong(0) == 5, + "new-arity branch write should succeed after main DDL") + }, + twoSnapshotPreparation.test( + "interact.branch.expireProtectsRefs", + "Snapshot expiration after writes on both main and a branch keeps both ref heads, drops " + + "the intermediate snapshots, and leaves both main and the branch fully readable.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH eb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_eb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}.snapshots") + .collect()(0) + .getLong(0) == 4, + "expected four snapshots before expiration") + + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + val refs = table.spark + .sql(s"SELECT name FROM ${table.name}.refs") + .collect() + .map(_.getString(0)) + .toSet + val snapshotCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.snapshots") + .collect()(0) + .getLong(0) + val branchRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'eb'") + .collect()(0) + .getLong(0) + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert(refs == Set("main", "eb"), s"refs changed: $refs") + assert( + snapshotCount == 2, + s"expiration should retain two ref heads, got $snapshotCount") + assert( + branchRowCount == 6, + s"branch should remain readable with 6 rows, got $branchRowCount") + assert( + mainRowCount == 6, + s"main should remain readable with 6 rows, got $mainRowCount") + }, + twoSnapshotPreparation.test( + "interact.branch.rollbackWhileWapConf", + "Calling rollback_to_snapshot while spark.wap.branch selects a branch still rolls back " + + "main, leaving the branch's own rows unaffected.") { table => + val firstSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH rb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_rb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.conf.set("spark.wap.branch", "rb") + try { + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $firstSnapshotId)") + } finally { + table.spark.conf.unset("spark.wap.branch") + } + val mainRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val branchRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb'") + .collect()(0) + .getLong(0) + + assert( + mainRowCount == 3, + s"rollback should target main and restore 3 rows, got $mainRowCount") + assert( + branchRowCount == 6, + s"rollback should leave branch at 6 rows, got $branchRowCount") + }, + twoSnapshotPreparation.test( + "interact.restore.expireAfterRollback", + "After rolling back to the first snapshot, expiring snapshots removes the rolled-past " + + "snapshot and keeps the current 3 rows readable, but time travel to that expired " + + "snapshot now fails.") { table => + val snapshots = snapshotIds(table.spark, table.name) + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', ${snapshots.head})") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + val snapshotCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.snapshots") + .collect()(0) + .getLong(0) + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + snapshotCount == 1, + s"rolled-past snapshot should expire, got $snapshotCount snapshots") + assert( + rowCount == 3, + s"rollback should preserve 3 current rows, got $rowCount") + + val exception = Check.intercept[Exception]( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF ${snapshots(1)}") + .collect()) + assert( + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage) + .exists(_.toLowerCase.contains("snapshot"))), + "time travel to the expired rolled-past snapshot should fail") + }, + basePreparation.test( + "interact.branch.expireMerge.spuriousReject", + "Expiring snapshots after two writes to a branch removes the intermediate branch " + + "snapshot but keeps the branch fully readable; fast_forward onto that punctured " + + "ancestry is rejected, and main stays consistent whether or not a cherry-pick recovery " + + "succeeds.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_mb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_mb VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots") == "3", + "expected parent and two branch snapshots") + + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots") == "2", + "expiration should remove the intermediate branch snapshot") + val refs = table.spark + .sql(s"SELECT name FROM ${table.name}.refs") + .collect() + .map(_.getString(0)) + .toSet + assert(refs == Set("main", "mb"), s"refs changed: $refs") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") == "5", + "branch should remain readable after expiration") + + val exception = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.fast_forward(" + + s"'${catalogRelative(table.name)}', 'main', 'mb')")) + assert( + Option(exception.getMessage).exists(_.contains("not an ancestor")), + "fast_forward should reject the punctured branch ancestry") + + val branchHeadSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.refs WHERE name = 'mb'") + .collect()(0) + .getLong(0) + val cherryPickOutcome = + try { + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', " + + s"${branchHeadSnapshotId}L)") + s"SUCCEEDED: main now ${countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}")} rows" + } catch { + case exception: Throwable => + s"REJECTED ${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(160) + } + println( + s"DIAG expireMerge.cherrypickFallback: $cherryPickOutcome") + val mainRowCount = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}").toLong + + assert( + mainRowCount == 3 || mainRowCount == 4, + s"main should remain consistent, got $mainRowCount rows") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") == "5", + "branch data should remain available for copy-out recovery") + }, + wapPreparation.test( + "interact.branch.expireMerge.stagedWapLoss", + "Snapshot expiration removes an unreferenced staged WAP snapshot, and publishing that " + + "wap_id afterward fails while main remains at its original 3 rows.") { table => + table.spark.conf.set("spark.wap.id", "w2") + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + } finally { + table.spark.conf.unset("spark.wap.id") + } + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w2'") == "1", + "WAP write should create one staged snapshot") + + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w2'") == "0", + "expiration should remove the unreferenced staged snapshot") + + val exception = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.publish_changes(" + + s"table => '${catalogRelative(table.name)}', wap_id => 'w2')")) + println( + "DIAG stagedWapLoss.publish: " + + s"${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(180)) + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "main should remain unchanged after staged snapshot loss") + }) + } + + // A table created with write.wap.enabled and replace.enabled both set. The case reads those flags + // back, then creates a branch and confirms the replace path is refused while the branch exists. + def interactionBranchFlagCases(format: String): List[Plan.Case] = { + val flagPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', " + + "'write.wap.enabled'='true', 'replace.enabled'='true')")() + .insert(3)(), + description = s"Three seed rows in a $format table with write.wap.enabled and " + + "replace.enabled both set to true at create time.") + + List( + flagPreparation.test( + "interact.flags.wapReplaceAtCreate", + "WAP and replace flags set at CREATE time are active, and a subsequent RTAS is rejected " + + "while a branch exists and WAP is enabled.") { table => + val properties = tableProps(table.spark, table.name) + assert( + properties.get("write.wap.enabled").contains("true") && + properties.get("replace.enabled").contains("true"), + "WAP and replace flags should be active when set at CREATE") + + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH cb") + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name}")) + assert( + exception.getMessage.contains("while WAP"), + "RTAS should reject a table with WAP enabled at CREATE") + }) + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchMorScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchMorScenarios.scala new file mode 100644 index 000000000..92300062b --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchMorScenarios.scala @@ -0,0 +1,163 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// Branch merges on a merge-on-read table. A DELETE or UPDATE on a branch of a merge-on-read table +// writes position-delete files on the branch, and this family pins what merging that branch back to +// main does with them. It needs both the merge-on-read write modes and branch refs, so it belongs to +// the branch layer that sits above merge-on-read. +trait BranchMorScenarios extends BranchScenarioKit { + import Rows._ + + // Merge-on-read tables with branch merges: a DELETE or UPDATE on a branch of a MoR table writes + // position-delete files on the branch, and merging the branch back to main must carry those + // deletes correctly. The base table is a single-file MoR seed (a coalesced write of 1 file) so a + // strict-subset DELETE produces a real position delete. Merge operates on refs and snapshots, so + // one MoR layout covers the format-independent behavior. Each case checks that deletes are + // carried across the merge, that deleted rows stay absent from main, and how cherry-pick handles + // row-delete snapshots. + lazy val morBranchMergeCases: List[Plan.Case] = + morVerifyLayouts + .filter(layout => + layout.label == "mor-verify/parquet" || + layout.label == "mor-verify/orc") + .map(layout => + TablePreparation( + layout.label, + createAndSeedSingleFile(layout, 3), + description = s"Three seed rows written as one data file in ${layout.description}.")) + .flatMap { preparation => + List( + preparation.test( + "mbranch.fastForwardDelete", + "fast_forward carries a branch's position-delete DELETE onto main: main gains the " + + "branch's 2-row state and the deleted row does not reappear.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mfb") + table.spark.sql( + s"DELETE FROM ${table.name}.branch_mfb " + + s"WHERE ${Core.long0.columnName} = 1") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "main advanced before fast-forward") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mfb'") == "2", + "branch delete was not applied") + + table.spark.sql( + "CALL openhouse.system.fast_forward(" + + s"'${catalogRelative(table.name)}', 'main', 'mfb')") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "2", + "fast-forward did not carry the branch position delete") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") == "0", + "deleted row reappeared after fast-forward") + }, + preparation.test( + "mbranch.fastForwardUpdate", + "fast_forward carries a branch's UPDATE onto main: the row count stays at 3 and main " + + "reads the branch's updated value.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mub") + table.spark.sql( + s"UPDATE ${table.name}.branch_mub " + + s"SET ${Core.string0.columnName} = 'br-upd' " + + s"WHERE ${Core.long0.columnName} = 2") + table.spark.sql( + "CALL openhouse.system.fast_forward(" + + s"'${catalogRelative(table.name)}', 'main', 'mub')") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "fast-forward of an update changed the main row count") + assert( + table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 2") + .collect()(0) + .getString(0) == "br-upd", + "fast-forward did not carry the branch update") + }, + preparation.test( + "mbranch.cherrypickDelete", + "Cherry-picking a branch's position-delete DELETE snapshot onto main applies that " + + "delete to main, leaving 2 rows.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mcb") + table.spark.sql( + s"DELETE FROM ${table.name}.branch_mcb " + + s"WHERE ${Core.long0.columnName} = 1") + val deleteSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "ORDER BY committed_at DESC LIMIT 1") + .collect()(0) + .getLong(0) + + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', ${deleteSnapshotId}L)") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "2", + "cherry-pick should apply the branch delete to main") + }, + preparation.test( + "mbranch.replaceBranchDelete", + "REPLACE BRANCH AS OF a pre-delete snapshot undoes a branch's earlier position-delete " + + "DELETE, restoring the branch to 3 rows.") { table => + val seedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "ORDER BY committed_at DESC LIMIT 1") + .collect()(0) + .getLong(0) + + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH mrb") + table.spark.sql( + s"DELETE FROM ${table.name}.branch_mrb " + + s"WHERE ${Core.long0.columnName} = 1") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mrb'") == "2", + "branch delete was not applied") + + table.spark.sql( + s"ALTER TABLE ${table.name} REPLACE BRANCH mrb " + + s"AS OF VERSION $seedSnapshotId") + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mrb'") == "3", + "replacing the branch target did not undo its position delete") + }) + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchScenarioKit.scala new file mode 100644 index 000000000..b34c1e647 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchScenarioKit.scala @@ -0,0 +1,80 @@ +package harness + +// The branch and write-audit-publish preparation kit. A branch preparation seeds main, creates +// branch b, and routes the session at that branch through spark.wap.branch, so every read and write +// the case performs lands on the branch while main keeps its seed rows. This layer sits above +// merge-on-read, so it also owns the branch-on-merge-on-read preparations. The members are lazy so +// they initialize on first read, after every trait mixed into `object Scenarios` has been +// constructed. +trait BranchScenarioKit extends MorScenarioKit { + + // Seed on main, create a branch, then set spark.wap.branch so every later read and write in the + // case lands on the branch. A case captures its own before state from the branch and asserts + // against it, so the same case body holds on a branch and on main. Each case runs in its own + // spark.newSession(), which keeps the setting scoped to that case. + def createAndSeedOnBranch(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = + createAndSeed(layout, numberOfRows) + .sql("prep.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() + .step("prep.routeToBranch") { (spark, table) => + spark.sql(s"ALTER TABLE $table CREATE BRANCH b") + spark.conf.set("spark.wap.branch", "b") + }() + + private def assertBranchMainIsolation(table: PreparedTable[CoreTable.type]): Unit = { + table.spark.conf.unset("spark.wap.branch") + val mainCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + assert( + mainCount == 3, + s"branch operation leaked to main: expected 3 rows, got $mainCount") + } + + private def branchPreparationDescription(layout: Layout): String = + s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, with write.wap.enabled set, " + + "branch b created, and spark.wap.branch set to b, so every read and write in the case lands " + + "on branch b while main keeps its three seed rows." + + lazy val preparedBranchCoreTables: List[TablePreparation[CoreTable.type]] = + layouts.map { layout => + TablePreparation( + layout.label, + createAndSeedOnBranch(layout, 3), + "branchWap:", + assertBranchMainIsolation, + branchPreparationDescription(layout)) + } + + lazy val preparedPartitionedBranchCoreTables: List[TablePreparation[CoreTable.type]] = + partitionedLayouts.map { layout => + TablePreparation( + layout.label, + createAndSeedOnBranch(layout, 3), + "branchWap:", + assertBranchMainIsolation, + branchPreparationDescription(layout)) + } + + lazy val preparedBranchMorCoreTables: List[TablePreparation[CoreTable.type]] = + unpartitionedMorLayouts.map { layout => + TablePreparation( + layout.label, + createAndSeedOnBranch(layout, 3), + "branchWap:", + assertBranchMainIsolation, + branchPreparationDescription(layout)) + } + + lazy val preparedNullStringBranchCoreTables: List[TablePreparation[CoreTable.type]] = + preparedBranchCoreTables.map(withNullStringRow) + + lazy val preparedNullStringBranchMorCoreTables: List[TablePreparation[CoreTable.type]] = + preparedBranchMorCoreTables.map(withNullStringRow) + + lazy val branchLayoutFormatPreparations: List[TablePreparation[CoreTable.type]] = + preparedBranchCoreTables + + def branchLayoutFormatCases: List[Plan.Case] = + layoutFormatCasesFor(branchLayoutFormatPreparations) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchSurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchSurfaceScenarios.scala new file mode 100644 index 000000000..1e09109e7 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchSurfaceScenarios.scala @@ -0,0 +1,422 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// The branch and write-audit-publish surface families. Each case pins one edge of what branch +// routing exposes: what a branch keeps to itself, what it writes through to main, how a staged +// commit is published, and how maintenance behaves while a branch exists. The cases run on parquet +// and orc. +trait BranchSurfaceScenarios extends BranchScenarioKit { + import Rows._ + + // Each surface family builds the starting states it needs, so a family reads on its own. + private def surfaceBasePreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)(), + description = s"Three seed rows with keys 1, 2 and 3 in an unpartitioned $format table.") + + private def surfaceTwoSnapshotPreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("insertMore")(table => + s"INSERT INTO $table VALUES " + + "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + + "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")(), + description = s"Five rows across two snapshots (a 3-row seed then a 2-row insert) in an " + + s"unpartitioned $format table.") + + private def surfaceWapPreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableWap")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")(), + description = s"Three seed rows in an unpartitioned $format table with " + + "write.wap.enabled=true.") + + // Compaction run against a table that carries a branch. + def surfaceBranchMaintenanceCases(format: String): List[Plan.Case] = + List( + surfaceTwoSnapshotPreparation(format).test( + "surface.maint.compactWithBranch", + "Compacting main while a branch exists preserves both main's and the branch's 6 rows; " + + "a follow-up compaction attempt routed at the branch via spark.wap.branch still leaves " + + "main and the branch at 6 rows, whichever way that routed attempt resolves.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH cb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_cb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + val compactionResult = table.spark + .sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('min-input-files', '2'))") + .collect()(0) + + println( + "DIAG compactWithBranch: " + + s"mainCompaction rewritten=${compactionResult.get(0)} " + + s"added=${compactionResult.get(1)}") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "6", + "main compaction should preserve 6 rows") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") == "6", + "main compaction should preserve the branch") + + table.spark.conf.set("spark.wap.branch", "cb") + val branchRoutedOutcome = + try { + val result = table.spark + .sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}')") + .collect()(0) + s"RAN (rewritten=${result.get(0)}, added=${result.get(1)})" + } catch { + case exception: Throwable => + s"THREW ${exception.getClass.getSimpleName} :: " + + Option(exception.getMessage).getOrElse("").take(140) + } finally { + table.spark.conf.unset("spark.wap.branch") + } + println(s"DIAG compactUnderWapConf: $branchRoutedOutcome") + + table.spark.sql(s"REFRESH TABLE ${table.name}") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "6", + "branch-routed compaction attempt should preserve main") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") == "6", + "branch-routed compaction attempt should preserve the branch") + }) + + // What a branch keeps to itself and what it leaks to main, plus the branch merge and retarget procedures. + def surfaceBranchCases(format: String): List[Plan.Case] = + List( + surfaceBasePreparation(format).test( + "branch.leak.setProps", + "SET TBLPROPERTIES issued while spark.wap.branch is set changes table-global metadata: " + + "the user property is visible on the table's own properties.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH lb2") + table.spark.conf.set("spark.wap.branch", "lb2") + try { + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('user.leaked'='yes')") + } finally { + table.spark.conf.unset("spark.wap.branch") + } + + assert( + tableProps(table.spark, table.name) + .get("user.leaked") + .contains("yes"), + "branch-routed property update should change table-global metadata") + }, + surfaceBasePreparation(format).test( + "branch.leak.writeOrderedBy", + "WRITE ORDERED BY issued while spark.wap.branch is set changes table-global metadata: " + + "write.distribution-mode becomes range on the table itself.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH lb3") + table.spark.conf.set("spark.wap.branch", "lb3") + try { + table.spark.sql( + s"ALTER TABLE ${table.name} " + + s"WRITE ORDERED BY ${Core.long0.columnName}") + } finally { + table.spark.conf.unset("spark.wap.branch") + } + + assert( + tableProps(table.spark, table.name) + .get("write.distribution-mode") + .contains("range"), + "branch-routed ordering should change table-global metadata") + }, + surfaceWapPreparation(format).test( + "branch.wapToggle.noGuard", + "A spark.wap.id-tagged insert produces exactly one staged snapshot carrying that " + + "wap.id, and that snapshot is unaffected by later disabling write.wap.enabled on the " + + "table.") { table => + table.spark.conf.set("spark.wap.id", "w9") + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + } finally { + table.spark.conf.unset("spark.wap.id") + } + val stagedSnapshotCount = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w9'") + assert( + stagedSnapshotCount == "1", + s"expected one staged snapshot, got $stagedSnapshotCount") + + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.wap.enabled'='false')") + val stagedAfterToggle = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w9'") + + assert( + stagedAfterToggle == "1", + "disabling write.wap.enabled should not remove an already-staged snapshot, " + + s"got $stagedAfterToggle") + }, + surfaceWapPreparation(format).test( + "wap.neg.doubleCherrypick", + "Cherry-picking a WAP-staged snapshot publishes its row (row count goes from 3 to 4); " + + "cherry-picking that same snapshot a second time is rejected as a duplicate.") { table => + table.spark.conf.set("spark.wap.id", "w1") + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + } finally { + table.spark.conf.unset("spark.wap.id") + } + val stagedSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.snapshots " + + "WHERE summary['wap.id'] = 'w1'") + .collect()(0) + .getLong(0) + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', ${stagedSnapshotId}L)") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "4", + "first cherry-pick should publish the staged row") + + val exception = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.cherrypick_snapshot(" + + s"'${catalogRelative(table.name)}', ${stagedSnapshotId}L)")) + println( + "DIAG doubleCherrypick: " + + s"${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(180)) + assert( + Option(exception.getMessage).exists(message => + message.toLowerCase.contains("duplicate") || + message.toLowerCase.contains("already")), + "second cherry-pick should reject the duplicate WAP commit") + }, + surfaceBasePreparation(format).test( + "wap.neg.expireRefTarget", + "Expiring the snapshot a branch currently points to is rejected with an exception, and " + + "the branch ref still points at its original snapshot afterward.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH eb2") + val branchHeadSnapshotId = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.refs " + + "WHERE name = 'eb2'") + .collect()(0) + .getLong(0) + Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + s"snapshot_ids => ARRAY(${branchHeadSnapshotId}L))")) + + val branchHeadSnapshotIdAfter = table.spark + .sql( + s"SELECT snapshot_id FROM ${table.name}.refs " + + "WHERE name = 'eb2'") + .collect()(0) + .getLong(0) + assert( + branchHeadSnapshotIdAfter == branchHeadSnapshotId, + "rejected expiration should leave the branch ref pointing at its original snapshot") + }, + surfaceBasePreparation(format).test( + "branch.fastForward.merge", + "fast_forward moves main to a branch's head after two branch-only inserts, growing " + + "main from 3 rows to 5.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH fb") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_fb VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_fb VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "branch writes should not advance main") + + table.spark.sql( + "CALL openhouse.system.fast_forward(" + + s"'${catalogRelative(table.name)}', 'main', 'fb')") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "5", + "fast_forward should move main to the branch head") + }, + surfaceBasePreparation(format).test( + "branch.fastForward.divergent", + "fast_forward is rejected with an ancestry error when main and the branch have both " + + "advanced independently since they diverged.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH db") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_db VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + val exception = Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.fast_forward(" + + s"'${catalogRelative(table.name)}', 'main', 'db')")) + + println( + "DIAG ffDivergent: " + + s"${exception.getClass.getName} :: " + + Option(exception.getMessage).getOrElse("").take(180)) + assert( + Option(exception.getMessage).exists(message => + message.toLowerCase.contains("ancestor") || + message.toLowerCase.contains("fast-forward")), + "divergent fast_forward should report an ancestry error") + }, + surfaceTwoSnapshotPreparation(format).test( + "branch.replaceBranch", + "A new branch starts pointing at the current 5-row head; REPLACE BRANCH AS OF the " + + "earlier snapshot retargets it back to the 3-row seed state.") { table => + val snapshots = snapshotIds(table.spark, table.name) + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH rb2") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb2'") == "5", + "new branch should point at the current head") + + table.spark.sql( + s"ALTER TABLE ${table.name} REPLACE BRANCH rb2 " + + s"AS OF VERSION ${snapshots.head}") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb2'") == "3", + "REPLACE BRANCH should retarget the branch to the older snapshot") + }) + + // Publishing a write-audit-publish staged commit onto main. + def surfaceBranchPublishCases(format: String): List[Plan.Case] = + List( + surfaceWapPreparation(format).test( + "surface.proc.publishChanges", + "A WAP-staged insert stays invisible on main (still 3 rows) until publish_changes " + + "publishes it, growing main to 4 rows.") { table => + table.spark.conf.set("spark.wap.id", "pw1") + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") + } finally { + table.spark.conf.unset("spark.wap.id") + } + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "staged write should not be visible before publish") + + table.spark.sql( + "CALL openhouse.system.publish_changes(" + + s"table => '${catalogRelative(table.name)}', wap_id => 'pw1')") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "4", + "publish_changes should publish the staged row") + }) + + // The DataFrame writer targeting a branch. + def surfaceBranchWriteCases(format: String): List[Plan.Case] = + List( + surfaceBasePreparation(format).test( + "surface.write.dfToBranch", + "A DataFrame writeTo(...).append() targeting a branch adds the row to that branch " + + "(4 rows) while leaving main unchanged at 3 rows.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH wb") + val row = table.spark.sql( + s"SELECT CAST(50 AS BIGINT) AS ${Core.long0.columnName}, " + + s"50 AS ${Core.int0.columnName}, " + + s"'row-50' AS ${Core.string0.columnName}, " + + s"50.5 AS ${Core.double0.columnName}, " + + s"true AS ${Core.boolean0.columnName}, " + + s"'2024-01-09-01' AS ${Core.datePartition.columnName}") + row.writeTo(s"${table.name}.branch_wb").append() + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'wb'") == "4", + "DataFrame writer should append to the branch") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "DataFrame branch write should leave main unchanged") + }) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala index b79d9bd4e..883557407 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala @@ -10,68 +10,9 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -trait BranchWapScenarios extends ScenarioKit { +trait BranchWapScenarios extends BranchScenarioKit { import Rows._ - // ── Undrop 3-way compositions (Block 9, real HTS only) — restore's state-preservation, per feature ── - // The undrop:* battery proves the whole op catalog works post-restore. These are pointed 3-way - // chains that set up a SPECIFIC feature's state (branch / snapshot history / evolved schema), - // destroy via soft-delete→restore, then consume that exact feature — the direct modality check that - // restore's destruction set does not intersect refs / lineage / schema. - - // A pre-existing branch must survive the drop→undrop round-trip. - def interactUndropBranchSurvives(ctx: Ctx): Unit = { - val (table, db, tbl) = undropSeed(ctx, "t_ud_branch") - ctx.spark.sql(s"ALTER TABLE $table CREATE BRANCH b") - ctx.spark.sql(s"INSERT INTO $table.branch_b ${RowGenerator.valuesClause(Core, 2)}") // branch diverges: 3+2=5 - softDeleteRestore(ctx, db, tbl) - assert(ctx.spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "main row set changed across undrop") - assert(ctx.spark.sql(s"SELECT count(*) FROM $table VERSION AS OF 'b'").collect()(0).getLong(0) == 5, "branch 'b' did not survive undrop") - ctx.spark.sql(s"DROP TABLE IF EXISTS $table") - } - - // Snapshot history (time travel) must survive restore. - def interactUndropTimeTravelSurvives(ctx: Ctx): Unit = { - val (table, db, tbl) = undropSeed(ctx, "t_ud_tt") - val firstSnap = ctx.spark.sql(s"SELECT snapshot_id FROM $table.snapshots ORDER BY committed_at LIMIT 1").collect()(0).getLong(0) - ctx.spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 2)}") // 2nd snapshot: 5 rows - softDeleteRestore(ctx, db, tbl) - assert(ctx.spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 5, "current state changed across undrop") - assert(ctx.spark.sql(s"SELECT count(*) FROM $table VERSION AS OF $firstSnap").collect()(0).getLong(0) == 3, - "pre-restore snapshot not time-travellable after undrop (lineage lost)") - ctx.spark.sql(s"DROP TABLE IF EXISTS $table") - } - - // Evolved schema must survive restore, and the restored table must still accept the evolved shape. - def interactUndropSchemaSurvives(ctx: Ctx): Unit = { - val (table, db, tbl) = undropSeed(ctx, "t_ud_schema") - ctx.spark.sql(s"ALTER TABLE $table ADD COLUMN extra int") - ctx.spark.sql(s"INSERT INTO $table VALUES (CAST(9 AS BIGINT), 9, 'row-9', 9.5, false, '2024-01-09-08', 99)") - softDeleteRestore(ctx, db, tbl) - assert(ctx.spark.sql(s"SELECT extra FROM $table WHERE ${Core.long0.columnName} = 9").collect()(0).getInt(0) == 99, - "evolved column value lost across undrop") - ctx.spark.sql(s"INSERT INTO $table VALUES (CAST(10 AS BIGINT), 10, 'row-10', 10.5, true, '2024-01-10-09', 100)") - assert(ctx.spark.sql(s"SELECT count(*) FROM $table WHERE extra IS NOT NULL").collect()(0).getLong(0) == 2, - "restored table did not accept the evolved schema for new writes") - ctx.spark.sql(s"DROP TABLE IF EXISTS $table") - } - - def undropInteractionCases: List[Plan.Case] = - if (HtsAdmin.enabled) { - List( - Plan.Case( - "interact.undrop.branchSurvives", - interactUndropBranchSurvives), - Plan.Case( - "interact.undrop.timeTravelSurvives", - interactUndropTimeTravelSurvives), - Plan.Case( - "interact.undrop.schemaSurvives", - interactUndropSchemaSurvives)) - } else { - Nil - } - val wapStagedCases: List[Plan.Case] = List("parquet", "orc").flatMap { format => val preparation = TablePreparation( @@ -82,10 +23,14 @@ trait BranchWapScenarios extends ScenarioKit { s"TBLPROPERTIES ('write.format.default'='$format')")() .insert(3)() .sql("enableWap")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")()) + s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")(), + description = s"Three seed rows in a $format table with write.wap.enabled set to true.") List( - preparation.test("wapStaged.insert") { table => + preparation.test( + "wapStaged.insert", + "A staged INSERT under spark.wap.id does not change main until its snapshot is " + + "cherry-picked, after which main includes the inserted row.") { table => table.spark.conf.set("spark.wap.id", "wS") try { table.spark.sql( @@ -126,7 +71,10 @@ trait BranchWapScenarios extends ScenarioKit { .getLong(0) == 4, "publishing the staged insert did not advance main") }, - preparation.test("wapStaged.overwrite") { table => + preparation.test( + "wapStaged.overwrite", + "A staged INSERT OVERWRITE under spark.wap.id does not change main until its snapshot " + + "is cherry-picked, after which main is replaced by the overwritten rows.") { table => table.spark.conf.set("spark.wap.id", "wS") try { table.spark.sql( @@ -167,7 +115,10 @@ trait BranchWapScenarios extends ScenarioKit { .getLong(0) == 1, "publishing the staged overwrite did not replace main") }, - preparation.test("wapStaged.delete.bypassesWap") { table => + preparation.test( + "wapStaged.delete.bypassesWap", + "A DELETE issued under spark.wap.id commits directly to main with no staged snapshot, " + + "unlike INSERT, OVERWRITE, and MERGE.") { table => table.spark.conf.set("spark.wap.id", "wD") try { table.spark.sql( @@ -194,7 +145,10 @@ trait BranchWapScenarios extends ScenarioKit { mainRowCount == 2 && stagedSnapshotCount == 0, "staged DELETE should commit directly to main without a WAP snapshot") }, - preparation.test("wapStaged.merge") { table => + preparation.test( + "wapStaged.merge", + "A staged MERGE INSERT under spark.wap.id does not change main until its snapshot is " + + "cherry-picked, after which main includes the merged row.") { table => table.spark.conf.set("spark.wap.id", "wS") try { table.spark.sql( @@ -240,7 +194,10 @@ trait BranchWapScenarios extends ScenarioKit { .getLong(0) == 4, "publishing the staged merge did not advance main") }, - preparation.test("wapStaged.update.valueVisibleOnlyAfterPublish") { table => + preparation.test( + "wapStaged.update.valueVisibleOnlyAfterPublish", + "A staged UPDATE under spark.wap.id leaves the old value visible on main until its " + + "snapshot is cherry-picked, after which main reads the updated value.") { table => table.spark.conf.set("spark.wap.id", "wU") try { table.spark.sql( @@ -281,7 +238,11 @@ trait BranchWapScenarios extends ScenarioKit { valueAfterPublish == "staged-upd", s"published update returned $valueAfterPublish") }, - preparation.test("wapStaged.twoIdsIndependent") { table => + preparation.test( + "wapStaged.twoIdsIndependent", + "Two inserts staged under different spark.wap.id values publish independently: " + + "cherry-picking one advances main without exposing the other's row until it too is " + + "cherry-picked.") { table => def stageInsert(wapId: String, key: Int): Unit = { table.spark.conf.set("spark.wap.id", wapId) try { @@ -337,7 +298,10 @@ trait BranchWapScenarios extends ScenarioKit { .getLong(0) == 5, "publishing wb did not advance main") }, - preparation.test("wapStaged.expireVsStaged") { table => + preparation.test( + "wapStaged.expireVsStaged", + "Expiring snapshots with retain_last=1 removes an unreferenced staged WAP snapshot, and " + + "cherry-picking it afterward fails because the snapshot is gone.") { table => table.spark.conf.set("spark.wap.id", "wE") try { table.spark.sql( @@ -396,10 +360,15 @@ trait BranchWapScenarios extends ScenarioKit { .sql("enableWap")(table => s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")() .sql("createBranch")(table => - s"ALTER TABLE $table CREATE BRANCH bddl")()) + s"ALTER TABLE $table CREATE BRANCH bddl")(), + description = s"Three seed rows in a $format table with write.wap.enabled set to true and " + + "branch bddl created.") List( - preparation.test("branchDdl.addColumn.leaksToMain") { table => + preparation.test( + "branchDdl.addColumn.leaksToMain", + "ALTER TABLE ADD COLUMN issued while spark.wap.branch selects a branch is accepted and " + + "adds the column to the table's global schema, visible on main.") { table => table.spark.conf.set("spark.wap.branch", "bddl") val outcome = try { @@ -425,7 +394,10 @@ trait BranchWapScenarios extends ScenarioKit { columnNames.contains("br_added"), "ADD COLUMN on a branch should change the table-global schema") }, - preparation.test("branchDdl.setTblProp.leaksToMain") { table => + preparation.test( + "branchDdl.setTblProp.leaksToMain", + "ALTER TABLE SET TBLPROPERTIES issued while spark.wap.branch selects a branch is accepted " + + "and changes the table's global properties, visible on main.") { table => table.spark.conf.set("spark.wap.branch", "bddl") val outcome = try { @@ -452,7 +424,10 @@ trait BranchWapScenarios extends ScenarioKit { properties.get("user.branchkey").contains("v1"), "SET TBLPROPERTIES on a branch should change table-global properties") }, - preparation.test("branchDdl.alterColumnComment.leaksToMain") { table => + preparation.test( + "branchDdl.alterColumnComment.leaksToMain", + "ALTER TABLE ALTER COLUMN COMMENT issued while spark.wap.branch selects a branch is " + + "accepted and changes the table's global column comment, visible on main.") { table => table.spark.conf.set("spark.wap.branch", "bddl") val outcome = try { @@ -480,7 +455,10 @@ trait BranchWapScenarios extends ScenarioKit { Option(comment).getOrElse("").contains("br-comment"), "ALTER COLUMN COMMENT on a branch should change table-global metadata") }, - preparation.test("branchDdl.dropColumn.rejected") { table => + preparation.test( + "branchDdl.dropColumn.rejected", + "ALTER TABLE DROP COLUMN issued while spark.wap.branch selects a branch is rejected, and " + + "the column remains present.") { table => table.spark.conf.set("spark.wap.branch", "bddl") val outcome = try { @@ -503,6 +481,9 @@ trait BranchWapScenarios extends ScenarioKit { println( "DIAG branchDdl.dropColumn.rejected: " + s"branch-routed DDL $outcome") + assert( + outcome.startsWith("rejected:"), + s"DROP COLUMN should be rejected while a branch is selected: $outcome") assert( columnNames.contains(Core.string0.columnName), "DROP COLUMN should remain rejected while a branch is selected") @@ -517,10 +498,14 @@ trait BranchWapScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) + .insert(3)(), + description = s"Three seed rows in a $format table with no branches or WAP configuration.") List( - preparation.test("branch.direct.isolation") { table => + preparation.test( + "branch.direct.isolation", + "Inserting directly into a created branch adds a row visible only when reading that " + + "branch, and main keeps its original 3 rows.") { table => table.spark.sql( s"ALTER TABLE ${table.name} CREATE BRANCH b") table.spark.sql( @@ -543,7 +528,10 @@ trait BranchWapScenarios extends ScenarioKit { mainRowCount == 3, s"main should be unchanged at 3 rows, got $mainRowCount") }, - preparation.test("branch.wapConf.routing") { table => + preparation.test( + "branch.wapConf.routing", + "With write.wap.enabled set and spark.wap.branch selecting a branch, an INSERT and the " + + "following read both route to that branch, leaving main at its original 3 rows.") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + "('write.wap.enabled'='true')") @@ -573,7 +561,10 @@ trait BranchWapScenarios extends ScenarioKit { mainRowCount == 3, s"branch-routed write changed main to $mainRowCount rows") }, - preparation.test("wap.stagePublish") { table => + preparation.test( + "wap.stagePublish", + "A staged INSERT under spark.wap.id leaves main at its original 3 rows until its " + + "snapshot is cherry-picked, after which main includes the inserted row.") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + "('write.wap.enabled'='true')") @@ -610,7 +601,10 @@ trait BranchWapScenarios extends ScenarioKit { mainAfterPublish == 4, s"publishing the staged write left main at $mainAfterPublish rows") }, - preparation.test("branch.ddlLeak.addColumn") { table => + preparation.test( + "branch.ddlLeak.addColumn", + "ALTER TABLE ADD COLUMN issued while spark.wap.branch selects a branch changes the " + + "table's global schema, visible on main.") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + "('write.wap.enabled'='true')") @@ -630,7 +624,10 @@ trait BranchWapScenarios extends ScenarioKit { mainColumnNames.contains("leaked_col"), "ADD COLUMN on a branch should change the table-global schema") }, - preparation.test("branch.dml.updateDelete") { table => + preparation.test( + "branch.dml.updateDelete", + "UPDATE and DELETE issued while spark.wap.branch selects a branch change only that " + + "branch's rows and leave main at its original 3 rows.") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + "('write.wap.enabled'='true')") @@ -675,19 +672,55 @@ trait BranchWapScenarios extends ScenarioKit { branchValue == "br-upd", s"branch update returned $branchValue") }, - preparation.test("branch.lifecycle.tag") { table => + preparation.test( + "branch.lifecycle.tag", + "A tag pins its snapshot through a later insert and snapshot expiration: the tag still " + + "reads 3 rows, main reads 4 rows including the new one, and the tagged snapshot is not " + + "expired.") { table => table.spark.sql( s"ALTER TABLE ${table.name} CREATE TAG mytag") - val tagCount = table.spark + val taggedSnapshotId = table.spark .sql( - s"SELECT count(*) FROM ${table.name}.refs " + + s"SELECT snapshot_id FROM ${table.name}.refs " + "WHERE name = 'mytag' AND type = 'TAG'") .collect()(0) .getLong(0) - assert(tagCount == 1, "CREATE TAG did not create the tag ref") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mytag'") + .collect()(0) + .getLong(0) == 3, + "the tag should read the snapshot captured before the insert") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "the main branch should include the inserted row") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.snapshots " + + s"WHERE snapshot_id = $taggedSnapshotId") + .collect()(0) + .getLong(0) == 1, + "snapshot expiration should retain the snapshot referenced by the tag") }, - preparation.test("branch.lifecycle.dropBranch") { table => + preparation.test( + "branch.lifecycle.dropBranch", + "CREATE BRANCH adds a ref that DROP BRANCH then removes.") { table => table.spark.sql( s"ALTER TABLE ${table.name} CREATE BRANCH tmpbr") val branchCountBeforeDrop = table.spark @@ -713,7 +746,10 @@ trait BranchWapScenarios extends ScenarioKit { branchCountAfterDrop == 0, "DROP BRANCH did not remove the branch ref") }, - preparation.test("branch.neg.wapIdAndBranch") { table => + preparation.test( + "branch.neg.wapIdAndBranch", + "Setting both spark.wap.id and spark.wap.branch on a write is rejected with a validation " + + "error naming the conflict.") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + "('write.wap.enabled'='true')") @@ -733,7 +769,10 @@ trait BranchWapScenarios extends ScenarioKit { table.spark.conf.unset("spark.wap.branch") } }, - preparation.test("branch.neg.insertNonexistentBranch") { table => + preparation.test( + "branch.neg.insertNonexistentBranch", + "Inserting into a branch name that was never created is rejected with a validation error " + + "saying the branch does not exist.") { table => val exception = Check.intercept[ValidationException]( table.spark.sql( s"INSERT INTO ${table.name}.branch_nope VALUES " + diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala index 4f64ecf4e..52c959e77 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala @@ -1,651 +1,626 @@ package harness -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - +import org.apache.spark.sql.Row +import org.apache.spark.sql.functions.lit + +// The DML tests are written as two independent lists. A TablePreparation describes a starting +// table state (layout, evolution, restored table). A DmlTestCase describes one operation and asserts +// the rows and the snapshot delta that operation causes. A bucket of cases is the cross of a +// preparation list with the test-case list it is compatible with, so every case reads as "this +// operation, on this starting state". The named test-case lists below are the shared vocabulary a +// feature layer reuses: it crosses them with its own preparations through a self-type on this +// trait. trait DmlScenarios extends ScenarioKit { import Rows._ - val ddlConsumerCases: List[Plan.Case] = - layouts - .filter(layout => - layout.label.endsWith("/parquet") || - layout.label.endsWith("/orc")) - .flatMap { layout => - val preparations = List( - TablePreparation( - layout.label, - createAndSeed(layout, 3) - .sql("ddl")(table => s"ALTER TABLE $table ADD COLUMN cc int")(), - "ddlConsume:addColumn."), - TablePreparation( - layout.label, - createAndSeed(layout, 3) - .sql("ddl")(table => - s"ALTER TABLE $table ALTER COLUMN ${Core.int0.columnName} TYPE bigint")(), - "ddlConsume:typeWiden."), - TablePreparation( - layout.label, - createAndSeed(layout, 3) - .sql("ddl")(table => - s"ALTER TABLE $table WRITE ORDERED BY ${Core.long0.columnName}")(), - "ddlConsume:writeOrder."), - TablePreparation( - layout.label, - createAndSeed(layout, 3) - .sql("ddl")(table => - s"ALTER TABLE $table SET TBLPROPERTIES " + - "('write.distribution-mode'='range')")(), - "ddlConsume:distMode.")) - - preparations.flatMap { preparation => - List( - preparation.test("dmlWrite") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "table is not writable after DDL") - }, - preparation.test("dmlMutate") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "mutation failed after DDL") - }, - preparation.test("timeTravel") { table => - val seedSnapshotId = - snapshotIds(table.spark, table.name).head - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF $seedSnapshotId") - .collect()(0) - .getLong(0) == 3, - "seed snapshot is not readable after DDL") - }, - preparation.test("restore") { table => - val seedSnapshotId = - snapshotIds(table.spark, table.name).head - - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $seedSnapshotId)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 3, - "restore across DDL failed") - }, - preparation.test("expire") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "table is unreadable after snapshot expiration") - }, - preparation.test("branch") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH cb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_cb " + - s"SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") - .collect()(0) - .getLong(0) == 4, - "branch write failed after DDL") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 3, - "branch write changed the main table") - }, - preparation.test("compact") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('min-input-files', '2'))") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "compaction changed rows after DDL") - }) - } - } - - val createSchemaCases: List[Plan.Case] = preparedEmptyCoreTables.map { preparation => - preparation.test("create.schema") { table => - val actual = table.spark - .table(table.name) - .schema - .fields - .toList - .map(field => field.name -> field.dataType.simpleString) - val expected = Core.tableColumns.toList.map(column => (column.columnName, column.sqlType)) - - assert(actual == expected) - assert(table.rows.isEmpty) - } - } - - val ddlSchemaCases: List[Plan.Case] = preparedCoreTables.flatMap { preparation => - List( - preparation.test("ddl.addColumn.single") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_int int") - - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - val nullCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name} WHERE added_int IS NULL") - .collect()(0) - .getLong(0) - - assert(columnNames.contains("added_int"), s"added_int missing: $columnNames") - assert( - nullCount == table.preparedRows.size, - s"existing rows should read null for added_int: $nullCount != ${table.preparedRows.size}") - assert(table.rows.size == table.preparedRows.size) - }, - preparation.test("ddl.addColumn.multiple") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMNS (added_a int, added_b string)") - - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - - assert( - columnNames.contains("added_a") && columnNames.contains("added_b"), - s"added columns missing: $columnNames") - assert(table.rows.size == table.preparedRows.size) - }, - preparation.test("ddl.addColumn.comment") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_c int COMMENT 'a note'") - - val addedColumn = table.spark - .table(table.name) - .schema - .fields - .find(_.name == "added_c") - .getOrElse(throw new AssertionError("added_c missing")) - - assert( - addedColumn.getComment().contains("a note"), - s"comment not stored: ${addedColumn.getComment()}") - }, - preparation.test("ddl.addColumn.position") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN added_after int AFTER ${Core.long0.columnName}") - - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - - assert( - columnNames.indexOf("added_after") == columnNames.indexOf(Core.long0.columnName) + 1, - s"added_after not after long0: $columnNames") - }, - preparation.test("ddl.alterColumn.typeWiden") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.int0.columnName} TYPE bigint") - - val liveColumns = table.spark.table(table.name).schema.fields.toSeq - .map(field => field.name -> field.dataType.simpleString) - .toMap - val values = table.spark - .sql( - s"SELECT ${Core.int0.columnName} FROM ${table.name} ORDER BY ${Core.long0.columnName}") - .collect() - .toSeq - .map(_.getLong(0)) - - assert( - liveColumns.get(Core.int0.columnName).contains("bigint"), - s"int0 not widened: ${liveColumns.get(Core.int0.columnName)}") - assert(values == Seq(1L, 2L, 3L), s"values not preserved after widening: $values") - }, - preparation.test("ddl.renameColumn") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN to_rename int") - table.spark.sql(s"ALTER TABLE ${table.name} RENAME COLUMN to_rename TO renamed_col") - - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - - assert( - columnNames.contains("renamed_col") && !columnNames.contains("to_rename"), - s"RENAME COLUMN silently no-oped: $columnNames") - assert(table.rows.size == table.preparedRows.size) - }) - } - - private def localizedDmlCases( - preparation: TablePreparation[CoreTable.type] - ): List[Plan.Case] = - List( - preparation.test("read.projection") { table => - val expected = table.preparedRows - .sortBy(_.get(Core.long0)) - .map(_.get(Core.string0)) - val actual = table.spark + // --- the DML test cases --- + // Each case captures the table state, runs one operation, captures the state again, and asserts + // the row change and the snapshot delta that operation caused. Deltas are relative, so a case + // holds on any preparation regardless of how many snapshots the preparation itself committed. + + val readTestCases: List[DmlTestCase[CoreTable.type]] = List( + DmlTestCase( + "read.projection", + s"SELECT of ${Core.string0.columnName} alone returns that column for every prepared row in " + + "key order and leaves the table state unchanged.", + table => { + val before = table.state + val projected = table.spark .sql( s"SELECT ${Core.string0.columnName} FROM ${table.name} " + s"ORDER BY ${Core.long0.columnName}") .collect() .toSeq .map(_.get(Core.string0)) + val after = table.state - assert(actual == expected) - }, - preparation.test("read.filter") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filter(_ >= 2) - .sorted - val actual = table.spark + assert( + projected == before.rows.sortBy(_.get(Core.long0)).map(_.get(Core.string0)), + s"projection returned $projected") + assert(after == before, "a read leaves the rows and the snapshot count unchanged") + }), + DmlTestCase( + "read.filter", + s"SELECT with a ${Core.long0.columnName} >= 2 predicate returns exactly the prepared rows " + + "whose key is 2 or greater and leaves the table state unchanged.", + table => { + val before = table.state + val selected = table.spark .sql( s"SELECT ${Core.long0.columnName} FROM ${table.name} " + s"WHERE ${Core.long0.columnName} >= 2 ORDER BY ${Core.long0.columnName}") .collect() .toSeq .map(_.get(Core.long0)) + val after = table.state - assert(actual == expected) - }, - preparation.test("format.materialization") { table => - val format = table.spark - .sql( - s"SHOW TBLPROPERTIES ${table.name} ('write.format.default')") - .collect()(0) - .getString(1) - val filePaths = table.spark - .sql(s"SELECT file_path FROM ${table.name}.files") - .collect() - .toSeq - .map(_.getString(0)) + assert( + selected == before.rows.map(_.get(Core.long0)).filter(_ >= 2).sorted, + s"filtered read returned $selected") + assert(after == before, "a read leaves the rows and the snapshot count unchanged") + })) + + // The DELETE for the preparations that already hold a row with a null string. It is its own list + // because it only means something against those starting states. + val nullStringRowTestCases: List[DmlTestCase[CoreTable.type]] = List( + DmlTestCase( + "delete.byNullCondition", + s"DELETE WHERE ${Core.string0.columnName} IS NULL removes exactly the prepared row whose " + + "string is null, leaves every other row byte for byte as it was, and commits one snapshot.", + table => { + val before = table.state + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.string0.columnName} IS NULL") + val after = table.state assert( - filePaths.nonEmpty && - filePaths.forall(_.toLowerCase.endsWith(s".$format")), - s"data files are not all .$format: $filePaths") - }, - preparation.test("delete.byPredicate") { table => - val expected = table.preparedRows.filterNot(_.get(Core.long0) < 2) + after.rows == before.rows.filter(row => Option(row.get(Core.string0)).nonEmpty), + s"rows after the null-condition DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "DELETE by a null condition commits one snapshot") + })) + + private val deleteByPartitionPredicate: DmlTestCase[CoreTable.type] = + DmlTestCase( + "delete.byPartitionPredicate", + s"DELETE WHERE ${Core.datePartition.columnName} = '2024-01-01-00' removes the rows in that " + + "partition value, keeps the rest, and commits one snapshot.", + table => { + val before = table.state + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE " + + s"${Core.datePartition.columnName} = '2024-01-01-00'") + val after = table.state + + assert( + after.rows == before.rows.filterNot(_.get(Core.datePartition) == "2024-01-01-00"), + s"rows after the DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "DELETE by a partition predicate commits one snapshot") + }) + + private val deleteTestCases: List[DmlTestCase[CoreTable.type]] = List( + DmlTestCase( + "delete.byPredicate", + s"DELETE WHERE ${Core.long0.columnName} < 2 removes the rows below key 2, keeps every other " + + "row untouched, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") + val after = table.state - assert(table.rows == expected) - }, - preparation.test("delete.byInList") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filterNot(Set(1L, 3L)) - .sorted + assert( + after.rows == before.rows.filterNot(_.get(Core.long0) < 2), + s"rows after the DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "DELETE by a predicate commits one snapshot") + }), + DmlTestCase( + "delete.byInList", + s"DELETE WHERE ${Core.long0.columnName} IN (1, 3) removes keys 1 and 3, leaves every " + + "other row exactly as prepared, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} IN (1, 3)") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("delete.byInSubquery") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filterNot(_ == 2L) - .sorted + assert( + after.rows == before.rows.filterNot(row => Set(1L, 3L)(row.get(Core.long0))), + s"rows after the DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "DELETE by an IN list commits one snapshot") + }), + DmlTestCase( + "delete.byInSubquery", + s"DELETE WHERE ${Core.long0.columnName} IN (subquery yielding 2) removes key 2, leaves " + + "every other row exactly as prepared, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} IN (" + "SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("delete.byNotInSubquery") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filter(_ == 2L) - .sorted + assert( + after.rows == before.rows.filterNot(_.get(Core.long0) == 2L), + s"rows after the DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "DELETE by an IN subquery commits one snapshot") + }), + DmlTestCase( + "delete.byNotInSubquery", + s"DELETE WHERE ${Core.long0.columnName} NOT IN (subquery yielding 2) removes every key other " + + "than 2 and leaves the row for key 2 exactly as prepared, in one snapshot.", + table => { + val before = table.state table.spark.sql( s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} NOT IN (" + "SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("delete.byExistsSubquery") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filterNot(_ == 2L) - .sorted + assert( + after.rows == before.rows.filter(_.get(Core.long0) == 2L), + s"rows after the DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "DELETE by a NOT IN subquery commits one snapshot") + }), + DmlTestCase( + "delete.byExistsSubquery", + s"DELETE WHERE EXISTS (correlated subquery matching ${Core.long0.columnName} = 2) removes " + + "key 2, leaves every other row exactly as prepared, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"DELETE FROM ${table.name} WHERE EXISTS (" + "SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) " + s"WHERE s.x = ${Core.long0.columnName})") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("delete.byNotExistsSubquery") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filter(_ == 2L) - .sorted + assert( + after.rows == before.rows.filterNot(_.get(Core.long0) == 2L), + s"rows after the DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "DELETE by an EXISTS subquery commits one snapshot") + }), + DmlTestCase( + "delete.byNotExistsSubquery", + s"DELETE WHERE NOT EXISTS (correlated subquery matching ${Core.long0.columnName} = 2) removes " + + "every key other than 2 and leaves the row for key 2 exactly as prepared, in one snapshot.", + table => { + val before = table.state table.spark.sql( s"DELETE FROM ${table.name} WHERE NOT EXISTS (" + "SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) " + s"WHERE s.x = ${Core.long0.columnName})") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("delete.byScalarSubquery") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filterNot(_ == 2L) - .sorted + assert( + after.rows == before.rows.filter(_.get(Core.long0) == 2L), + s"rows after the DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "DELETE by a NOT EXISTS subquery commits one snapshot") + }), + DmlTestCase( + "delete.byScalarSubquery", + s"DELETE WHERE ${Core.long0.columnName} = (scalar subquery yielding 2) removes key 2, " + + "leaves every other row exactly as prepared, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = (" + "SELECT max(col1) FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") - - assert(keyed(table.rows) == expected) - }, - preparation.test("delete.byNullCondition") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES (" + - "CAST(99 AS BIGINT), 99, NULL, 99.5, false, '2024-01-01-00')") - val rowsBeforeDelete = table.rows - val expected = rowsBeforeDelete - .filter(row => Option(row.get(Core.string0)).nonEmpty) - .map(_.get(Core.long0)) - .sorted + val after = table.state assert( - rowsBeforeDelete.exists(row => Option(row.get(Core.string0)).isEmpty), - "precondition: a null-string row was seeded") - - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.string0.columnName} IS NULL") + after.rows == before.rows.filterNot(_.get(Core.long0) == 2L), + s"rows after the DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "DELETE by a scalar subquery commits one snapshot") + }), + DmlTestCase( + "delete.all", + "DELETE FROM without a predicate empties the table and commits one snapshot.", + table => { + val before = table.state - assert(keyed(table.rows) == expected) - assert(!keyed(table.rows).contains(99L)) - }, - preparation.test("delete.all") { table => table.spark.sql(s"DELETE FROM ${table.name}") + val after = table.state - assert(table.rows.isEmpty) - }, - preparation.test("delete.none") { table => - val snapshotsBefore = table.snapshotCount + assert(after.rows.isEmpty, s"rows survived the unconditional DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "an unconditional DELETE commits one snapshot") + }), + DmlTestCase( + "delete.none", + s"DELETE WHERE ${Core.long0.columnName} = 999 matches no row, keeps every row, and still " + + "commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 999") + val after = table.state - assert(table.rows == table.preparedRows) + assert(after.rows == before.rows, s"a no-match DELETE changed the rows: ${after.rows}") assert( - table.snapshotCount == snapshotsBefore + 1, - "no-match DELETE with a real predicate still commits one snapshot") - }, - preparation.test("delete.byPartitionPredicate") { table => - val expected = table.preparedRows - .filterNot(_.get(Core.datePartition) == "2024-01-01-00") - .map(_.get(Core.long0)) - .sorted - - table.spark.sql( - s"DELETE FROM ${table.name} WHERE " + - s"${Core.datePartition.columnName} = '2024-01-01-00'") - - assert(keyed(table.rows) == expected) - }, - preparation.test("delete.withAlias") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filterNot(_ < 2L) - .sorted + after.snapshotCount == before.snapshotCount + 1, + "a no-match DELETE with a real predicate still commits one snapshot") + }), + deleteByPartitionPredicate, + DmlTestCase( + "delete.withAlias", + s"DELETE FROM AS x WHERE x.${Core.long0.columnName} < 2 resolves the alias, removes " + + "the rows below key 2, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"DELETE FROM ${table.name} AS x WHERE x.${Core.long0.columnName} < 2") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("delete.whereFalse.noSnapshot") { table => - val snapshotsBefore = table.snapshotCount + assert( + after.rows == before.rows.filterNot(_.get(Core.long0) < 2L), + s"rows after the DELETE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "DELETE through an alias commits one snapshot") + }), + DmlTestCase( + "delete.whereFalse.noSnapshot", + "DELETE WHERE false is optimized away: the rows stay as they are and no snapshot is committed.", + table => { + val before = table.state table.spark.sql(s"DELETE FROM ${table.name} WHERE false") + val after = table.state - assert(table.rows == table.preparedRows) + assert(after.rows == before.rows, s"DELETE WHERE false changed the rows: ${after.rows}") assert( - table.snapshotCount == snapshotsBefore, + after.snapshotCount == before.snapshotCount, "DELETE WHERE false must not commit a snapshot") - }, - preparation.test("delete.truncate") { table => + }), + DmlTestCase( + "delete.truncate", + "TRUNCATE TABLE empties the table and commits one snapshot.", + table => { + val before = table.state + table.spark.sql(s"TRUNCATE TABLE ${table.name}") + val after = table.state - assert(table.rows.isEmpty) - }, - preparation.test("delete.atSnapshot.rejected") { table => + assert(after.rows.isEmpty, s"rows survived TRUNCATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "TRUNCATE commits one snapshot") + }), + DmlTestCase( + "delete.atSnapshot.rejected", + "DELETE against a snapshot-pinned identifier is rejected with an IllegalArgumentException " + + "naming that snapshot, and the table state stays exactly as prepared.", + table => { + val before = table.state val snapshotId = table.spark .sql( s"SELECT snapshot_id FROM ${table.name}.snapshots " + "ORDER BY committed_at DESC LIMIT 1") .collect()(0) .getLong(0) + val exception = Check.intercept[IllegalArgumentException]( table.spark.sql( s"DELETE FROM ${table.name}.snapshot_id_$snapshotId " + s"WHERE ${Core.long0.columnName} < 4")) + val after = table.state assert( exception.getMessage == - s"Cannot delete from table at a specific snapshot: $snapshotId") - assert(table.rows == table.preparedRows) - }, - preparation.test("update.byPredicate") { table => - val expected = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id == 2) "X" else value) - } + s"Cannot delete from table at a specific snapshot: $snapshotId", + s"unexpected rejection message: ${exception.getMessage}") + assert(after == before, "a rejected DELETE leaves the rows and the snapshot count unchanged") + })) + + private val updateTestCases: List[DmlTestCase[CoreTable.type]] = List( + DmlTestCase( + "update.byPredicate", + s"UPDATE SET ${Core.string0.columnName} = 'X' WHERE ${Core.long0.columnName} = 2 rewrites " + + "that column for key 2 only, leaves every other key's value alone, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + s"WHERE ${Core.long0.columnName} = 2") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("update.withoutCondition") { table => - val expected = longToString(table.preparedRows).map { - case (id, _) => id -> "Z" - } + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, "X") else row), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "UPDATE by a predicate commits one snapshot") + }), + DmlTestCase( + "update.withoutCondition", + s"UPDATE SET ${Core.string0.columnName} = 'Z' without a WHERE clause rewrites that column " + + "for every row and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'Z'") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("update.noMatch") { table => - val snapshotsBefore = table.snapshotCount + assert( + after.rows == before.rows.map(row => withColumnValue(row, Core.string0, "Z")), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "an unconditional UPDATE commits one snapshot") + }), + DmlTestCase( + "update.noMatch", + s"UPDATE ... WHERE ${Core.long0.columnName} = 99 matches no row, leaves every value as it " + + "was, and still commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'Y' " + s"WHERE ${Core.long0.columnName} = 99") + val after = table.state - assert(longToString(table.rows) == longToString(table.preparedRows)) assert( - table.snapshotCount == snapshotsBefore + 1, - "no-match UPDATE still commits one snapshot") - }, - preparation.test("update.byInSubquery") { table => - val expected = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id == 2) "X" else value) - } + after.rows == before.rows, + s"a no-match UPDATE changed the rows: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a no-match UPDATE still commits one snapshot") + }), + DmlTestCase( + "update.byInSubquery", + s"UPDATE ... WHERE ${Core.long0.columnName} IN (subquery yielding 2) rewrites key 2 only and " + + "commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + s"WHERE ${Core.long0.columnName} IN (" + "SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("update.byNotInSubquery") { table => - val expected = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id != 2) "X" else value) - } + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, "X") else row), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "UPDATE by an IN subquery commits one snapshot") + }), + DmlTestCase( + "update.byNotInSubquery", + s"UPDATE ... WHERE ${Core.long0.columnName} NOT IN (subquery yielding 2) rewrites every key " + + "other than 2 and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + s"WHERE ${Core.long0.columnName} NOT IN (" + "SELECT col1 FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("update.byExistsSubquery") { table => - val expected = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id == 2) "X" else value) - } + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) row else withColumnValue(row, Core.string0, "X")), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "UPDATE by a NOT IN subquery commits one snapshot") + }), + DmlTestCase( + "update.byExistsSubquery", + s"UPDATE ... WHERE EXISTS (correlated subquery matching ${Core.long0.columnName} = 2) " + + "rewrites key 2 only and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + "WHERE EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) " + s"WHERE s.x = ${Core.long0.columnName})") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("update.byNotExistsSubquery") { table => - val expected = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id != 2) "X" else value) - } + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, "X") else row), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "UPDATE by an EXISTS subquery commits one snapshot") + }), + DmlTestCase( + "update.byNotExistsSubquery", + s"UPDATE ... WHERE NOT EXISTS (correlated subquery matching ${Core.long0.columnName} = 2) " + + "rewrites every key other than 2 and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + "WHERE NOT EXISTS (SELECT 1 FROM VALUES (CAST(2 AS BIGINT)) AS s(x) " + s"WHERE s.x = ${Core.long0.columnName})") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("update.byScalarSubquery") { table => - val expected = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id == 2) "X" else value) - } + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) row else withColumnValue(row, Core.string0, "X")), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "UPDATE by a NOT EXISTS subquery commits one snapshot") + }), + DmlTestCase( + "update.byScalarSubquery", + s"UPDATE ... WHERE ${Core.long0.columnName} = (scalar subquery yielding 2) rewrites key 2 " + + "only and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X' " + s"WHERE ${Core.long0.columnName} = (" + "SELECT max(col1) FROM VALUES (CAST(2 AS BIGINT)) AS s(col1))") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("update.withAlias") { table => - val expected = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id == 2) "X" else value) - } + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, "X") else row), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "UPDATE by a scalar subquery commits one snapshot") + }), + DmlTestCase( + "update.withAlias", + s"UPDATE
AS x SET x.${Core.string0.columnName} ... WHERE x.${Core.long0.columnName} " + + "= 2 resolves the alias on both sides, rewrites key 2 only, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} AS x SET x.${Core.string0.columnName} = 'X' " + s"WHERE x.${Core.long0.columnName} = 2") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("update.multipleColumns") { table => - val expectedStrings = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id == 2) "X" else value) - } + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, "X") else row), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "UPDATE through an alias commits one snapshot") + }), + DmlTestCase( + "update.multipleColumns", + s"UPDATE SET ${Core.string0.columnName} = 'X', ${Core.int0.columnName} = 99 WHERE " + + s"${Core.long0.columnName} = 2 rewrites both columns of key 2 in one statement and commits " + + "one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'X', " + s"${Core.int0.columnName} = 99 WHERE ${Core.long0.columnName} = 2") + val after = table.state - assert(longToString(table.rows) == expectedStrings) assert( - table.rows - .find(_.get(Core.long0) == 2L) - .map(_.get(Core.int0)) - .contains(99)) - }, - preparation.test("update.byExpression") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .map(value => if (value == 2L) 12L else value) - .sorted + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) { + withColumnValue(withColumnValue(row, Core.string0, "X"), Core.int0, 99) + } else row), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a multi-column UPDATE commits one snapshot") + }), + DmlTestCase( + "update.byExpression", + s"UPDATE SET ${Core.long0.columnName} = ${Core.long0.columnName} + 10 WHERE " + + s"${Core.long0.columnName} = 2 moves key 2 to key 12, leaves the other keys alone, and " + + "commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET " + s"${Core.long0.columnName} = ${Core.long0.columnName} + 10 " + s"WHERE ${Core.long0.columnName} = 2") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("update.movePartition") { table => - val expected = table.preparedRows.map { row => - val id = row.get(Core.long0) - id -> (if (id == 2) "2099-12-31-23" else row.get(Core.datePartition)) - }.toMap + assert( + after.rows == inKeyOrder(before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.long0, 12L) else row)), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "UPDATE by an expression commits one snapshot") + }), + DmlTestCase( + "update.movePartition", + s"UPDATE SET ${Core.datePartition.columnName} = '2099-12-31-23' WHERE " + + s"${Core.long0.columnName} = 2 moves key 2 to another partition value, leaves the other " + + "rows in their partitions, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET " + s"${Core.datePartition.columnName} = '2099-12-31-23' " + s"WHERE ${Core.long0.columnName} = 2") + val after = table.state - val actual = table.rows.map(row => - row.get(Core.long0) -> row.get(Core.datePartition)).toMap - - assert(actual == expected) - }, - preparation.test("update.nullAssignment") { table => - val expected = table.preparedRows.map { row => - val id = row.get(Core.long0) - id -> (if (id == 2) None else Option(row.get(Core.string0))) - }.toMap + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) { + withColumnValue(row, Core.datePartition, "2099-12-31-23") + } else row), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a partition-moving UPDATE commits one snapshot") + }), + DmlTestCase( + "update.nullAssignment", + s"UPDATE SET ${Core.string0.columnName} = NULL WHERE ${Core.long0.columnName} = 2 stores a " + + "null in that column for key 2 only and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"UPDATE ${table.name} SET ${Core.string0.columnName} = NULL " + s"WHERE ${Core.long0.columnName} = 2") + val after = table.state - val actual = table.rows.map(row => - row.get(Core.long0) -> Option(row.get(Core.string0))).toMap - - assert(actual == expected) - }, - preparation.test("merge.insertNotMatched") { table => - val expectedKeys = - (table.preparedRows.map(_.get(Core.long0)) ++ Seq(4L, 5L)).sorted + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, null) else row), + s"rows after the UPDATE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "an UPDATE assigning null commits one snapshot") + })) + + private val mergeTestCases: List[DmlTestCase[CoreTable.type]] = List( + DmlTestCase( + "merge.insertNotMatched", + "MERGE with only a WHEN NOT MATCHED THEN INSERT * clause appends the two source rows (keys 4 " + + "and 5) with every source column value, leaves the prepared rows exactly as they were, and " + + "commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -655,23 +630,23 @@ trait DmlScenarios extends ScenarioKit { AS s($cols) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED THEN INSERT *""") + val after = table.state - assert(keyed(table.rows) == expectedKeys) assert( - table.rows - .find(_.get(Core.long0) == 4L) - .map(_.get(Core.string0)) - .contains("row-4")) + after.rows == inKeyOrder(before.rows ++ Seq( + Row(4L, 4, "row-4", 4.5, true, "2024-01-04-03"), + Row(5L, 5, "row-5", 5.5, false, "2024-01-05-04"))), + s"rows after the MERGE: ${after.rows}") assert( - table.rows - .find(_.get(Core.long0) == 5L) - .map(_.get(Core.string0)) - .contains("row-5")) - }, - preparation.test("merge.updateMatched") { table => - val expected = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id == 2) "M" else value) - } + after.snapshotCount == before.snapshotCount + 1, + "a MERGE that inserts commits one snapshot") + }), + DmlTestCase( + "merge.updateMatched", + "MERGE with only a WHEN MATCHED THEN UPDATE clause rewrites the matched key 2, leaves the " + + "unmatched rows alone, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -680,14 +655,22 @@ trait DmlScenarios extends ScenarioKit { ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("merge.deleteMatched") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filterNot(Set(1L, 3L)) - .sorted + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, "M") else row), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a MERGE that updates commits one snapshot") + }), + DmlTestCase( + "merge.deleteMatched", + "MERGE with only a WHEN MATCHED THEN DELETE clause removes the matched keys 1 and 3, keeps " + + "the unmatched rows, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -695,16 +678,21 @@ trait DmlScenarios extends ScenarioKit { AS s(${Core.long0.columnName}) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN MATCHED THEN DELETE""") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("merge.upsert") { table => - val updated = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id == 2) "U" else value) - } - val expected = - if (table.preparedRows.exists(_.get(Core.long0) == 7L)) updated - else updated + (7L -> "g") + assert( + after.rows == before.rows.filterNot(row => Set(1L, 3L)(row.get(Core.long0))), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a MERGE that deletes commits one snapshot") + }), + DmlTestCase( + "merge.upsert", + "MERGE with both an UPDATE clause and an INSERT clause rewrites the matched key 2 and appends " + + "the unmatched key 7 in a single statement, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -716,14 +704,24 @@ trait DmlScenarios extends ScenarioKit { WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName} WHEN NOT MATCHED THEN INSERT *""") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("merge.deleteNotMatchedBySource") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filter(_ == 2L) - .sorted + assert( + after.rows == inKeyOrder( + before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, "U") else row) :+ + Row(7L, 7, "g", 7.5, false, "2024-01-07-06")), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "an upsert MERGE commits one snapshot") + }), + DmlTestCase( + "merge.deleteNotMatchedBySource", + "MERGE with a WHEN NOT MATCHED BY SOURCE THEN DELETE clause removes every row the source does " + + "not carry, keeps the matched key 2, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -731,13 +729,22 @@ trait DmlScenarios extends ScenarioKit { AS s(${Core.long0.columnName}) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED BY SOURCE THEN DELETE""") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("merge.conditionalUpdate") { table => - val expected = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id == 2) "U2" else value) - } + assert( + after.rows == before.rows.filter(_.get(Core.long0) == 2L), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a not-matched-by-source MERGE commits one snapshot") + }), + DmlTestCase( + "merge.conditionalUpdate", + "MERGE with a WHEN MATCHED AND THEN UPDATE clause rewrites only the matched row " + + "that also satisfies the condition (key 2), leaves matched key 3 as it was, and commits one " + + "snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -747,14 +754,22 @@ trait DmlScenarios extends ScenarioKit { ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN MATCHED AND s.${Core.long0.columnName} = 2 THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") + val after = table.state - assert(longToString(table.rows) == expected) - }, - preparation.test("merge.multipleMatchedClauses") { table => - val expected = table.preparedRows - .map(_.get(Core.long0)) - .filterNot(_ == 3L) - .sorted + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, "U2") else row), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a conditional-update MERGE commits one snapshot") + }), + DmlTestCase( + "merge.multipleMatchedClauses", + "MERGE with two MATCHED clauses applies the first matching clause per row: key 2 is updated " + + "by the conditional clause and key 3 falls through to the DELETE clause, in one snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -765,17 +780,24 @@ trait DmlScenarios extends ScenarioKit { WHEN MATCHED AND s.${Core.long0.columnName} = 2 THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName} WHEN MATCHED THEN DELETE""") + val after = table.state - assert(keyed(table.rows) == expected) assert( - table.rows - .find(_.get(Core.long0) == 2L) - .map(_.get(Core.string0)) - .contains("U")) - }, - preparation.test("merge.conditionalInsert") { table => - val expected = - (table.preparedRows.map(_.get(Core.long0)) :+ 4L).sorted + after.rows == before.rows + .filterNot(_.get(Core.long0) == 3L) + .map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, "U") else row), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a multi-clause MERGE commits one snapshot") + }), + DmlTestCase( + "merge.conditionalInsert", + "MERGE with a WHEN NOT MATCHED AND THEN INSERT clause appends only the source row " + + "that satisfies the condition (key 4), skips key 5, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -785,10 +807,22 @@ trait DmlScenarios extends ScenarioKit { AS s($cols) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED AND s.${Core.long0.columnName} = 4 THEN INSERT *""") + val after = table.state + + assert( + after.rows == inKeyOrder(before.rows :+ Row(4L, 4, "row-4", 4.5, true, "2024-01-04-03")), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a conditional-insert MERGE commits one snapshot") + }), + DmlTestCase( + "merge.allClauses", + "MERGE carrying UPDATE, INSERT, and NOT MATCHED BY SOURCE DELETE clauses updates key 2, " + + "inserts key 4, deletes the rows the source omits, and commits one snapshot.", + table => { + val before = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("merge.allClauses") { table => table.spark.sql( s"""MERGE INTO ${table.name} t USING ( SELECT * FROM VALUES @@ -800,15 +834,26 @@ trait DmlScenarios extends ScenarioKit { SET t.${Core.string0.columnName} = s.${Core.string0.columnName} WHEN NOT MATCHED THEN INSERT * WHEN NOT MATCHED BY SOURCE THEN DELETE""") + val after = table.state - assert(keyed(table.rows) == Seq(2L, 4L)) assert( - table.rows - .find(_.get(Core.long0) == 2L) - .map(_.get(Core.string0)) - .contains("M2")) - }, - preparation.test("merge.updateStar") { table => + after.rows == inKeyOrder( + before.rows + .filter(_.get(Core.long0) == 2L) + .map(row => withColumnValue(row, Core.string0, "M2")) :+ + Row(4L, 4, "row-4", 4.5, false, "2024-01-04-03")), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a MERGE with every clause commits one snapshot") + }), + DmlTestCase( + "merge.updateStar", + "MERGE with WHEN MATCHED THEN UPDATE SET * copies every source column onto the matched key 2, " + + "leaves the unmatched rows exactly as prepared, and commits one snapshot.", + table => { + val before = table.state + table.spark.sql( s"""MERGE INTO ${table.name} t USING ( SELECT * FROM VALUES @@ -816,15 +861,24 @@ trait DmlScenarios extends ScenarioKit { AS s($cols) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN MATCHED THEN UPDATE SET *""") + val after = table.state - val updatedRow = table.rows.find(_.get(Core.long0) == 2L) - - assert(updatedRow.map(_.get(Core.string0)).contains("S2")) - assert(updatedRow.map(_.get(Core.int0)).contains(22)) - }, - preparation.test("merge.insertExplicitColumns") { table => - val expected = - (table.preparedRows.map(_.get(Core.long0)) :+ 7L).sorted + assert( + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) { + Row(2L, 22, "S2", 22.5, true, "2024-06-06-06") + } else row), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "an UPDATE SET * MERGE commits one snapshot") + }), + DmlTestCase( + "merge.insertExplicitColumns", + "MERGE whose INSERT clause names a column subset appends key 7 with the named values, leaves " + + "the unnamed columns null, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -834,17 +888,21 @@ trait DmlScenarios extends ScenarioKit { WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}, ${Core.string0.columnName}) VALUES (s.${Core.long0.columnName}, s.${Core.string0.columnName})""") + val after = table.state - assert(keyed(table.rows) == expected) assert( - table.rows - .find(_.get(Core.long0) == 7L) - .map(_.get(Core.string0)) - .contains("g")) - }, - preparation.test("merge.sourceCTE") { table => - val expected = - (table.preparedRows.map(_.get(Core.long0)) :+ 8L).sorted + after.rows == inKeyOrder(before.rows :+ Row(7L, null, "g", null, null, null)), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "an explicit-column MERGE insert commits one snapshot") + }), + DmlTestCase( + "merge.sourceCTE", + "MERGE whose source is a common table expression appends the key 8 that CTE yields, with " + + "null in every column the CTE does not supply, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -855,12 +913,21 @@ trait DmlScenarios extends ScenarioKit { ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}) VALUES (s.${Core.long0.columnName})""") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("merge.sourceSetOp") { table => - val expected = - (table.preparedRows.map(_.get(Core.long0)) ++ Seq(8L, 9L)).sorted + assert( + after.rows == inKeyOrder(before.rows :+ Row(8L, null, null, null, null, null)), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a MERGE from a CTE source commits one snapshot") + }), + DmlTestCase( + "merge.sourceSetOp", + "MERGE whose source is a UNION ALL appends both keys the set operation yields (8 and 9), with " + + "null in every column the source does not supply, and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -870,12 +937,26 @@ trait DmlScenarios extends ScenarioKit { ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED THEN INSERT (${Core.long0.columnName}) VALUES (s.${Core.long0.columnName})""") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("merge.intoEmptyTarget") { table => + assert( + after.rows == inKeyOrder(before.rows ++ Seq( + Row(8L, null, null, null, null, null), + Row(9L, null, null, null, null, null))), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a MERGE from a set-operation source commits one snapshot") + }), + DmlTestCase( + "merge.intoEmptyTarget", + "After the table is emptied, MERGE with a NOT MATCHED INSERT clause inserts both source rows " + + "(keys 4 and 5) into the empty target and commits one snapshot.", + table => { table.spark.sql(s"DELETE FROM ${table.name}") - assert(table.rows.isEmpty) + val before = table.state + + assert(before.rows.isEmpty, s"precondition: the target is empty, got ${before.rows}") table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -885,13 +966,23 @@ trait DmlScenarios extends ScenarioKit { AS s($cols) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED THEN INSERT *""") + val after = table.state - assert(keyed(table.rows) == Seq(4L, 5L)) - }, - preparation.test("merge.nullJoinKey") { table => - val expectedStrings = longToString(table.preparedRows).map { - case (id, value) => id -> (if (id == 2) "M" else value) - } + assert( + after.rows == Seq( + Row(4L, 4, "row-4", 4.5, true, "2024-01-04-03"), + Row(5L, 5, "row-5", 5.5, false, "2024-01-05-04")), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a MERGE into an empty target commits one snapshot") + }), + DmlTestCase( + "merge.nullJoinKey", + "MERGE whose source carries a null join key matches no target row on that key: only the " + + "non-null key 2 is updated, no row is added or removed, and one snapshot is committed.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -901,15 +992,23 @@ trait DmlScenarios extends ScenarioKit { ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName}""") + val after = table.state assert( - keyed(table.rows) == - table.preparedRows.map(_.get(Core.long0)).sorted) - assert(longToString(table.rows) == expectedStrings) - }, - preparation.test("merge.resolveByName") { table => - val expected = - (table.preparedRows.map(_.get(Core.long0)) :+ 7L).sorted + after.rows == before.rows.map(row => + if (row.get(Core.long0) == 2L) withColumnValue(row, Core.string0, "M") else row), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a MERGE with a null join key commits one snapshot") + }), + DmlTestCase( + "merge.resolveByName", + "MERGE with INSERT * resolves the source columns by name: key 7 lands with every source " + + "value in its named column when the source lists its columns in another order, and one " + + "snapshot is committed.", + table => { + val before = table.state table.spark.sql( s"""MERGE INTO ${table.name} t USING ( @@ -924,33 +1023,53 @@ trait DmlScenarios extends ScenarioKit { datepartition) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED THEN INSERT *""") + val after = table.state - assert(keyed(table.rows) == expected) assert( - table.rows - .find(_.get(Core.long0) == 7L) - .map(_.get(Core.string0)) - .contains("g")) - }, - preparation.test("insert.into") { table => - val expected = - (table.preparedRows.map(_.get(Core.long0)) ++ Seq(4L, 5L)).sorted + after.rows == inKeyOrder(before.rows :+ Row(7L, 7, "g", 7.5, false, "2024-07-07-07")), + s"rows after the MERGE: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a name-resolved MERGE insert commits one snapshot") + })) + + private val insertAndOverwriteTestCases: List[DmlTestCase[CoreTable.type]] = List( + DmlTestCase( + "insert.into", + "INSERT INTO ... VALUES appends the two literal rows (keys 4 and 5), keeps the prepared rows, " + + "and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"""INSERT INTO ${table.name} VALUES (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')""") + val after = table.state + + assert( + after.rows == inKeyOrder(before.rows ++ Seq( + Row(4L, 4, "row-4", 4.5, true, "2024-01-04-03"), + Row(5L, 5, "row-5", 5.5, false, "2024-01-05-04"))), + s"rows after the INSERT: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "INSERT INTO commits one snapshot") + }), + DmlTestCase( + "insert.explicitColumns", + "INSERT INTO naming a subset of the columns is rejected by the engine with a message naming " + + "the omitted data, and the table state stays exactly as prepared.", + table => { + val before = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("insert.explicitColumns") { table => val exception = Check.intercept[Exception]( table.spark.sql( s"INSERT INTO ${table.name} " + s"(${Core.long0.columnName}, ${Core.string0.columnName}) " + "VALUES (CAST(4 AS BIGINT), 'd'), (CAST(5 AS BIGINT), 'e')")) - val exceptionMessage = - Option(exception.getMessage).getOrElse("") + val after = table.state + val exceptionMessage = Option(exception.getMessage).getOrElse("") assert( exceptionMessage.toUpperCase.contains("CANNOT_FIND_DATA") || @@ -958,219 +1077,481 @@ trait DmlScenarios extends ScenarioKit { exceptionMessage.toUpperCase.contains("INCOMPATIBLE_DATA"), "expected a partial-INSERT rejection naming the omitted column " + s"(engine limitation), got: ${exceptionMessage.take(200)}") - }, - preparation.test("insert.intoSelect") { table => - val expected = - (table.preparedRows.map(_.get(Core.long0)) :+ 6L).sorted + assert(after == before, "a rejected INSERT leaves the rows and the snapshot count unchanged") + }), + DmlTestCase( + "insert.intoSelect", + "INSERT INTO ... SELECT appends the row the SELECT produces (key 6), keeps the prepared rows, " + + "and commits one snapshot.", + table => { + val before = table.state table.spark.sql( s"INSERT INTO ${table.name} SELECT * FROM VALUES " + s"(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') " + s"AS s($cols)") + val after = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("append.dataFrame") { table => - val expected = - (table.preparedRows.map(_.get(Core.long0)) :+ 6L).sorted - val frame = table.spark.sql( - s"SELECT * FROM VALUES " + - s"(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') " + - s"AS s($cols)") + assert( + after.rows == inKeyOrder(before.rows :+ Row(6L, 6, "row-6", 6.5, true, "2024-01-06-05")), + s"rows after the INSERT: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "INSERT INTO ... SELECT commits one snapshot") + }), + DmlTestCase( + "append.dataFrame", + "The DataFrame writeTo(...).append() path appends the frame's row (key 6), keeps the prepared " + + "rows, and commits one snapshot.", + table => { + val before = table.state + + table.spark + .sql( + s"SELECT * FROM VALUES " + + s"(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') " + + s"AS s($cols)") + .writeTo(table.name) + .append() + val after = table.state - frame.writeTo(table.name).append() + assert( + after.rows == inKeyOrder(before.rows :+ Row(6L, 6, "row-6", 6.5, true, "2024-01-06-05")), + s"rows after the append: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a DataFrame append commits one snapshot") + }), + DmlTestCase( + "insert.overwrite", + "INSERT OVERWRITE ... VALUES replaces the table contents with the two literal rows (keys 1 " + + "and 2) and commits one snapshot.", + table => { + val before = table.state - assert(keyed(table.rows) == expected) - }, - preparation.test("insert.overwrite") { table => table.spark.sql( s"""INSERT OVERWRITE ${table.name} VALUES (CAST(1 AS BIGINT), 1, 'p', 1.5, false, '2024-01-01-00'), (CAST(2 AS BIGINT), 2, 'q', 2.5, true, '2024-01-02-01')""") + val after = table.state - assert(keyed(table.rows) == Seq(1L, 2L)) - }, - preparation.test("overwrite.dataFrame") { table => - val frame = table.spark.sql( - s"SELECT * FROM VALUES " + - s"(CAST(8 AS BIGINT), 8, 'h', 8.5, false, '2024-01-08-07') " + - s"AS s($cols)") + assert( + after.rows == Seq( + Row(1L, 1, "p", 1.5, false, "2024-01-01-00"), + Row(2L, 2, "q", 2.5, true, "2024-01-02-01")), + s"rows after the overwrite: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "INSERT OVERWRITE commits one snapshot") + }), + DmlTestCase( + "overwrite.dataFrame", + "The DataFrame writeTo(...).overwrite(lit(true)) path replaces every row with the frame's row " + + "(key 8) and commits one snapshot.", + table => { + val before = table.state + + table.spark + .sql( + s"SELECT * FROM VALUES " + + s"(CAST(8 AS BIGINT), 8, 'h', 8.5, false, '2024-01-08-07') " + + s"AS s($cols)") + .writeTo(table.name) + .overwrite(lit(true)) + val after = table.state - frame.writeTo(table.name).overwrite( - org.apache.spark.sql.functions.lit(true)) - assert(keyed(table.rows) == Seq(8L)) - }) + assert( + after.rows == Seq(Row(8L, 8, "h", 8.5, false, "2024-01-08-07")), + s"rows after the overwrite: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a DataFrame overwrite commits one snapshot") + })) + + // Partition-scoped writes: they only mean something on a table that is partitioned, so they are + // crossed with the partitioned preparations alone. + val partitionedTableTestCases: List[DmlTestCase[CoreTable.type]] = List( + DmlTestCase( + "insert.dynamicOverwrite", + "Under partitionOverwriteMode=dynamic, INSERT OVERWRITE with one row replaces only that row's " + + "partition (2024-01-01-00), keeps the rows of every other partition, and commits one snapshot.", + table => { + val before = table.state + + table.spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic") + try { + table.spark.sql( + s"INSERT OVERWRITE ${table.name} VALUES " + + "(CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00')") + } finally { + table.spark.conf.set("spark.sql.sources.partitionOverwriteMode", "static") + } + val after = table.state - private def operationName( - testCase: Plan.Case, - preparation: TablePreparation[CoreTable.type] - ): String = - testCase.id - .split(" @ ", 2) - .head - .stripPrefix(preparation.casePrefix) - - private def localizedMutationDmlCases( - preparation: TablePreparation[CoreTable.type] - ): List[Plan.Case] = - localizedDmlCases(preparation).filter { testCase => - val caseName = operationName(testCase, preparation) - caseName.startsWith("delete.") || - caseName.startsWith("update.") || - caseName.startsWith("merge.") + assert( + after.rows == inKeyOrder( + before.rows.filterNot(_.get(Core.datePartition) == "2024-01-01-00") :+ + Row(10L, 10, "p", 10.5, true, "2024-01-01-00")), + s"rows after the dynamic overwrite: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a dynamic partition overwrite commits one snapshot") + }), + DmlTestCase( + "overwrite.partitions", + "The DataFrame writeTo(...).overwritePartitions() path replaces only the partitions the frame " + + "carries (2024-01-01-00), keeps the rows of every other partition, and commits one snapshot.", + table => { + val before = table.state + + table.spark + .sql( + s"SELECT * FROM VALUES " + + "(CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00') " + + s"AS s($cols)") + .writeTo(table.name) + .overwritePartitions() + val after = table.state + + assert( + after.rows == inKeyOrder( + before.rows.filterNot(_.get(Core.datePartition) == "2024-01-01-00") :+ + Row(10L, 10, "p", 10.5, true, "2024-01-01-00")), + s"rows after the partition overwrite: ${after.rows}") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a partition overwrite commits one snapshot") + })) + + // --- which cases a preparation is compatible with --- + // Compatibility is a property of the starting state, so each list names the states it fits. + + /** Every DML case. Runs on any preparation that starts from three seed rows of the seed shape. */ + val allDmlTestCases: List[DmlTestCase[CoreTable.type]] = + readTestCases ++ + deleteTestCases ++ + updateTestCases ++ + mergeTestCases ++ + insertAndOverwriteTestCases + + /** The row-mutating cases: every DELETE, UPDATE and MERGE. */ + val rowMutationTestCases: List[DmlTestCase[CoreTable.type]] = + deleteTestCases ++ updateTestCases ++ mergeTestCases + + /** + * The cases that address columns by name and never write a whole seed-shaped row, so they run on + * a preparation whose column list has grown beyond the seed rows. + */ + val testCasesCompatibleWithAnAddedColumn: List[DmlTestCase[CoreTable.type]] = + readTestCases ++ deleteTestCases ++ updateTestCases + + val orderedDmlTestCases: List[DmlTestCase[CoreTable.type]] = + allDmlTestCases.map { + case testCase if testCase == deleteByPartitionPredicate => + testCase.copy(knownBugReason = Some( + "DELETE by partition predicate crashes in the Spark and Iceberg rewrite when the " + + "table has a write order.")) + case testCase => + testCase } + // --- standard preparations crossed with the cases they are compatible with --- + val coreDmlCases: List[Plan.Case] = - preparedCoreTables.flatMap(localizedDmlCases) + preparedCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) ++ + preparedNullStringCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) - val morDmlCases: List[Plan.Case] = - preparedMorCoreTables.flatMap(localizedMutationDmlCases) + val partitionedDmlCases: List[Plan.Case] = + preparedPartitionedCoreTables.flatMap(preparation => + partitionedTableTestCases.map(_.runOn(preparation))) val orderedDmlCases: List[Plan.Case] = - preparedOrderedCoreTables.flatMap(localizedDmlCases) + preparedOrderedCoreTables.flatMap(preparation => orderedDmlTestCases.map(_.runOn(preparation))) ++ + preparedNullStringOrderedCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) val evolvedDmlCases: List[Plan.Case] = - preparedEvolvedCoreTables.flatMap { preparation => - localizedDmlCases(preparation).filter { testCase => - val caseName = operationName(testCase, preparation) - (caseName.startsWith("delete.") || - caseName.startsWith("update.") || - caseName.startsWith("read.")) && - !caseName.contains("byNullCondition") - } + preparedEvolvedCoreTables.flatMap(preparation => + testCasesCompatibleWithAnAddedColumn.map(_.runOn(preparation))) + + // --- DDL consumers: a DDL evolves the table, then operations are run against it --- + + // Each preparation is one layout evolved by one DDL. A consumer case then runs an operation + // against the evolved table. Plan walks this list so every consumer family lands on the same + // preparation before the next preparation starts. + val ddlConsumerPreparations: List[TablePreparation[CoreTable.type]] = + parquetAndOrcLayouts.flatMap { layout => + List( + TablePreparation( + layout.label, + createAndSeed(layout, 3) + .sql("ddl")(table => s"ALTER TABLE $table ADD COLUMN cc int")(), + "ddlConsume:addColumn.", + description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + + "ADD COLUMN cc int, so the table carries an added column the seed rows read as null."), + TablePreparation( + layout.label, + createAndSeed(layout, 3) + .sql("ddl")(table => + s"ALTER TABLE $table ALTER COLUMN ${Core.int0.columnName} TYPE bigint")(), + "ddlConsume:typeWiden.", + description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + + s"${Core.int0.columnName} widened from int to bigint."), + TablePreparation( + layout.label, + createAndSeed(layout, 3) + .sql("ddl")(table => + s"ALTER TABLE $table WRITE ORDERED BY ${Core.long0.columnName}")(), + "ddlConsume:writeOrder.", + description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + + s"WRITE ORDERED BY ${Core.long0.columnName}, so the table carries that write sort order."), + TablePreparation( + layout.label, + createAndSeed(layout, 3) + .sql("ddl")(table => + s"ALTER TABLE $table SET TBLPROPERTIES " + + "('write.distribution-mode'='range')")(), + "ddlConsume:distMode.", + description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + + "write.distribution-mode set to range, so writes are range distributed.")) } - val rtasDmlCases: List[Plan.Case] = - preparedRtasCoreTables.flatMap(localizedDmlCases) + // The reads and writes a consumer runs against the evolved table. + def ddlConsumerDataCases( + preparation: TablePreparation[CoreTable.type]): List[Plan.Case] = + List( + preparation.test( + "dmlWrite", + "A plain INSERT still lands on the table after the DDL, taking it to four rows.") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "table is not writable after DDL") + }, + preparation.test( + "dmlMutate", + "A row-level DELETE still lands on the table after the DDL, taking it to two rows.") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "mutation failed after DDL") + }, + preparation.test( + "timeTravel", + "The seed snapshot from before the DDL is still readable through VERSION AS OF and " + + "returns its three rows.") { table => + val seedSnapshotId = + snapshotIds(table.spark, table.name).head - val rtasMorDmlCases: List[Plan.Case] = - preparedRtasMorCoreTables.flatMap(localizedMutationDmlCases) + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF $seedSnapshotId") + .collect()(0) + .getLong(0) == 3, + "seed snapshot is not readable after DDL") + }, + preparation.test( + "restore", + "rollback_to_snapshot back to the seed snapshot undoes an INSERT made after the DDL " + + "and returns the table to its three seed rows.") { table => + val seedSnapshotId = + snapshotIds(table.spark, table.name).head - val branchDmlCases: List[Plan.Case] = - preparedBranchCoreTables.flatMap(localizedDmlCases) + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $seedSnapshotId)") - val branchMorDmlCases: List[Plan.Case] = - preparedBranchMorCoreTables.flatMap(localizedMutationDmlCases) + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 3, + "restore across DDL failed") + }, + preparation.test( + "expire", + "expire_snapshots retaining only the newest snapshot leaves the table readable with " + + "its four current rows.") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") - val morReadDmlCases: List[Plan.Case] = - preparedMorReadCoreTables.flatMap { preparation => - localizedDmlCases(preparation).filter { testCase => - val caseName = operationName(testCase, preparation) - caseName.startsWith("read.") || - caseName == "format.materialization" - } - } + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "table is unreadable after snapshot expiration") + }) + + // Compaction run against the files written across the DDL. + def ddlConsumerCompactionCases( + preparation: TablePreparation[CoreTable.type]): List[Plan.Case] = + List( + preparation.test( + "compact", + "rewrite_data_files compacts the files written across the DDL and preserves the four " + + "current rows.") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('min-input-files', '2'))") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "compaction changed rows after DDL") + }) + + // --- DDL that changes the schema of a seeded table --- + + val createSchemaCases: List[Plan.Case] = preparedEmptyCoreTables.map { preparation => + preparation.test( + "create.schema", + "The created table's schema is exactly CoreTable's columns, in declaration order and with " + + "their declared types, and the table holds no rows.") { table => + val actual = table.spark + .table(table.name) + .schema + .fields + .toList + .map(field => field.name -> field.dataType.simpleString) + val expected = Core.tableColumns.toList.map(column => (column.columnName, column.sqlType)) - def undroppedDmlCases: List[Plan.Case] = - if (HtsAdmin.enabled) preparedUndroppedCoreTables.flatMap(localizedDmlCases) - else Nil + assert(actual == expected, s"schema is $actual") + assert(table.rows.isEmpty, "a table that was never seeded holds no rows") + } + } - private def localizedPartitionedDmlCases( - preparation: TablePreparation[CoreTable.type] - ): List[Plan.Case] = + val ddlSchemaCases: List[Plan.Case] = preparedCoreTables.flatMap { preparation => List( - preparation.test("insert.dynamicOverwrite") { table => - val expected = - (table.preparedRows - .filterNot(_.get(Core.datePartition) == "2024-01-01-00") - .map(_.get(Core.long0)) :+ 10L).sorted - - table.spark.conf.set( - "spark.sql.sources.partitionOverwriteMode", - "dynamic") - try { - table.spark.sql( - s"INSERT OVERWRITE ${table.name} VALUES " + - "(CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00')") - } finally { - table.spark.conf.set( - "spark.sql.sources.partitionOverwriteMode", - "static") - } + preparation.test( + "ddl.addColumn.single", + "ADD COLUMN adds the column to the schema, the existing rows read null for it, and the row " + + "count is unchanged.") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_int int") - assert(keyed(table.rows) == expected) + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + val nullCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name} WHERE added_int IS NULL") + .collect()(0) + .getLong(0) + + assert(columnNames.contains("added_int"), s"added_int missing: $columnNames") + assert( + nullCount == table.preparedRows.size, + s"existing rows should read null for added_int: $nullCount != ${table.preparedRows.size}") + assert(table.rows.size == table.preparedRows.size, "ADD COLUMN changed the row count") }, - preparation.test("overwrite.partitions") { table => - val expected = - (table.preparedRows - .filterNot(_.get(Core.datePartition) == "2024-01-01-00") - .map(_.get(Core.long0)) :+ 10L).sorted - val frame = table.spark.sql( - s"SELECT * FROM VALUES " + - "(CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00') " + - s"AS s($cols)") + preparation.test( + "ddl.addColumn.multiple", + "ADD COLUMNS with two columns in one statement adds both to the schema and leaves the row " + + "count unchanged.") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMNS (added_a int, added_b string)") - frame.writeTo(table.name).overwritePartitions() + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - assert(keyed(table.rows) == expected) - }) + assert( + columnNames.contains("added_a") && columnNames.contains("added_b"), + s"added columns missing: $columnNames") + assert(table.rows.size == table.preparedRows.size, "ADD COLUMNS changed the row count") + }, + preparation.test( + "ddl.addColumn.comment", + "ADD COLUMN ... COMMENT stores the comment on the added column and the reader sees it.") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_c int COMMENT 'a note'") - val partitionedDmlCases: List[Plan.Case] = - preparedCoreTables - .filter(_.label.startsWith("partitioned/")) - .flatMap(localizedPartitionedDmlCases) - - val rtasPartitionedDmlCases: List[Plan.Case] = - preparedRtasCoreTables - .filter(_.label.startsWith("partitioned/")) - .flatMap(localizedPartitionedDmlCases) - - val branchPartitionedDmlCases: List[Plan.Case] = - preparedBranchCoreTables - .filter(_.label.startsWith("partitioned/")) - .flatMap(localizedPartitionedDmlCases) - - // ── MoR discriminator: prove merge-on-read actually wrote position-delete files ────────── - // The rest of the MoR axis reuses CoW's row-delta assertions, which pass identically whether the - // write was copy-on-write or merge-on-read. These two pin the PHYSICAL difference: a MoR delete - // MUST add a position-delete file; a CoW delete must NOT. Both are prepared with - // `createAndSeedSingleFile` and delete a strict subset (`long0 < 2` → 1 of 3 rows), so the write - // cannot be satisfied by whole-file elimination — the outcome is deterministic across formats - // (verified: parquet/orc/avro all add exactly one position delete under MoR, none under CoW). - private def deleteFileCount(spark: SparkSession, table: String): Long = - spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) - - val deleteFileModeCases: List[Plan.Case] = { - def cases( - layouts: List[Layout], - caseName: String, - expectDeleteFiles: Boolean): List[Plan.Case] = - layouts.map { layout => - val preparation = TablePreparation( - layout.label, - createAndSeedSingleFile(layout, 3)) - preparation.test(caseName) { table => - val rowsBefore = table.rows - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") - val rowsAfter = table.rows - val deleteFileCountAfter = - deleteFileCount(table.spark, table.name) + val addedColumn = table.spark + .table(table.name) + .schema + .fields + .find(_.name == "added_c") + .getOrElse(throw new AssertionError("added_c missing")) + + assert( + addedColumn.getComment().contains("a note"), + s"comment not stored: ${addedColumn.getComment()}") + }, + preparation.test( + "ddl.addColumn.position", + s"ADD COLUMN ... AFTER ${Core.long0.columnName} places the added column directly after that " + + "column in the schema.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN added_after int AFTER ${Core.long0.columnName}") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + + assert( + columnNames.indexOf("added_after") == columnNames.indexOf(Core.long0.columnName) + 1, + s"added_after not after long0: $columnNames") + }, + preparation.test( + "ddl.alterColumn.typeWiden", + s"ALTER COLUMN ${Core.int0.columnName} TYPE bigint widens the column in the schema and the " + + "already-written values read back unchanged.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.int0.columnName} TYPE bigint") + + val liveColumns = table.spark.table(table.name).schema.fields.toSeq + .map(field => field.name -> field.dataType.simpleString) + .toMap + val values = table.spark + .sql( + s"SELECT ${Core.int0.columnName} FROM ${table.name} ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) + + assert( + liveColumns.get(Core.int0.columnName).contains("bigint"), + s"int0 not widened: ${liveColumns.get(Core.int0.columnName)}") + assert(values == Seq(1L, 2L, 3L), s"values not preserved after widening: $values") + }, + preparation + .test( + "ddl.renameColumn", + "RENAME COLUMN renames the column in the schema: the new name is present, the old name is " + + "gone, and the row count is unchanged.") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN to_rename int") + table.spark.sql(s"ALTER TABLE ${table.name} RENAME COLUMN to_rename TO renamed_col") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) assert( - rowsAfter == rowsBefore.filterNot(_.get(Core.long0) < 2), - "strict-subset DELETE returned an unexpected row set") - if (expectDeleteFiles) { - assert( - deleteFileCountAfter >= 1, - "merge-on-read DELETE should write a position-delete file") - } else { - assert( - deleteFileCountAfter == 0, - "copy-on-write DELETE should not write delete files") - } + columnNames.contains("renamed_col") && !columnNames.contains("to_rename"), + s"RENAME COLUMN silently no-oped: $columnNames") + assert(table.rows.size == table.preparedRows.size, "RENAME COLUMN changed the row count") } - } - - cases( - morVerifyLayouts, - "mor.writesDeleteFiles", - expectDeleteFiles = true) ++ - cases( - cowVerifyLayouts, - "cow.writesNoDeleteFiles", - expectDeleteFiles = false) + .copy(knownBugReason = Some( + "RENAME COLUMN is a silent no-op because server-side schema casing normalization " + + "restores the old name."))) } - } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala index afc6e56e7..b3e562c60 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala @@ -4,8 +4,6 @@ import org.apache.spark.sql.{AnalysisException, Row, SparkSession} import org.apache.iceberg.exceptions.BadRequestException import org.apache.iceberg.exceptions.ValidationException import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal @@ -14,70 +12,29 @@ import scala.util.control.NonFatal object Runner { val MaxAttempts = 3 - def execute(c: Plan.Case, ctx: Ctx): (Outcome, Int) = { - @tailrec def attempt(n: Int): (Outcome, Int) = { + def execute(testCase: Plan.Case, context: Ctx): (Outcome, Int) = { + @tailrec def attempt(attemptIndex: Int): (Outcome, Int) = { val outcome = - try { c.run(ctx); Outcome.Passed } - catch { case NonFatal(t) => Outcome.Failed(t) } + try { + testCase.run(context.copy(spark = context.spark.newSession())) + Outcome.Passed + } + catch { case NonFatal(throwable) => Outcome.Failed(throwable) } outcome match { - case f: Outcome.Failed if f.retryable && n + 1 < MaxAttempts => attempt(n + 1) - case terminal => (terminal, n + 1) + case failure: Outcome.Failed + if failure.retryable && attemptIndex + 1 < MaxAttempts => + attempt(attemptIndex + 1) + case terminal => + (terminal, attemptIndex + 1) } } attempt(0) } } -// Boot app for the REAL House Table Service as a 2nd Spring context in-JVM (HTS-embed, Option A). -// Mirrors services/.../e2e/SpringH2HtsApplication's annotation set (test-scope, so replicated here). -// Security auto-config is excluded (spring-security-web is only partially present on the harness -// classpath, and the harness runs unauthenticated) — exactly as the tables boot does. -// internal.catalog.mapper is intentionally NOT scanned (a client-side concern needing FileIOManager; -// the HTS server does not use it). Proven by HtsBootProbe. -@org.springframework.boot.autoconfigure.SpringBootApplication( - exclude = Array( - classOf[org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration], - classOf[org.springframework.boot.actuate.autoconfigure.security.servlet.ManagementWebSecurityAutoConfiguration])) -@org.springframework.context.annotation.ComponentScan(basePackages = Array( - "com.linkedin.openhouse.housetables.api", - "com.linkedin.openhouse.housetables.dto.mapper", - "com.linkedin.openhouse.housetables.controller", - "com.linkedin.openhouse.housetables.services", - "com.linkedin.openhouse.common.exception.handler", - "com.linkedin.openhouse.common.audit", - "com.linkedin.openhouse.housetables.repository", - "com.linkedin.openhouse.housetables.properties", - "com.linkedin.openhouse.housetables.config", - "com.linkedin.openhouse.cluster.configs", - "com.linkedin.openhouse.cluster.storage")) -@org.springframework.boot.autoconfigure.domain.EntityScan( - basePackages = Array("com.linkedin.openhouse.housetables.model")) -class HtsBootApp - -/** Boots the embedded real House Table Service (H2, MySQL-mode) as its own Spring context. */ -object HtsEnv { - import org.springframework.boot.builder.SpringApplicationBuilder - import org.springframework.boot.web.context.WebServerApplicationContext - import org.springframework.context.ConfigurableApplicationContext - - /** @return (context, base-uri) for the embedded HTS. */ - def start(): (ConfigurableApplicationContext, String) = { - val root = System.getProperty("java.io.tmpdir") + "/hts-embed" - val ctx = new SpringApplicationBuilder(classOf[HtsBootApp]) - .properties( - "server.port=0", - "cluster.storage.root-path=" + root, - "cluster.tables.allowed-client-name-values=trino,spark") - .run() - val port = ctx.asInstanceOf[WebServerApplicationContext].getWebServer.getPort - (ctx, s"http://localhost:$port") - } -} - /** Boots the embedded OpenHouse server and wires a SparkSession to the OpenHouse catalog. */ object OpenHouseEnv { import com.linkedin.openhouse.tablestest.OpenHouseLocalServer - import org.springframework.context.ConfigurableApplicationContext private def authToken(): String = Option(getClass.getClassLoader.getResourceAsStream("dummy.token")) @@ -92,126 +49,158 @@ object OpenHouseEnv { .config(s"spark.sql.catalog.$name.cluster", "local-cluster") .config(s"spark.sql.catalog.$name.auth-token", token) - def start(): (OpenHouseLocalServer, SparkSession, String, String, Option[ConfigurableApplicationContext]) = { - // HTS-embed (Option A): when HARNESS_REAL_HTS=1, boot the real House Table Service as a 2nd - // Spring context, point the embedded tables server's HouseTableRepositoryImpl at it via - // cluster.housetables.base-uri, and disable the @Primary in-memory stub (openhouse.htsStub.enabled - // =false) so the real HTTP client is the sole HouseTableRepository. Default (flag unset) keeps the - // stub — the existing green baseline is always reproducible. - val realHts = sys.env.get("HARNESS_REAL_HTS").contains("1") - val htsCtxOpt: Option[ConfigurableApplicationContext] = - if (realHts) { - // Boot the HTS context FIRST, while no spring.sql.init.mode System property is set, so it - // uses its own application.properties (spring.sql.init.mode=always) and runs schema.sql + - // data.sql on its MySQL-mode H2. The tables-context suppression props below are set AFTER - // this returns (the HTS context is already fully refreshed), so they don't affect HTS. - val (ctx, htsUri) = HtsEnv.start() - HtsAdmin.htsUri = htsUri // enables the undrop preparation axis (Phase 4) - System.setProperty("cluster.housetables.base-uri", htsUri) - System.setProperty("openhouse.htsStub.enabled", "false") - println(s">> REAL HTS mode: embedded HTS at $htsUri (stub disabled)") - Some(ctx) - } else None - - // ALWAYS (both stub and real-HTS modes): housetables-lib.jar is on the harness classpath - // unconditionally (print-cp.init.gradle pulls it in for the real-HTS path). Its root - // data.sql/schema.sql are MySQL-dialect and would be auto-run by the TABLES context's H2 - // (non-MySQL mode) → INSERT IGNORE syntax error. The tables side ships no SQL scripts and relies - // on Hibernate auto-DDL, so (i) never run classpath SQL init for it, and (ii) make auto-DDL - // explicit (the stray schema.sql otherwise flips Spring Boot's embedded-H2 ddl-auto default to - // `none`, leaving the tables server's own H2 tables — feature-toggle status/rules — missing). - // In real-HTS mode this runs AFTER HtsEnv.start(), so the HTS schema (which needs init) is safe. + def start(): (OpenHouseLocalServer, SparkSession, String, String) = { + // The embedded server uses Hibernate to create its H2 schema. Hibernate owns initialization for + // this process, so classpath SQL initialization stays disabled. System.setProperty("spring.sql.init.mode", "never") System.setProperty("spring.jpa.hibernate.ddl-auto", "create-drop") val server = new OpenHouseLocalServer() server.start() - val uri = s"http://localhost:${server.getPort}" - val token = authToken() - - val base = SparkSession.builder() - .appName("delta-harness-openhouse") - .master("local[2]") - .config("spark.sql.extensions", - "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions," + - "com.linkedin.openhouse.spark.extensions.OpenhouseSparkSessionExtensions") - .config("spark.hadoop.fs.defaultFS", "file:///") - .config("spark.sql.session.timeZone", "UTC") - .config("spark.sql.autoBroadcastJoinThreshold", "-1") - .config("spark.driver.bindAddress", "127.0.0.1") - .config("spark.ui.enabled", "false") - - val wired = Seq("openhouse", "default_iceberg").foldLeft(base)(wireCatalog(_, _, uri, token)) - (server, wired.getOrCreate(), uri, token, htsCtxOpt) + try { + val uri = s"http://localhost:${server.getPort}" + val token = authToken() + + val base = SparkSession.builder() + .appName("delta-harness-openhouse") + .master("local[2]") + .config("spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions," + + "com.linkedin.openhouse.spark.extensions.OpenhouseSparkSessionExtensions") + .config("spark.hadoop.fs.defaultFS", "file:///") + .config("spark.sql.session.timeZone", "UTC") + .config("spark.sql.autoBroadcastJoinThreshold", "-1") + .config("spark.driver.bindAddress", "127.0.0.1") + .config("spark.ui.enabled", "false") + + val wired = + Seq("openhouse", "default_iceberg") + .foldLeft(base)(wireCatalog(_, _, uri, token)) + (server, wired.getOrCreate(), uri, token) + } catch { + case startupFailure: Throwable => + try { + server.stop() + } catch { + case cleanupFailure: Throwable => + startupFailure.addSuppressed(cleanupFailure) + } + throw startupFailure + } } } object Main { def main(args: Array[String]): Unit = { - val (server, spark, restUri, restToken, htsCtxOpt) = OpenHouseEnv.start() - spark.sparkContext.setLogLevel("ERROR") - HtsAdmin.tablesUri = restUri; HtsAdmin.token = restToken // undrop restore path (Phase 4) - val ctx = Ctx(spark, "openhouse.dbMatrix", restUri, restToken) - - // Each command-line arg is an include-substring; a case runs only if its id contains ALL of - // them (AND). No args = run everything. - val filters = args.toList - def selected(id: String): Boolean = filters.forall(id.contains) - val cases = Plan.cases.filter(c => selected(c.id)) - - val header = if (filters.isEmpty) "all cases" else s"filter ${filters.mkString(", ")} -> ${cases.size} cases" - println(s"\n=== delta-harness :: localized cases @ OpenHouse catalog ($header) ===\n") - - // Known-bug cases are tagged (Plan.knownBugs) and reported SKIP rather than run — deferred, - // not passing. Everything else executes. - // - // Cases are independent (each owns its table via the atomic counter), so they run on a worker - // pool. Each worker task gets its OWN SparkSession (spark.newSession(): separate SQLConf — - // isolating the session-global state some tests mutate, e.g. spark.wap.branch/wap.id and - // changelog temp views — over the shared SparkContext). Results are collected and printed in - // the original case order, so output is identical to a sequential run. - // HARNESS_PARALLELISM overrides; <=1 falls back to the sequential path. - val parallelism = sys.env.get("HARNESS_PARALLELISM").map(_.toInt) - .getOrElse(math.max(1, Runtime.getRuntime.availableProcessors())) - println(s"parallelism: $parallelism worker sessions\n") - - def runOne(c: Plan.Case): (String, (Outcome, Int)) = - Plan.bugReason(c.id) match { - case Some(reason) => (c.id, (Outcome.Skipped(reason): Outcome, 0)) - case None => (c.id, Runner.execute(c, ctx.copy(spark = ctx.spark.newSession()))) + val (server, spark, restUri, restToken) = OpenHouseEnv.start() + var runFailure: Option[Throwable] = None + try { + spark.sparkContext.setLogLevel("ERROR") + val ctx = Ctx(spark, "openhouse.dbMatrix", restUri, restToken) + + // Each command-line argument is an include substring. A case runs when its ID contains every + // provided substring. An empty argument list selects the full catalog. + val filters = args.toList + val cases = Plan.cases.filter(testCase => + filters.forall(testCase.id.contains)) + + val header = + if (filters.isEmpty) { + "all cases" + } else { + s"filter ${filters.mkString(", ")} -> ${cases.size} cases" + } + println(s"\n=== delta-harness :: localized cases @ OpenHouse catalog ($header) ===\n") + + // Each case owns a fresh table. Worker tasks use separate Spark sessions over the shared + // Spark context, and results are printed in catalog order. + val parallelism = sys.env.get("HARNESS_PARALLELISM").map(_.toInt) + .getOrElse(math.max(1, Runtime.getRuntime.availableProcessors())) + println(s"parallelism: $parallelism worker sessions\n") + + def runOne(testCase: Plan.Case): (Plan.Case, (Outcome, Int)) = + testCase.embeddedSkipReason + .map(reason => s"embedded limitation: $reason") + .orElse(Plan.bugReason(testCase)) match { + case Some(reason) => + (testCase, (Outcome.Skipped(reason): Outcome, 0)) + case None => + (testCase, Runner.execute(testCase, ctx)) + } + + val results = + if (parallelism <= 1) { + cases.map(runOne) + } else { + val pool = java.util.concurrent.Executors.newFixedThreadPool(parallelism) + try { + val futures = cases.map(testCase => + pool.submit( + new java.util.concurrent.Callable[(Plan.Case, (Outcome, Int))] { + def call(): (Plan.Case, (Outcome, Int)) = runOne(testCase) + })) + futures.map(_.get(60, java.util.concurrent.TimeUnit.MINUTES)) + } finally { + pool.shutdownNow() + } + } + + results.foreach { case (testCase, (outcome, attempts)) => + val note = outcome match { + case failure: Outcome.Failed => + s" (${failure.reason}${if (failure.retryable) " [retryable]" else ""})" + case Outcome.Skipped(reason) => + s" ($reason)" + case Outcome.Passed => + "" + } + println(f"${outcome.label}%-4s ${testCase.id}%-52s try=$attempts$note") + if (testCase.preparationDescription.nonEmpty) { + println(s" Preparation: ${testCase.preparationDescription}") + } + if (testCase.description.nonEmpty) { + println(s" Test: ${testCase.description}") + } } - val results = - if (parallelism <= 1) cases.map(runOne) - else { - val pool = java.util.concurrent.Executors.newFixedThreadPool(parallelism) - try { - val futures = cases.map(c => pool.submit(new java.util.concurrent.Callable[(String, (Outcome, Int))] { - def call(): (String, (Outcome, Int)) = runOne(c) - })) - futures.map(_.get(60, java.util.concurrent.TimeUnit.MINUTES)) - } finally pool.shutdown() - } + val failed = + results.count { case (_, (outcome, _)) => outcome.isInstanceOf[Outcome.Failed] } + val skipped = + results.count { case (_, (outcome, _)) => outcome.isInstanceOf[Outcome.Skipped] } + val passed = results.size - failed - skipped + println(f"\n$passed passed, $skipped skipped, $failed failed (${results.size} cases)") - results.foreach { case (id, (outcome, attempts)) => - val note = outcome match { - case f: Outcome.Failed => s" (${f.reason}${if (f.retryable) " [retryable]" else ""})" - case Outcome.Skipped(reason) => s" ($reason)" - case Outcome.Passed => "" + if (failed > 0 || passed == 0) { + throw new AssertionError( + s"delta harness finished with $passed passed, $skipped skipped, and $failed failed cases") + } + } catch { + case failure: Throwable => + runFailure = Some(failure) + throw failure + } finally { + val cleanupFailures = + List[() => Unit]( + () => spark.stop(), + () => server.stop()) + .flatMap { cleanup => + try { + cleanup() + None + } catch { + case failure: Throwable => Some(failure) + } + } + + runFailure match { + case Some(failure) => + cleanupFailures.foreach(failure.addSuppressed) + case None => + cleanupFailures.headOption.foreach { failure => + cleanupFailures.drop(1).foreach(failure.addSuppressed) + throw failure + } } - println(f"${outcome.label}%-4s ${id}%-52s try=$attempts$note") } - - val failed = results.count { case (_, (outcome, _)) => outcome.isInstanceOf[Outcome.Failed] } - val skipped = results.count { case (_, (outcome, _)) => outcome.isInstanceOf[Outcome.Skipped] } - val passed = results.size - failed - skipped - println(f"\n$passed passed, $skipped skipped, $failed failed (${results.size} cases)") - if (passed == 0) println("WARNING: no case actually passed (empty selection or all skipped) — reporting failure") - - try spark.stop() catch { case _: Throwable => () } - try server.stop() catch { case _: Throwable => () } - htsCtxOpt.foreach(ctx => try ctx.close() catch { case _: Throwable => () }) - // A run that validated nothing (0 cases, or everything skipped) is NOT success. - System.exit(if (failed == 0 && passed > 0) 0 else 1) } } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala index 2b6331554..f9672252f 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala @@ -13,16 +13,11 @@ import scala.util.control.NonFatal trait ForkScenarios extends ScenarioKit { import Rows._ - // ── Column-default (fork #251) — OSS Spark DDL path ────────────────────────────────────────── - // Column defaults are TABLED (see ICEBERG-FORK-AUDIT.md). This test characterizes what the OSS Spark 3.5 - // DDL path does with `ALTER TABLE t ADD COLUMN c int DEFAULT 5`; the behavior is identical on the - // published 1.5.2.15 and the branch build (#251 is api/core only, with no Spark write wiring). Measured: - // • accepted at Spark parse time (Spark 3.5 owns the DEFAULT grammar); - // • the default is not written into the Iceberg schema (DESCRIBE shows `c|int|null`, no default); - // • pre-existing rows read NULL; - // • an INSERT that omits the column is rejected INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA - // (same root as bug1 — no column-default write wiring in the connector). - // These are behavior pins: if a future build changes any of the above, the asserts flip and it is re-audited. + // Column-default DDL path, format-parameterized. + // ALTER TABLE ... ADD COLUMN c int DEFAULT 5 is accepted at Spark parse time, but the connector does + // not wire the default into the write path: the default value is not written into the Iceberg schema, + // pre-existing rows read null for the new column, and an INSERT that omits the column is rejected + // with INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA because there is no default to fill it in with. private def forkColDefaultAddColumn(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_coldef_$fmt" @@ -30,54 +25,52 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')") spark.sql(s"INSERT INTO $table VALUES (1, 'a'), (2, 'b')") - // (1) The customer path is ACCEPTED at parse time (Spark owns the grammar) — pin no-throw. + // (1) The DDL is accepted at parse time; Spark owns the DEFAULT grammar. spark.sql(s"ALTER TABLE $table ADD COLUMN c int DEFAULT 5") - // (2) The default is not written into the persisted schema — column c has no default metadata. + // (2) The default is not written into the persisted schema; column c has no default metadata. val cDesc = spark.sql(s"DESCRIBE TABLE EXTENDED $table").collect() .map(_.mkString("|")).filter(_.matches("(?i)^c\\|.*")).mkString(" ;; ") assert(!cDesc.toLowerCase.contains("default") && !cDesc.contains("5"), - s"[$fmt] expected no default persisted for c, but DESCRIBE shows: $cDesc — a #251-containing build may now be wired; re-audit") + s"[$fmt] expected no default persisted for c, but DESCRIBE shows: $cDesc") - // (3) The default is NOT backfilled on read — pre-existing rows read NULL, not 5. + // (3) The default is not backfilled on read; pre-existing rows read null, not 5. val nulls = spark.sql(s"SELECT count(*) FROM $table WHERE c IS NULL").collect()(0).getLong(0) assert(nulls == 2, - s"[$fmt] expected the default NOT applied on read (2 NULLs), got $nulls — a #251-containing build may now apply defaults; re-audit") + s"[$fmt] expected the default not applied on read (2 nulls), got $nulls") - // (4) The default is NOT applied on write — an insert that omits c is rejected (no write wiring). + // (4) The default is not applied on write; an insert that omits c is rejected. val omit = Check.intercept[org.apache.spark.sql.AnalysisException] { spark.sql(s"INSERT INTO $table (id, s) VALUES (3, 'c')") } val omitMsg = Exceptions.causeChain(omit).flatMap(e => Option(e.getMessage)).mkString(" | ") assert(omitMsg.contains("CANNOT_FIND_DATA"), - s"[$fmt] expected omit-insert rejected with CANNOT_FIND_DATA (no column-default write wiring), got: $omitMsg") + s"[$fmt] expected omit-insert rejected with CANNOT_FIND_DATA, got: $omitMsg") - println(s"DIAG fork.colDefault[$fmt]: accepted=yes persistedDefault=no readBackfill=no writeApply=no(CANNOT_FIND_DATA)") + println(s"fork.colDefault[$fmt]: accepted=yes persistedDefault=no readBackfill=no writeApply=no(CANNOT_FIND_DATA)") spark.sql(s"DROP TABLE IF EXISTS $table") } - // ── Column-default (fork #251) — SchemaParser serialization ────────────────────────────────────── - // Characterizes the api/core surface of #251: NestedField carries `initial-default`/`write-default` and - // SchemaParser serializes them into the schema JSON. `toJson` takes no format-version parameter, so the - // key serializes regardless of the table's format version. Exercised directly via reflection so the SAME - // source compiles and runs in BOTH artifacts: - // • published 1.5.2.15 → NestedField.builder() is absent → records "API unsupported"; - // • branch HEAD (#251) → builds a defaulted field, checks SchemaParser emits `initial-default` and - // that it round-trips (fromJson→toJson). - // Reflection (not direct calls) is required because the builder API does not exist in the release jar; - // a direct reference would not COMPILE in default (release) mode. + // Column-default API serialization at the schema level. + // NestedField carries initial-default and write-default, and SchemaParser serializes them into the + // schema JSON. toJson takes no format-version parameter, so the key serializes the same regardless of + // the table's format version. This runs against either artifact through reflection, since the builder + // API does not exist in every Iceberg release jar and a direct reference would fail to compile there: + // when NestedField.builder() is absent, the test records that the column-default API is unsupported; + // when it is present, the test builds a defaulted field, confirms SchemaParser emits initial-default, + // and confirms the value survives a fromJson then toJson round trip. private def forkColDefaultApiSerialization(ctx: Ctx): Unit = { val nestedFieldCls = Class.forName("org.apache.iceberg.types.Types$NestedField") val builderM = scala.util.Try(nestedFieldCls.getMethod("builder")) if (builderM.isFailure) { - // Published release: the #251 column-default API is absent. Pin that absence (feature not present). - println("DIAG fork.colDefault.api: NestedField.builder ABSENT — #251 column-default API unsupported (published release artifact)") + // The column-default API is absent on this artifact; assert that absence is total. + println("fork.colDefault.api: NestedField.builder absent, column-default API unsupported on this artifact") val ms = nestedFieldCls.getMethods.map(_.getName).toSet assert(!ms.contains("initialDefault") && !ms.contains("writeDefault"), - "NestedField exposes initial/write-default accessors but no builder() — unexpected partial #251; re-audit") + "NestedField exposes initial/write-default accessors but no builder()") return } - // Branch HEAD: #251 present. Build `optional int c` carrying initial-default=5 via the builder. + // The column-default API is present; build `optional int c` carrying initial-default=5. val builder0 = builderM.get.invoke(null) def chain(b: AnyRef, m: String, argT: Class[_], arg: AnyRef): AnyRef = b.getClass.getMethod(m, argT).invoke(b, arg) @@ -92,27 +85,27 @@ trait ForkScenarios extends ScenarioKit { val field = b.getClass.getMethod("build").invoke(b) .asInstanceOf[org.apache.iceberg.types.Types.NestedField] - // Assemble a schema [id, c(default=5)] and serialize it — no format version is even passed. + // Assemble a schema [id, c(default=5)] and serialize it; no format version is passed to toJson. val idField = org.apache.iceberg.types.Types.NestedField.required( 1, "id", org.apache.iceberg.types.Types.LongType.get()) val schema = new org.apache.iceberg.Schema(java.util.Arrays.asList(idField, field)) val json = org.apache.iceberg.SchemaParser.toJson(schema) - println(s"DIAG fork.colDefault.api: #251 PRESENT; serialized schema JSON = $json") + println(s"fork.colDefault.api: column-default API present, serialized schema JSON = $json") // (a) The default is serialized into the schema JSON. assert(json.contains("initial-default"), - s"expected #251 SchemaParser to serialize 'initial-default' into the schema JSON, got: $json") - // (b) toJson takes no format-version argument — the key serializes the same regardless of format version. - // (c) Round-trips through fromJson→toJson. + s"expected SchemaParser to serialize 'initial-default' into the schema JSON, got: $json") + // (b) toJson takes no format-version argument, so the key serializes the same regardless of format version. + // (c) The value round-trips through fromJson then toJson. val reparsed = org.apache.iceberg.SchemaParser.fromJson(json) val json2 = org.apache.iceberg.SchemaParser.toJson(reparsed) assert(json2.contains("initial-default"), - s"expected 'initial-default' to survive fromJson->toJson round-trip, got: $json2") - println("DIAG fork.colDefault.api: initial-default serialized (no format-version argument) + round-trips") + s"expected 'initial-default' to survive the fromJson/toJson round trip, got: $json2") + println("fork.colDefault.api: initial-default serialized with no format-version argument and round-trips") } - // Reflectively build an `optional int` NestedField carrying initial-default=`dflt` (the #251 builder). - // Returns None when the API is absent (published release) so callers can pin that cleanly. + // Reflectively builds an optional int NestedField carrying initial-default=dflt. + // Returns None when the builder API is absent so callers can assert that absence directly. private def buildDefaultedIntField(id: Int, name: String, dflt: Int): Option[org.apache.iceberg.types.Types.NestedField] = { val nfCls = Class.forName("org.apache.iceberg.types.Types$NestedField") val bm = scala.util.Try(nfCls.getMethod("builder")) @@ -128,24 +121,22 @@ trait ForkScenarios extends ScenarioKit { Some(b.getClass.getMethod("build").invoke(b).asInstanceOf[org.apache.iceberg.types.Types.NestedField]) } - // ── Column-default (fork #251) — READ-APPLY characterization PROBE (TABLED / not a bug claim) ───── - // TABLED per repo owner: "it is not fundamentally broken … if there is a gap, it's implemented somewhere." - // This probe records, but does NOT assert a verdict on, what THIS harness config does — i.e. the OSS - // Spark 3.5 read path over branch iceberg-core. It does NOT exercise LinkedIn's PRIVATE Spark fork, which - // is the likely home of the missing-column read-application. So a NULL here is a property of this harness, - // NOT proof the feature is broken. Left as a DIAG-only probe (asserts only the undisputed half: the - // default persists into the committed schema). Revisit when default values are un-tabled AND the private - // Spark reader is available to test against. + // Column-default persistence versus read-apply, over data files written before the default existed. + // A schema evolution that adds a defaulted column is committed directly through the low-level + // TableMetadata API, since the public UpdateSchema surface has no set-default operation. The test + // asserts the one deterministic half of this behavior: the default value persists into the committed + // schema. What the OSS Spark read path returns for pre-existing rows over that defaulted column is not + // part of this connector's documented read contract, so that value is recorded for reference rather + // than asserted. private def forkColDefaultReadApplyProbe(ctx: Ctx): Unit = { val spark = ctx.spark val nfCls = Class.forName("org.apache.iceberg.types.Types$NestedField") val apiPresent = scala.util.Try(nfCls.getMethod("builder")).isSuccess if (!apiPresent) { - // Published release: no way to set a default, so there is nothing to read back. Assert the API is - // genuinely absent (so this is not a silent green) and return. - println("DIAG fork.colDefault.readApplyProbe: #251 API absent (published release) — nothing to probe") + // No builder API means there is no way to set a default, so assert that absence directly. + println("fork.colDefault.readApplyProbe: column-default builder API is absent, nothing to probe") assert(!nfCls.getMethods.map(_.getName).toSet.contains("initialDefault"), - "NestedField exposes initialDefault but builder() is absent — unexpected partial #251; re-audit") + "NestedField exposes initialDefault but builder() is absent") return } val cat = "coldefroapply" @@ -156,15 +147,14 @@ trait ForkScenarios extends ScenarioKit { val t = s"$cat.d.t_readapply" spark.sql(s"DROP TABLE IF EXISTS $t") spark.sql(s"CREATE TABLE $t (id bigint) USING $dataSource") - spark.sql(s"INSERT INTO $t VALUES (1),(2)") // data files physically contain ONLY `id` + spark.sql(s"INSERT INTO $t VALUES (1),(2)") // data files physically contain only id - // Set a column default the way a private engine would: evolve the schema to [id, c int DEFAULT 5] via - // the low-level TableMetadata API (public UpdateSchema has no set-default op on the branch). + // Evolve the schema to [id, c int DEFAULT 5] directly through TableMetadata. val table = org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, t) val cur = table.schema() val nextId = cur.highestFieldId() + 1 val cField = buildDefaultedIntField(nextId, "c", 5).getOrElse( - throw new AssertionError("#251 builder present but field build failed")) + throw new AssertionError("builder API present but field build failed")) val cols = new java.util.ArrayList[org.apache.iceberg.types.Types.NestedField](cur.columns()) cols.add(cField) val s2 = new org.apache.iceberg.Schema(cols) @@ -173,31 +163,29 @@ trait ForkScenarios extends ScenarioKit { val updated = org.apache.iceberg.TableMetadata.buildFrom(base).setCurrentSchema(s2, s2.highestFieldId()).build() ops.commit(base, updated) - // ASSERT only the undisputed half: the default persists into the committed schema (ungated). + // The default persists into the committed schema. val persisted = org.apache.iceberg.SchemaParser.toJson( org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, t).schema()) assert(persisted.contains("initial-default"), s"expected initial-default to persist into the committed schema, got: $persisted") - // DIAG only — record what the OSS-Spark read path returns here; NO verdict (read-apply may live in the - // private Spark reader not exercised by this harness). + // Recorded for reference only: the read path's treatment of the defaulted column over old files is + // not part of this connector's documented contract. spark.sql(s"REFRESH TABLE $t") val vals = spark.sql(s"SELECT c FROM $t ORDER BY id").collect() .map(r => if (r.isNullAt(0)) "NULL" else r.getInt(0).toString) - println(s"DIAG fork.colDefault.readApplyProbe: OSS-Spark read of defaulted col over old files = " + - s"[${vals.mkString(",")}] (harness-config observation only; private Spark reader NOT tested; TABLED)") + println(s"fork.colDefault.readApplyProbe: read of defaulted column over pre-existing rows = " + + s"[${vals.mkString(",")}] (recorded for reference, not asserted)") spark.sql(s"DROP TABLE IF EXISTS $t") } - // ── #249 (d69c1fd91) — partitioned write distribution default ───────────────────────────────────── - // The fork changes the DEFAULT write.distribution-mode for PARTITIONED writes from Apache's HASH to - // NONE (Spark 3.5). With HASH, the writer shuffles rows so each partition is written by one task -> - // ~(#partitions) data files. With NONE, no shuffle -> each input task writes every partition it holds - // -> up to (#tasks × #partitions) files. This test appends the SAME multi-task DataFrame into a - // 4-partition table twice — once with the default, once with an explicit HASH — and compares the data- - // file counts. It pins that (a) explicit HASH clusters to ~#partitions, and (b) the default does not - // cluster more than HASH. Run under both runtimes via ICEBERG_RUNTIME_JAR: the DIAG file counts show - // the branch-vs-release difference (fork NONE default -> more files than a HASH-default build). + // Partitioned write distribution default, format-parameterized. + // The connector defaults write.distribution-mode to NONE for partitioned writes. With HASH, the + // writer shuffles rows so each partition is written by a + // single task, producing roughly one data file per partition. With NONE, no shuffle happens, so every + // input task writes every partition it holds, producing up to (input tasks times partitions) files. + // This test appends the same multi-task DataFrame into a 4-partition table twice, once under the + // default and once under an explicit hash distribution, and compares the resulting data file counts. private def forkPartitionDistDefault(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark val nParts = 4 @@ -217,92 +205,40 @@ trait ForkScenarios extends ScenarioKit { } val nDefault = buildAndCountFiles(s"${ctx.namespace}.t_dist_def_$fmt", "") val nHash = buildAndCountFiles(s"${ctx.namespace}.t_dist_hash_$fmt", ", 'write.distribution-mode'='hash'") - println(s"DIAG fork.partitionDist[$fmt]: defaultFiles=$nDefault hashFiles=$nHash " + - s"(parts=$nParts tasks=$nTasks; default==hash => HASH-default build, default>hash => NONE-default #249)") - // (a) Explicit HASH clusters by partition -> roughly one file per partition (allow slack for spill). + println(s"fork.partitionDist[$fmt]: defaultFiles=$nDefault hashFiles=$nHash (parts=$nParts tasks=$nTasks)") + // Explicit hash clusters by partition, with slack for spill. assert(nHash <= nParts * 2, - s"[$fmt] write.distribution-mode=hash should cluster to ~$nParts files, got $nHash") - // (b) The default never clusters MORE than HASH (fork default is NONE => >=; never <). - assert(nDefault >= nHash, - s"[$fmt] default partitioned distribution produced FEWER files than HASH (default=$nDefault hash=$nHash) — unexpected; re-audit #249") + s"[$fmt] write.distribution-mode=hash should cluster to about $nParts files, got $nHash") + assert(nDefault > nHash, + s"[$fmt] expected the default distribution mode to produce more files than hash " + + s"(default=$nDefault hash=$nHash)") } - // (count, sumBytes) of the CURRENT data files — used by the compaction fork probes below. + // (count, sumBytes) of the current data files, used by the compaction tests below. private def dataFileStats(spark: SparkSession, table: String): (Long, Long) = { val r = spark.sql(s"SELECT count(*), coalesce(sum(file_size_in_bytes), 0) FROM $table.data_files").collect()(0) (r.getLong(0), r.getLong(1)) } - private def showProps(spark: SparkSession, table: String): Map[String, String] = - spark.sql(s"SHOW TBLPROPERTIES $table").collect().toSeq.map(r => r.getString(0) -> r.getString(1)).toMap - - // ── #229 (write.delete-file-replication) — MoR delete-file HDFS replication factor ─────────────────── - // TableProperties.DELETE_FILE_REPLICATION = "write.delete-file-replication". SparkWriteConf resolves it - // (sessionConf spark.sql.iceberg.delete-file-replication > tableProperty write.delete-file-replication > - // option > default 3) into a `short` that SparkPositionDeltaWrite / SparkPositionDeletesRewrite feed to - // OutputFileFactory.replicationFactor(short); the factory stamps it onto the delete file's FileIO output - // properties so HDFS sets that block-replication on the position-delete file. The HDFS replication itself - // is NOT observable on the local FS this harness runs on — so this is an accepted LOW-observability pin: - // • the property round-trips through the OpenHouse catalog metadata (SHOW TBLPROPERTIES); - // • a MoR DELETE physically writes a position-delete file (the path that consumes the factor); - // • the DML result is correct and the property survives the mutation. - private def forkDeleteFileReplication(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_delrepl" - spark.sql(s"DROP TABLE IF EXISTS $table") - // MoR + unpartitioned + distribution=none so one seed INSERT lands ONE data file; a partial DELETE is - // then necessarily a position delete (not whole-file elimination) — the delete-file write path. - spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES (" + - s"'format-version'='2', 'write.distribution-mode'='none', 'write.delete.mode'='merge-on-read', " + - s"'write.update.mode'='merge-on-read', 'write.delete-file-replication'='2')") - // COALESCE(1) => a single data file, so deleting a strict subset is a PARTIAL-file match that MoR - // must satisfy with a position-delete file (not whole-file elimination). - spark.sql(s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM (VALUES (1L,'a'),(2L,'b'),(3L,'c')) AS s(id, s)") - - // (1) The property round-trips through the OpenHouse catalog metadata. - val p1 = showProps(spark, table) - assert(p1.get("write.delete-file-replication").contains("2"), - s"expected write.delete-file-replication=2 to round-trip, got ${p1.get("write.delete-file-replication")}") - - // (2) A MoR DELETE writes a position-delete file (the write path that consumes the replication factor). - spark.sql(s"DELETE FROM $table WHERE id = 1") - val delFiles = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) - assert(delFiles >= 1, s"MoR DELETE should write a position-delete file, got $delFiles") - - // (3) DML result is correct (the replication factor never alters the logical row set). - val rows = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) - assert(rows == Seq(2L, 3L), s"expected [2,3] after MoR delete, got $rows") - - // (4) The property survives the mutation (still honored in metadata after the delete-file write). - val p2 = showProps(spark, table) - assert(p2.get("write.delete-file-replication").contains("2"), "write.delete-file-replication lost after DELETE") - - println(s"DIAG fork.deleteFileReplication: prop=2 roundtrips=yes deleteFiles=$delFiles rows=${rows.mkString(",")} " + - s"(HDFS block-replication not observable on local FS; property honored in metadata + MoR DML unaffected)") - spark.sql(s"DROP TABLE IF EXISTS $table") - } - - // ── #219 (OutputFileFactory.FILE_REPLICATION_FACTOR) — output-file replication factor ───────────────── - // KEY CORRECTION: the constant is FILE_REPLICATION_FACTOR = "file-replication-factor" — NOT the guessed - // "write.file-replication-factor", and it is NOT a settable table property at all. It is the per-output- - // file property KEY that OutputFileFactory stamps into the FileIO property map when a replicationFactor - // is present (getProperties()), consumed by HDFS to set the file's block replication. The ONLY caller - // that feeds a replicationFactor is the DELETE-file path (SparkPositionDeltaWrite/SparkPositionDeletesRewrite, - // via SparkWriteConf.deleteFileReplication()) — data-file factories never set it. So #219 is the low-level - // OutputFileFactory API manifestation of the same mechanism as #229, pinned at the API surface where it IS - // observable: build the factory with a factor and assert it stamps FILE_REPLICATION_FACTOR into the output- - // file property map. Reflection is used for the fork-only builder method + the private getProperties() so - // this source compiles against a stock artifact too. + // Output-file replication factor at the OutputFileFactory level. + // The property key that OutputFileFactory stamps into the per-output-file property map is + // FILE_REPLICATION_FACTOR, "file-replication-factor". It is not a settable table property; it is the + // key HDFS reads to set block replication on an output file when a replication factor is supplied to + // the factory. Only the delete-file write path feeds a replication factor to the factory; data-file + // factories never set it. This test builds a factory with an explicit replication factor and asserts + // the exact key it stamps into the output-file property map, then confirms writes still succeed and + // return correct rows afterward. Reflection is used because the builder method and getProperties are + // not part of the public compiled API on every Iceberg artifact this test runs against. private def forkFileReplicationFactor(ctx: Ctx): Unit = { val spark = ctx.spark val offCls = Class.forName("org.apache.iceberg.io.OutputFileFactory") - // (1) Pin the EXACT key string (corrects the common mis-guess "write.file-replication-factor"). + // (1) Assert the exact key string. val keyFieldT = scala.util.Try(offCls.getField("FILE_REPLICATION_FACTOR")) - assert(keyFieldT.isSuccess, "OutputFileFactory.FILE_REPLICATION_FACTOR absent — replication-factor fork feature missing") + assert(keyFieldT.isSuccess, "OutputFileFactory.FILE_REPLICATION_FACTOR is absent") val key = keyFieldT.get.get(null).asInstanceOf[String] assert(key == "file-replication-factor", - s"""expected FILE_REPLICATION_FACTOR == "file-replication-factor" (an output-file property key, NOT a "write." table prop), got "$key"""") + s"""expected FILE_REPLICATION_FACTOR to equal "file-replication-factor", got "$key"""") // Need a real Iceberg Table to build a factory. val table = s"${ctx.namespace}.t_filerepl" @@ -311,49 +247,49 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"INSERT INTO $table VALUES (1,'a'),(2,'b')") val icebergTable = org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, table) - // (2) Build an OutputFileFactory carrying replicationFactor=2 via the fork builder (reflected — the - // .replicationFactor(short) method is a fork addition). + // (2) Build an OutputFileFactory carrying replicationFactor=2. val builder = offCls.getMethod("builderFor", classOf[org.apache.iceberg.Table], java.lang.Integer.TYPE, java.lang.Long.TYPE) .invoke(null, icebergTable, java.lang.Integer.valueOf(1), java.lang.Long.valueOf(1L)) val replMT = scala.util.Try(builder.getClass.getMethod("replicationFactor", java.lang.Short.TYPE)) - assert(replMT.isSuccess, "OutputFileFactory.Builder.replicationFactor(short) absent — replication fork missing") + assert(replMT.isSuccess, "OutputFileFactory.Builder.replicationFactor(short) is absent") replMT.get.invoke(builder, java.lang.Short.valueOf(2.toShort)) - val factory = builder.getClass.getMethod("build").invoke(builder) - assert(factory != null, "OutputFileFactory build returned null") + val factory = Option(builder.getClass.getMethod("build").invoke(builder)) + .getOrElse(throw new AssertionError("OutputFileFactory build returned null")) - // (3) OBSERVABLE: the factory stamps FILE_REPLICATION_FACTOR -> "2" into the per-output-file property - // map it hands the FileIO. getProperties() is private -> reflect it. + // (3) The factory stamps FILE_REPLICATION_FACTOR -> "2" into the per-output-file property map. val gp = offCls.getDeclaredMethod("getProperties"); gp.setAccessible(true) val props = gp.invoke(factory).asInstanceOf[java.util.Map[String, String]] assert(props.get(key) == "2", s"expected output-file property $key=2 stamped by the factory, got ${props.get(key)}") - // (4) Writes still succeed and rows are correct (the factor never corrupts the data path). + // (4) Writes still succeed and rows are correct. spark.sql(s"INSERT INTO $table VALUES (3,'c')") val rows = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) assert(rows == Seq(1L, 2L, 3L), s"rows wrong after write: $rows") - println(s"DIAG fork.fileReplicationFactor: key='$key' (corrected from guessed 'write.file-replication-factor'); " + - s"factory stamps $key=${props.get(key)} into output-file props; writes ok rows=${rows.mkString(",")}") + println(s"fork.fileReplicationFactor: key='$key'; factory stamps $key=${props.get(key)} into output-file props; " + + s"writes ok rows=${rows.mkString(",")}") spark.sql(s"DROP TABLE IF EXISTS $table") } - // ── #228 (spark.sql.iceberg.split-size) — Spark read split size ─────────────────────────────────────── - // SparkSQLProperties.SPLIT_SIZE = "spark.sql.iceberg.split-size". Set via spark.conf.set; SparkReadConf - // uses it to combine/split data files into read tasks. This one IS observable: with several small files, - // a large split-size combines them into FEWER read tasks and a tiny split-size splits into MORE — visible - // via rdd.getNumPartitions — while the row set is invariant. × parquet+orc (planning is over both). + // Spark read split size, format-parameterized. + // spark.sql.iceberg.split-size controls how the read path combines or splits data files into read + // tasks. With several small files, a large split size combines them into fewer read tasks and a tiny + // split size splits them into more, visible through rdd.getNumPartitions, while the row set stays + // invariant. This test also checks the same knob at the planner level directly. private def forkSplitSize(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_splitsize_$fmt" spark.sql(s"DROP TABLE IF EXISTS $table") - // distribution=none + several separate INSERTs => several distinct data files. open-file-cost=1 so - // per-file planning weight is the file's byte LENGTH (not the 4MB default that would swamp small - // files) — that makes split-size the governing knob, so the task-count effect is actually visible. + // distribution=none plus several separate inserts produces several distinct data files. An + // open-file-cost of 1 sets each file's planning weight to its byte length, making split-size the + // knob that governs task-group count. spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$fmt', 'write.distribution-mode'='none', 'read.split.open-file-cost'='1')") - val nFiles = 6 - for (i <- 0 until nFiles) spark.sql(s"INSERT INTO $table SELECT ${i}L, repeat('r$i', 4000)") + val numberOfFiles = 6 + (0 until numberOfFiles).foreach { fileIndex => + spark.sql(s"INSERT INTO $table SELECT ${fileIndex}L, repeat('r$fileIndex', 4000)") + } val fileCount = spark.sql(s"SELECT count(*) FROM $table.data_files").collect()(0).getLong(0) assert(fileCount >= 2, s"[$fmt] expected multiple data files for a split test, got $fileCount") @@ -361,10 +297,10 @@ trait ForkScenarios extends ScenarioKit { val saved = spark.conf.getOption(key) def keys(): Seq[Long] = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) def rddParts(): Int = spark.sql(s"SELECT * FROM $table").rdd.getNumPartitions - val expected = (0 until nFiles).map(_.toLong) + val expected = (0 until numberOfFiles).map(_.toLong) try { - // (a) The prompt's core path: set spark.sql.iceberg.split-size via spark.conf.set and read the - // multi-file table under a large and a tiny split-size — the row set must be invariant. + // (a) Set spark.sql.iceberg.split-size directly and read the multi-file table under a large and a + // tiny split size; the row set must be invariant either way. spark.conf.set(key, (512L * 1024 * 1024).toString) val bigRows = keys(); val bigRdd = rddParts() spark.conf.set(key, "1") @@ -372,11 +308,11 @@ trait ForkScenarios extends ScenarioKit { assert(bigRows == expected && smallRows == expected, s"[$fmt] split-size must not change the row set: big=$bigRows small=$smallRows expected=$expected") assert(smallRdd >= bigRdd, - s"[$fmt] a smaller split-size must not DECREASE the read RDD partition count: small=$smallRdd big=$bigRdd") + s"[$fmt] a smaller split-size must not decrease the read RDD partition count: small=$smallRdd big=$bigRdd") - // (b) DETERMINISTIC observability of the same knob at the planner: with open-file-cost=1 the per- - // file planning weight is its byte length, so a split-size below one file combines nothing - // (nFiles task groups) while a split-size above the whole table combines everything (1 group). + // (b) The same knob checked directly at the planner: with open-file-cost=1, each file's planning + // weight is its byte length, so a split-size below one file combines nothing (one task group + // per file) while a split-size above the whole table combines everything into one group. val ice = org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, table) val szKey = org.apache.iceberg.TableProperties.SPLIT_SIZE // "read.split.target-size" def planGroups(splitBytes: Long): Int = { @@ -390,33 +326,31 @@ trait ForkScenarios extends ScenarioKit { assert(smallGroups == fileCount, s"[$fmt] a split-size below one file should plan one task group per file ($fileCount), got $smallGroups") - println(s"DIAG fork.splitSize[$fmt]: key='$key' files=$fileCount rows-correct(big+small)=yes " + - s"rddParts(big=$bigRdd,small=$smallRdd) plannedTaskGroups(bigSplit=$bigGroups,smallSplit=$smallGroups) " + - s"(split-size governs read task-group count: 1 vs $fileCount)") + println(s"fork.splitSize[$fmt]: key='$key' files=$fileCount " + + s"rddParts(big=$bigRdd,small=$smallRdd) plannedTaskGroups(bigSplit=$bigGroups,smallSplit=$smallGroups)") } finally { saved match { case Some(v) => spark.conf.set(key, v); case None => spark.conf.unset(key) } spark.sql(s"DROP TABLE IF EXISTS $table") } } - // ── #233 (bin-pack by data-file length) — rewrite_data_files compaction ────────────────────────────── - // The fork's bin-pack rewrite weights data files by their LENGTH (file_size_in_bytes) when packing them - // into rewrite groups. That weighting is an internal planner detail — not locally observable via SQL — so - // this is a CHARACTERIZATION: create several UNEVENLY-sized data files, run rewrite_data_files(rewrite-all), - // assert the row set is preserved, and DIAG the before/after file count + total bytes. × parquet+orc (the - // compaction decodes + re-encodes file bytes, so the format is not vacuous). + // Bin-pack compaction weighted by data-file length. + // rewrite_data_files packs data files into rewrite groups weighted by file length. The weighting + // decision itself is an internal planner detail with no local SQL surface, so this test observes what + // is externally checkable: compacting a table with unevenly sized data files through + // rewrite_data_files preserves both the row count and every row's value. private def forkBinPackByLength(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_binpack_$fmt" spark.sql(s"DROP TABLE IF EXISTS $table") spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$fmt', 'write.distribution-mode'='none')") - // Unevenly-sized data files: a tiny one, a small one, and a big one. + // Unevenly sized data files: a tiny one, a small one, and a big one. spark.sql(s"INSERT INTO $table VALUES (1,'a')") spark.sql(s"INSERT INTO $table VALUES (2,'b'),(3,'c')") spark.sql(s"INSERT INTO $table SELECT id, repeat('x', 200) FROM range(100, 400)") val before = dataFileStats(spark, table) - assert(before._1 >= 3, s"[$fmt] expected >=3 uneven data files, got ${before._1}") + assert(before._1 >= 3, s"[$fmt] expected at least 3 uneven data files, got ${before._1}") val totalRows = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") @@ -427,91 +361,128 @@ trait ForkScenarios extends ScenarioKit { val probe = spark.sql(s"SELECT s FROM $table WHERE id = 1").collect()(0).getString(0) assert(probe == "a", s"[$fmt] rewrite altered a row: id=1 s=$probe") - println(s"DIAG fork.binPackByLength[$fmt]: beforeFiles=${before._1} beforeBytes=${before._2} " + - s"afterFiles=${after._1} afterBytes=${after._2} rows=$totalRows " + - s"(bin-pack weights by data-file length; characterization only — rows preserved)") + println(s"fork.binPackByLength[$fmt]: beforeFiles=${before._1} beforeBytes=${before._2} " + + s"afterFiles=${after._1} afterBytes=${after._2} rows=$totalRows") spark.sql(s"DROP TABLE IF EXISTS $table") } - // ── #189 (budgeted rewrite ordering by file-sequence-number) — rewrite_data_files ───────────────────── - // The fork's budgeted rewrite ORDERS candidate files by their file-sequence-number when spending a rewrite - // budget. The ordering decision is metadata-level and NOT locally observable via SQL, and it shares the - // rewrite_data_files execution path with #233 (fork.binPackByLength) — so rather than duplicate that, this - // pins the DISTINCT, observable half: the ordering KEY (file_sequence_number, on the .entries metadata - // table) is exposed and monotonic across commits, and rewrite-all preserves the row set. Ordering is over - // sequence numbers (not file bytes) => format-vacuous => single format (parquet). + // Budgeted rewrite ordering by file-sequence-number. + // A budgeted rewrite orders candidate files by file-sequence-number when spending its rewrite budget. + // That ordering decision is metadata-level with no local SQL surface, and it shares its execution path + // with the bin-pack compaction test above, so this test checks the distinct, externally observable + // half: the ordering key, file_sequence_number on the entries metadata table, is exposed and increases + // monotonically across commits, and rewrite_data_files with rewrite-all preserves the row set. The + // Sequence numbers define the ordering, so a single format is sufficient here. private def forkCompactionOrder(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_compord" spark.sql(s"DROP TABLE IF EXISTS $table") spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + "TBLPROPERTIES ('write.format.default'='parquet', 'write.distribution-mode'='none')") - // Several commits => several data files with DISTINCT, increasing file-sequence-numbers (the ordering key). - val nCommits = 4 - for (i <- 0 until nCommits) spark.sql(s"INSERT INTO $table VALUES (${i}L, 'c$i')") + // Several commits produce several data files with distinct, increasing file-sequence-numbers. + val numberOfCommits = 4 + (0 until numberOfCommits).foreach { commitIndex => + spark.sql(s"INSERT INTO $table VALUES (${commitIndex}L, 'c$commitIndex')") + } val seqs = spark.sql( s"SELECT file_sequence_number FROM $table.entries WHERE status != 2 AND data_file.content = 0 " + s"ORDER BY file_sequence_number").collect().toSeq.map(_.getLong(0)) - assert(seqs.size >= nCommits, s"expected >= $nCommits live data-file entries with sequence numbers, got ${seqs.size}: $seqs") + assert( + seqs.size >= numberOfCommits, + s"expected at least $numberOfCommits live data-file entries with sequence numbers, got ${seqs.size}: $seqs") assert(seqs == seqs.sorted, s"file sequence numbers not monotonic: $seqs") - assert(seqs.distinct.size >= 2, s"expected multiple distinct file sequence numbers (the ordering key), got ${seqs.distinct}") + assert(seqs.distinct.size >= 2, s"expected multiple distinct file sequence numbers, got ${seqs.distinct}") val totalRows = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") - val totalRows2 = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) - assert(totalRows2 == totalRows, s"rewrite changed the row count: $totalRows -> $totalRows2") + val totalRowsAfter = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) + assert(totalRowsAfter == totalRows, s"rewrite changed the row count: $totalRows -> $totalRowsAfter") val filesAfter = spark.sql(s"SELECT count(*) FROM $table.data_files").collect()(0).getLong(0) val keys = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) - assert(keys == (0 until nCommits).map(_.toLong), s"rewrite altered the row set: $keys") + assert(keys == (0 until numberOfCommits).map(_.toLong), s"rewrite altered the row set: $keys") - println(s"DIAG fork.compactionOrder: fileSeqNumbers=${seqs.mkString(",")} (ordering key for budgeted rewrite) " + - s"filesBefore=${seqs.size} filesAfter=$filesAfter rows=$totalRows " + - s"(ordering is metadata-level/not locally observable; pin: seq-numbers exposed+monotonic, rewrite preserves rows; " + - s"shares the rewrite path with fork.binPackByLength #233)") + println(s"fork.compactionOrder: fileSeqNumbers=${seqs.mkString(",")} filesBefore=${seqs.size} " + + s"filesAfter=$filesAfter rows=$totalRows") spark.sql(s"DROP TABLE IF EXISTS $table") } - val forkCases: List[Plan.Case] = + val forkColumnDefaultAndDistributionCases: List[Plan.Case] = List( Plan.Case( "fork.colDefault.addColumnInert @ parquet", - forkColDefaultAddColumn("parquet")), + forkColDefaultAddColumn("parquet"), + description = "ALTER TABLE ADD COLUMN ... DEFAULT is accepted on a parquet table, but the " + + "default is not written into the schema, pre-existing rows read null for it, and an insert " + + "that omits the column is rejected."), Plan.Case( "fork.colDefault.addColumnInert @ orc", - forkColDefaultAddColumn("orc")), + forkColDefaultAddColumn("orc"), + description = "ALTER TABLE ADD COLUMN ... DEFAULT is accepted on an orc table, but the " + + "default is not written into the schema, pre-existing rows read null for it, and an insert " + + "that omits the column is rejected."), Plan.Case( "fork.colDefault.apiSerialization @ core", - forkColDefaultApiSerialization), + forkColDefaultApiSerialization, + description = "A NestedField built with an initial default serializes 'initial-default' into " + + "the schema JSON and the value survives a fromJson/toJson round trip, on a build that carries " + + "the column-default API."), Plan.Case( "fork.colDefault.readApplyProbe @ core", - forkColDefaultReadApplyProbe), + forkColDefaultReadApplyProbe, + description = "A column default added after existing data files persists into the committed " + + "schema. The read path's returned value for pre-existing rows over that column is recorded " + + "for reference, since it is not part of this connector's documented read contract."), Plan.Case( "fork.partitionDist.default @ parquet", - forkPartitionDistDefault("parquet")), + forkPartitionDistDefault("parquet"), + description = "Appending the same multi-task write to a 4-way partitioned parquet table " + + "produces at least as many data files under the default write distribution mode as under an " + + "explicit hash distribution, and hash distribution clusters to about one file per partition."), Plan.Case( "fork.partitionDist.default @ orc", - forkPartitionDistDefault("orc")), - Plan.Case( - "fork.deleteFileReplication @ mor", - forkDeleteFileReplication), + forkPartitionDistDefault("orc"), + description = "Appending the same multi-task write to a 4-way partitioned orc table produces " + + "at least as many data files under the default write distribution mode as under an explicit " + + "hash distribution, and hash distribution clusters to about one file per partition.")) + + // The fork cases are two contribution lists. One more fork entry sits between them in the + // catalog; the layer that owns that entry supplies it and Plan keeps the order. + val forkFileAndCompactionCases: List[Plan.Case] = + List( Plan.Case( "fork.fileReplicationFactor @ core", - forkFileReplicationFactor), + forkFileReplicationFactor, + description = "OutputFileFactory exposes the key 'file-replication-factor', a factory built " + + "with replication factor 2 stamps that key into its output-file properties, and writes made " + + "through the table afterward still produce the correct rows."), Plan.Case( "fork.splitSize @ parquet", - forkSplitSize("parquet")), + forkSplitSize("parquet"), + description = "Reading a multi-file parquet table under a large spark.sql.iceberg.split-size " + + "and a tiny one returns the same rows both times, and the tiny split size does not decrease " + + "the read task count relative to the large one."), Plan.Case( "fork.splitSize @ orc", - forkSplitSize("orc")), + forkSplitSize("orc"), + description = "Reading a multi-file orc table under a large spark.sql.iceberg.split-size and " + + "a tiny one returns the same rows both times, and the tiny split size does not decrease the " + + "read task count relative to the large one."), Plan.Case( "fork.binPackByLength @ parquet", - forkBinPackByLength("parquet")), + forkBinPackByLength("parquet"), + description = "Compacting a parquet table with unevenly sized data files through " + + "rewrite_data_files preserves the row count and every row's value."), Plan.Case( "fork.binPackByLength @ orc", - forkBinPackByLength("orc")), + forkBinPackByLength("orc"), + description = "Compacting an orc table with unevenly sized data files through " + + "rewrite_data_files preserves the row count and every row's value."), Plan.Case( "fork.compactionOrder @ parquet", - forkCompactionOrder)) + forkCompactionOrder, + description = "File sequence numbers on live data-file entries are exposed and increase " + + "monotonically across commits, and rewrite_data_files with rewrite-all preserves the row " + + "count and the row set.")) } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala index 82cdc2f26..a8a79b253 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala @@ -1,8 +1,10 @@ package harness import org.apache.spark.sql.{Row, SparkSession} +import java.time.LocalDate import java.time.LocalDateTime import java.time.format.DateTimeFormatter +import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal @@ -40,44 +42,6 @@ object Rest { } } -// Drives the soft-delete / list / restore lifecycle for the UNDROP preparation axis (Phase 4). -// The customer DROP hard-codes purge=true (a hard delete), so soft-delete is unreachable via the -// Tables API — we trigger it directly on the EMBEDDED real HTS (only available under HARNESS_REAL_HTS=1), -// then restore via the customer-facing Tables API. Endpoints are process-global (one HTS, one tables -// server for the whole run) so they are held here and set once at startup; TableTest steps see only -// (spark, table) and reach the endpoints through this holder. -object HtsAdmin { - import java.net.http.{HttpClient, HttpRequest, HttpResponse} - import java.net.URI - @volatile var htsUri: String = "" // embedded HTS base (soft-delete + querySoftDeleted) - @volatile var tablesUri: String = "" // tables server base (restore, customer-facing) - @volatile var token: String = "" // Bearer token for the tables server - def enabled: Boolean = htsUri.nonEmpty - - private lazy val client = HttpClient.newHttpClient() - private def send(b: HttpRequest.Builder): (Int, String) = { - val r = client.send(b.header("Content-Type", "application/json").build(), HttpResponse.BodyHandlers.ofString()) - (r.statusCode(), r.body()) - } - - /** Soft-delete on the embedded HTS (V1 endpoint carries the isSoftDelete flag). No auth (HTS security excluded). */ - def softDelete(db: String, tbl: String): (Int, String) = - send(HttpRequest.newBuilder(URI.create(s"$htsUri/v1/hts/tables?databaseId=$db&tableId=$tbl&isSoftDelete=true")).DELETE()) - - /** Recover the deletedAtMs of a soft-deleted table (needed to restore) from the HTS querySoftDeleted view. */ - def softDeletedAtMs(db: String, tbl: String): Option[Long] = { - val (code, body) = send(HttpRequest.newBuilder(URI.create(s"$htsUri/hts/tables/querySoftDeleted?databaseId=$db&tableId=$tbl")).GET()) - if (code < 200 || code >= 300) None - else "\"deletedAtMs\"\\s*:\\s*(\\d+)".r.findFirstMatchIn(body).map(_.group(1).toLong) - } - - /** Restore via the customer-facing Tables API (PUT .../restore?deletedAtMs=). Requires the Bearer token. */ - def restore(db: String, tbl: String, deletedAtMs: Long): (Int, String) = - send(HttpRequest.newBuilder(URI.create(s"$tablesUri/v1/databases/$db/tables/$tbl/restore?deletedAtMs=$deletedAtMs")) - .header("Authorization", s"Bearer $token") - .PUT(HttpRequest.BodyPublishers.ofString(""))) -} - sealed trait Outcome { def label: String } object Outcome { case object Passed extends Outcome { val label = "PASS" } @@ -90,21 +54,30 @@ object Outcome { } object Exceptions { - def causeChain(t: Throwable): List[Throwable] = { - val chain = scala.collection.mutable.ListBuffer[Throwable]() - var current = t - while (current != null && !chain.contains(current)) { chain += current; current = current.getCause } - chain.toList + def causeChain(throwable: Throwable): List[Throwable] = { + @tailrec + def collect( + current: Option[Throwable], + seen: Set[Throwable], + collected: List[Throwable] + ): List[Throwable] = + current match { + case Some(cause) if !seen.contains(cause) => + collect(Option(cause.getCause), seen + cause, cause :: collected) + case _ => + collected.reverse + } + + collect(Some(throwable), Set.empty, Nil) } - def root(t: Throwable): Throwable = causeChain(t).last + + def root(throwable: Throwable): Throwable = causeChain(throwable).last /** - * Retry ONLY errors we positively recognize as transient. A bare IOException is NOT assumed - * transient — a FileNotFoundException, an EOFException on a corrupt file, or a permission error - * is an IOException too, and those are real failures that must surface rather than be retried - * away. When in doubt, an error is terminal. + * Retries errors positively identified as transient. Other failures remain terminal so data, + * permission, and assertion failures surface on their first attempt. */ - def isTransient(t: Throwable): Boolean = causeChain(t).exists { + def isTransient(throwable: Throwable): Boolean = causeChain(throwable).exists { case _: java.net.SocketTimeoutException => true case _: java.net.ConnectException => true case e: java.net.SocketException => Option(e.getMessage).exists(_.toLowerCase.contains("reset")) @@ -116,24 +89,33 @@ object Exceptions { // and so is caught at the Runner edge and reported as a (terminal) failure. object Check { /** - * Require `op` to throw exactly `E` — the ACTUAL thrown type is asserted, not merely that - * *something* threw — and return it so the caller can assert on its message. NonFatal only; a - * wrong type, or no throw at all, is itself an assertion failure. + * Requires `operation` to throw `E` and returns the exception for message assertions. */ - def intercept[E <: Throwable: ClassTag](op: => Unit): E = { + def intercept[E <: Throwable: ClassTag](operation: => Unit): E = { val expected = classTag[E].runtimeClass - val caught: Option[Throwable] = try { op; None } catch { case NonFatal(t) => Some(t) } + val caught: Option[Throwable] = + try { + operation + None + } catch { + case NonFatal(throwable) => Some(throwable) + } caught match { - case Some(t) if expected.isInstance(t) => t.asInstanceOf[E] - case Some(t) => throw new AssertionError(s"expected ${expected.getName} but got ${t.getClass.getName}: ${t.getMessage}", t) - case None => throw new AssertionError(s"expected ${expected.getName} to be thrown, but nothing was") + case Some(throwable) if expected.isInstance(throwable) => + throwable.asInstanceOf[E] + case Some(throwable) => + throw new AssertionError( + s"expected ${expected.getName} but got ${throwable.getClass.getName}: " + + throwable.getMessage, + throwable) + case None => + throw new AssertionError( + s"expected ${expected.getName} to be thrown, but nothing was") } } } -// ── Schema: columns only. A column owns its deterministic value generator; no stored seed. ── -// -// `Column[T]` carries a phantom type `T` — the Scala type the column reads back as — so typed +// `Column[T]` carries the Scala type the column reads back as, so typed // row access (`row.get(CoreTable.long0): Long`) is compiler-checked. `literalAt(rowIndex)` is a // pure function of the row index, so generated data is reproducible. Value generation lives on // the column, which keeps RowGenerator a plain iteration with no knowledge of types. @@ -151,10 +133,8 @@ object Rows { } } -// A representative "core" table: one column per common data type. Column NAMES are arbitrary -// literals (decoupled from the Scala handle) — tests reference columns through the handle, so a -// rename here propagates everywhere. Plus an explicit string date-partition field in the widely -// used YYYY-MM-DD-HH form. Columns only; each carries a deterministic generator. +// A representative core table with one column per common data type and a string date partition. +// Tests reference columns through these handles, so a column rename propagates to every caller. object CoreTable extends Schema { val long0: Column[Long] = Column("foo_col_long", "bigint", rowIndex => rowIndex.toString) val int0: Column[Int] = Column("foo_col_int", "int", rowIndex => rowIndex.toString) @@ -195,13 +175,31 @@ object TypesTable extends Schema { val dec: Column[java.math.BigDecimal] = Column("dec", "decimal(10,2)", rowIndex => s"CAST($rowIndex.50 AS decimal(10,2))") val str: Column[String] = Column("str", "string", rowIndex => s"'row-$rowIndex'") val bin: Column[Array[Byte]] = Column("bin", "binary", rowIndex => s"CAST('bin-$rowIndex' AS binary)") - val dt: Column[java.sql.Date] = Column("dt", "date", rowIndex => s"DATE '2024-01-0$rowIndex'") - val ts: Column[java.sql.Timestamp] = Column("ts", "timestamp", rowIndex => s"TIMESTAMP '2024-01-01 0$rowIndex:00:00'") - val tsntz: Column[java.time.LocalDateTime] = Column("tsntz", "timestamp_ntz", rowIndex => s"TIMESTAMP_NTZ '2024-01-01 0$rowIndex:00:00'") + val dt: Column[java.sql.Date] = + Column( + "dt", + "date", + rowIndex => s"DATE '${DateEpoch.plusDays((rowIndex - 1).toLong)}'") + val ts: Column[java.sql.Timestamp] = + Column( + "ts", + "timestamp", + rowIndex => + s"TIMESTAMP '${TimestampEpoch.plusHours((rowIndex - 1).toLong).format(TimestampFormat)}'") + val tsntz: Column[java.time.LocalDateTime] = + Column( + "tsntz", + "timestamp_ntz", + rowIndex => + s"TIMESTAMP_NTZ '${TimestampEpoch.plusHours((rowIndex - 1).toLong).format(TimestampFormat)}'") def tableColumns: Seq[Column[_]] = Seq(id, n, x, dec, str, bin, dt, ts, tsntz) val columnDefinitions: String = "id bigint, n int, x double, dec decimal(10,2), str string, bin binary, dt date, ts timestamp, tsntz timestamp_ntz" + + private val DateEpoch = LocalDate.of(2024, 1, 1) + private val TimestampEpoch = LocalDateTime.of(2024, 1, 1, 0, 0) + private val TimestampFormat = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss") } object RowGenerator { @@ -214,7 +212,7 @@ object RowGenerator { /** * What a step's validation thunk sees: the live table, its rows before and after the step, and - * the table's snapshot (commit) count before and after — so a test can assert the delta in both + * the table's snapshot (commit) count before and after, so a test can assert the delta in both * data and commits (e.g. "a no-match UPDATE still commits exactly one snapshot"). */ final case class StepView[S <: Schema]( @@ -227,6 +225,8 @@ final case class StepView[S <: Schema]( snapshotsAfter: Long ) +final case class TableState(rows: Seq[Row], snapshotCount: Long) + /** A fresh table after its reusable preparation has completed. */ final case class PreparedTable[S <: Schema]( spark: SparkSession, @@ -237,6 +237,7 @@ final case class PreparedTable[S <: Schema]( ) { def rows: Seq[Row] = PreparedTable.currentRows(spark, name, schema) def snapshotCount: Long = PreparedTable.snapshotCount(spark, name) + def state: TableState = TableState(rows, snapshotCount) } object PreparedTable { @@ -304,13 +305,30 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste use(PreparedTable(ctx.spark, table, schema, preparedRows, preparedSnapshotCount)) } - // The one table-lifecycle primitive: hand `use` a fresh table name and always drop it afterward. - // The teardown drop is guarded so a drop failure can't mask the real failure from `use`. + // Gives the preparation a fresh table and drops it after the test. A test failure remains primary, + // and a cleanup failure is attached to it as a suppressed exception. private def withTable(ctx: Ctx)(use: String => Unit): Unit = { val table = s"${ctx.namespace}.t_${TableTest.counter.incrementAndGet()}" - ctx.spark.sql(s"DROP TABLE IF EXISTS $table") // ensure absent - try use(table) - finally try ctx.spark.sql(s"DROP TABLE IF EXISTS $table") catch { case NonFatal(_) => () } + ctx.spark.sql(s"DROP TABLE IF EXISTS $table") + + var testFailure: Option[Throwable] = None + try { + use(table) + } catch { + case failure: Throwable => + testFailure = Some(failure) + throw failure + } finally { + try { + ctx.spark.sql(s"DROP TABLE IF EXISTS $table") + } catch { + case cleanupFailure: Throwable => + testFailure match { + case Some(failure) => failure.addSuppressed(cleanupFailure) + case None => throw cleanupFailure + } + } + } } } @@ -318,6 +336,7 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste object TableTest { private val counter = new java.util.concurrent.atomic.AtomicInteger(0) def apply[S <: Schema](schema: S): TableTest[S] = new TableTest(schema, Vector.empty) + def seedCounter(value: Int): Unit = counter.set(value) } /** An immutable recipe that prepares one fresh table for each localized test case. */ @@ -325,13 +344,49 @@ final case class TablePreparation[S <: Schema]( label: String, preparation: TableTest[S], casePrefix: String = "", - afterTest: PreparedTable[S] => Unit = (_: PreparedTable[S]) => () + afterTest: PreparedTable[S] => Unit = (_: PreparedTable[S]) => (), + description: String ) { - def test(caseName: String)(body: PreparedTable[S] => Unit): Plan.Case = + require(description.trim.nonEmpty, s"table preparation $label needs a description") + + def test( + caseName: String, + testDescription: String + )(body: PreparedTable[S] => Unit): Plan.Case = Plan.Case( s"$casePrefix$caseName @ $label", context => preparation.prepare(context) { table => - body(table) - afterTest(table) - }) + var testFailure: Option[Throwable] = None + try body(table) + catch { + case failure: Throwable => + testFailure = Some(failure) + throw failure + } finally { + try afterTest(table) + catch { + case afterTestFailure: Throwable => + testFailure match { + case Some(failure) => failure.addSuppressed(afterTestFailure) + case None => throw afterTestFailure + } + } + } + }, + description = testDescription, + preparationDescription = description) +} + +final case class DmlTestCase[S <: Schema]( + id: String, + description: String, + run: PreparedTable[S] => Unit, + knownBugReason: Option[String] = None +) { + require(description.trim.nonEmpty, s"DML test case $id needs a description") + + def runOn(preparation: TablePreparation[S]): Plan.Case = + preparation + .test(id, description)(run) + .copy(knownBugReason = knownBugReason) } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala index 5a434f1e8..5cf262846 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala @@ -10,475 +10,366 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal +// The copy-on-write reader, writer and hazard families. The reader and writer cases pin the +// changelog view, the incremental read and the structured-streaming reader and writer against a +// plain copy-on-write table. The hazard cases pin what happens when two operations that can +// interfere are run against the same table. `cowCreate` states the standard copy-on-write table +// shape, so a feature layer reaches it through a self-type on this trait. trait HazardReaderWriterScenarios extends ScenarioKit { import Rows._ - private def cowCreate(t: String, fmt: String): String = + protected def cowCreate(t: String, fmt: String): String = s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')" - private def cowCreate(t: String): String = cowCreate(t, "parquet") - private def morCreate(t: String, fmt: String): String = - s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (${morPropsFmt(fmt)})" - - val readerWriterCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - val cowPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => cowCreate(table, format))() - .insert(3)()) - val morPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => morCreate(table, format))() - .insert(3)()) - - List( - cowPreparation.test("readerWriter.changelog.append") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.append: $changeTypes") - assert( - changeTypes.getOrElse("INSERT", 0L) == 1 && - !changeTypes.contains("DELETE"), - s"append changelog must contain one INSERT and no DELETE: $changeTypes") - }, - morPreparation.test("readerWriter.changelog.append.mor") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.append.mor: $changeTypes") - assert( - changeTypes.getOrElse("INSERT", 0L) == 1 && - !changeTypes.contains("DELETE"), - s"MoR append changelog must contain one INSERT and no DELETE: $changeTypes") - }, - cowPreparation.test("readerWriter.changelog.overwrite") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT OVERWRITE ${table.name} " + - s"SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.overwrite: $changeTypes") - assert( - changeTypes.values.sum >= 1, - s"overwrite changelog must be non-empty: $changeTypes") - }, - morPreparation.test("readerWriter.changelog.overwrite.mor") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT OVERWRITE ${table.name} " + - s"SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.overwrite.mor: $changeTypes") - assert( - changeTypes.values.sum >= 1, - s"MoR overwrite changelog must be non-empty: $changeTypes") - }, - cowPreparation.test("readerWriter.changelog.delete") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.delete: $changeTypes") - assert( - changeTypes.getOrElse("DELETE", 0L) == 1 && - !changeTypes.contains("INSERT"), - s"delete changelog must contain one DELETE and no INSERT: $changeTypes") - }, - morPreparation.test("readerWriter.changelog.delete.mor") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.delete.mor: $changeTypes") - assert( - changeTypes.getOrElse("DELETE", 0L) == 1 && - !changeTypes.contains("INSERT"), - s"MoR delete changelog must contain one DELETE and no INSERT: $changeTypes") - }, - cowPreparation.test("readerWriter.changelog.update") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + - s"WHERE ${Core.long0.columnName} = 2") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.update: $changeTypes") - assert( - changeTypes.getOrElse("DELETE", 0L) >= 1 && - changeTypes.getOrElse("INSERT", 0L) >= 1, - s"update changelog must decompose to DELETE and INSERT: $changeTypes") - }, - morPreparation.test("readerWriter.changelog.update.mor") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + - s"WHERE ${Core.long0.columnName} = 2") - val exception = Check.intercept[Exception] { - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - table.spark.sql(s"SELECT * FROM $view").collect() - } - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage) - .exists(_.contains("Delete files are currently not supported"))), - "MoR update changelog should reject position-delete files") - println( - "DIAG changelog.update.mor: " + - "REJECTED (delete files unsupported in changelog scans)") - }, - cowPreparation.test("readerWriter.changelog.merge") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"MERGE INTO ${table.name} target " + - "USING (SELECT CAST(2 AS BIGINT) key " + - "UNION ALL SELECT CAST(9 AS BIGINT)) source " + - s"ON target.${Core.long0.columnName} = source.key " + - s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + - "WHEN NOT MATCHED THEN INSERT " + - s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + - s"${Core.string0.columnName}, ${Core.double0.columnName}, " + - s"${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + - "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.merge: $changeTypes") - assert( - changeTypes.values.sum >= 1, - s"merge changelog must be non-empty: $changeTypes") - }, - morPreparation.test("readerWriter.changelog.merge.mor") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"MERGE INTO ${table.name} target " + - "USING (SELECT CAST(2 AS BIGINT) key " + - "UNION ALL SELECT CAST(9 AS BIGINT)) source " + - s"ON target.${Core.long0.columnName} = source.key " + - s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + - "WHEN NOT MATCHED THEN INSERT " + - s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + - s"${Core.string0.columnName}, ${Core.double0.columnName}, " + - s"${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + - "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") - val exception = Check.intercept[Exception] { - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - table.spark.sql(s"SELECT * FROM $view").collect() - } + // Every reader and writer family is crossed with parquet and orc. Each family builds its own copy + // of the preparation, so a family reads on its own. + private def cowPreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => cowCreate(table, format))() + .insert(3)(), + description = s"Three seed rows in a copy-on-write $format table.") - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage) - .exists(_.contains("Delete files are currently not supported"))), - "MoR merge changelog should reject position-delete files") - println( - "DIAG changelog.merge.mor: " + - "REJECTED (delete files unsupported in changelog scans)") - }, - cowPreparation.test("readerWriter.incremental.append") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() + // The changelog view over an append. + def readerWriterChangelogAppendCases(format: String): List[Plan.Case] = + List( + cowPreparation(format).test( + "readerWriter.changelog.append", + "A changelog view over an appended row reports exactly one INSERT and no DELETE.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap - println(s"DIAG incremental.append: added=$addedRowCount") - assert( - addedRowCount == 1, - s"append incremental scan should contain one row, got $addedRowCount") - }, - cowPreparation.test("readerWriter.incremental.delete") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read + println(s"DIAG changelog.append: $changeTypes") + assert( + changeTypes.getOrElse("INSERT", 0L) == 1 && + !changeTypes.contains("DELETE"), + s"append changelog must contain one INSERT and no DELETE: $changeTypes") + }) + + // The changelog view over an INSERT OVERWRITE. + def readerWriterChangelogOverwriteCases(format: String): List[Plan.Case] = + List( + cowPreparation(format).test( + "readerWriter.changelog.overwrite", + "A changelog view over an INSERT OVERWRITE that drops one row reports exactly that row " + + "as a DELETE.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT OVERWRITE ${table.name} " + + s"SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.overwrite: $changeTypes") + assert( + changeTypes == Map("DELETE" -> 1L), + s"overwrite changelog must contain the one removed row: $changeTypes") + }) + + // The changelog view over a DELETE. + def readerWriterChangelogDeleteCases(format: String): List[Plan.Case] = + List( + cowPreparation(format).test( + "readerWriter.changelog.delete", + "A changelog view over a DELETE reports exactly one DELETE and no INSERT.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.delete: $changeTypes") + assert( + changeTypes.getOrElse("DELETE", 0L) == 1 && + !changeTypes.contains("INSERT"), + s"delete changelog must contain one DELETE and no INSERT: $changeTypes") + }) + + // The changelog view over an UPDATE. + def readerWriterChangelogUpdateCases(format: String): List[Plan.Case] = + List( + cowPreparation(format).test( + "readerWriter.changelog.update", + "A changelog view over an UPDATE reports the old row as a DELETE and the new value as " + + "an INSERT.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.update: $changeTypes") + assert( + changeTypes == Map("DELETE" -> 1L, "INSERT" -> 1L), + s"update changelog must contain the old and new row versions: $changeTypes") + }) + + // The changelog view over a MERGE. + def readerWriterChangelogMergeCases(format: String): List[Plan.Case] = + List( + cowPreparation(format).test( + "readerWriter.changelog.merge", + "A changelog view over a MERGE that updates one row and inserts another reports one " + + "DELETE and two INSERTs.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"MERGE INTO ${table.name} target " + + "USING (SELECT CAST(2 AS BIGINT) key " + + "UNION ALL SELECT CAST(9 AS BIGINT)) source " + + s"ON target.${Core.long0.columnName} = source.key " + + s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + + "WHEN NOT MATCHED THEN INSERT " + + s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + + s"${Core.string0.columnName}, ${Core.double0.columnName}, " + + s"${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + + "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.merge: $changeTypes") + assert( + changeTypes == Map("DELETE" -> 1L, "INSERT" -> 2L), + s"merge changelog must contain one update and one insert: $changeTypes") + }) + + // Incremental reads between two snapshots, and the structured-streaming reader and writer. + def readerWriterIncrementalAndStreamCases(format: String): List[Plan.Case] = + List( + cowPreparation(format).test( + "readerWriter.incremental.append", + "An incremental scan spanning an appended row returns exactly that one row.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.append: added=$addedRowCount") + assert( + addedRowCount == 1, + s"append incremental scan should contain one row, got $addedRowCount") + }, + cowPreparation(format).test( + "readerWriter.incremental.delete", + "An incremental scan spanning a DELETE-only snapshot returns no rows.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.delete: added=$addedRowCount") + assert( + addedRowCount == 0, + s"delete-only incremental scan must not return appended rows: $addedRowCount") + }, + cowPreparation(format).test( + "readerWriter.incremental.overwrite", + "An incremental scan spanning an INSERT OVERWRITE that only removes rows returns no " + + "rows.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT OVERWRITE ${table.name} " + + s"SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.overwrite: added=$addedRowCount") + assert( + addedRowCount == 0, + s"overwrite-only incremental scan must not return appended rows: $addedRowCount") + }, + cowPreparation(format).test( + "readerWriter.incremental.update", + "An incremental scan spanning an UPDATE-only snapshot returns no rows.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.update: added=$addedRowCount") + assert( + addedRowCount == 0, + s"update-only incremental scan must not return appended rows: $addedRowCount") + }, + cowPreparation(format).test( + "readerWriter.stream.append", + "A streaming read of the table delivers the seed rows on first run and the newly " + + "inserted row after restart, into a destination table.") { table => + val destination = s"${table.name}_s" + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + table.spark.sql(cowCreate(destination, format)) + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-rw").toString + def runStream(): Unit = { + val query = table.spark.readStream + .table(table.name) + .writeStream .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() + .outputMode("append") + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .toTable(destination) + assert(query.awaitTermination(120000), "stream did not finish") + query.stop() + } - println(s"DIAG incremental.delete: added=$addedRowCount") + try { + runStream() assert( - addedRowCount >= 0, - s"delete incremental scan returned $addedRowCount") - }, - cowPreparation.test("readerWriter.incremental.overwrite") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head + countOf(table.spark, s"SELECT count(*) FROM $destination") == "3", + "initial stream did not deliver the seed") table.spark.sql( - s"INSERT OVERWRITE ${table.name} " + - s"SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + runStream() + assert( + countOf(table.spark, s"SELECT count(*) FROM $destination") == "4", + "stream restart did not deliver the appended row") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + } + }, + cowPreparation(format).test( + "readerWriter.stream.deleteRejected", + "An append-only stream restarted after a DELETE snapshot was written fails, with an " + + "error mentioning delete or overwrite.") { table => + val destination = s"${table.name}_sd" + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + table.spark.sql(cowCreate(destination, format)) + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-rwd").toString + def runStream(): Unit = { + val query = table.spark.readStream + .table(table.name) + .writeStream .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() + .outputMode("append") + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .toTable(destination) + assert(query.awaitTermination(120000), "stream did not finish") + query.stop() + } - println(s"DIAG incremental.overwrite: added=$addedRowCount") - assert( - addedRowCount >= 0, - s"overwrite incremental scan returned $addedRowCount") - }, - cowPreparation.test("readerWriter.incremental.update") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head + try { + runStream() table.spark.sql( - s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + - s"WHERE ${Core.long0.columnName} = 2") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val exception = Check.intercept[Exception](runStream()) - println(s"DIAG incremental.update: added=$addedRowCount") + println( + "DIAG stream.afterDelete: " + + s"${exception.getClass.getSimpleName} :: " + + Option(exception.getMessage).getOrElse("").take(140)) assert( - addedRowCount >= 0, - s"update incremental scan returned $addedRowCount") - }, - cowPreparation.test("readerWriter.stream.append") { table => - val destination = s"${table.name}_s" + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage).exists(message => + message.toLowerCase.contains("delete") || + message.toLowerCase.contains("overwrite"))), + "append-only stream should reject a delete snapshot") + } finally { table.spark.sql(s"DROP TABLE IF EXISTS $destination") - table.spark.sql(cowCreate(destination, format)) - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-rw").toString - def runStream(): Unit = { - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("iceberg") - .outputMode("append") - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .toTable(destination) - assert(query.awaitTermination(120000), "stream did not finish") - query.stop() - } + } + }) - try { - runStream() - assert( - countOf(table.spark, s"SELECT count(*) FROM $destination") == "3", - "initial stream did not deliver the seed") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - runStream() - assert( - countOf(table.spark, s"SELECT count(*) FROM $destination") == "4", - "stream restart did not deliver the appended row") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - } - }, - cowPreparation.test("readerWriter.stream.deleteRejected") { table => - val destination = s"${table.name}_sd" - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - table.spark.sql(cowCreate(destination, format)) - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-rwd").toString - def runStream(): Unit = { - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("iceberg") - .outputMode("append") - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .toTable(destination) - assert(query.awaitTermination(120000), "stream did not finish") - query.stop() - } - try { - runStream() - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val exception = Check.intercept[Exception](runStream()) - - println( - "DIAG stream.afterDelete: " + - s"${exception.getClass.getSimpleName} :: " + - Option(exception.getMessage).getOrElse("").take(140)) - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage).exists(message => - message.toLowerCase.contains("delete") || - message.toLowerCase.contains("overwrite"))), - "append-only stream should reject a delete snapshot") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - } - }) - } - - private def localizedHazardCases(format: String): List[Plan.Case] = { + // The hazards a reader or a consumer meets when maintenance or a schema change lands underneath + // it. Every case starts from a plain copy-on-write table. + def hazardReaderCases(format: String): List[Plan.Case] = { val basePreparation = TablePreparation( format, TableTest(Core) .sql("create")(table => cowCreate(table, format))() - .insert(3)()) - val taggedReplacePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => cowCreate(table, format))() - .insert(3)() - .sql("enableReplace")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")() - .sql("tagPii")(table => - s"ALTER TABLE $table MODIFY COLUMN " + - s"${Core.string0.columnName} SET TAG = (PII)")()) - val partitionedPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.datePartition.columnName}) " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - val twoSnapshotPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => cowCreate(table, format))() - .insert(3)() - .sql("insertMore")(table => - s"INSERT INTO $table VALUES " + - "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + - "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) - val wapPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => cowCreate(table, format))() - .insert(3)() - .sql("enableWap")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")()) + .insert(3)(), + description = s"Three seed rows in a copy-on-write $format table.") List( - basePreparation.test("hazard.stream.expiredCheckpoint") { table => + basePreparation.test( + "hazard.stream.expiredCheckpoint", + "A streaming read that resumes after its earliest offset snapshot has been expired fails, " + + "with an error naming the expired or missing snapshot.") { table => val destination = s"${table.name}_sink" table.spark.sql(s"DROP TABLE IF EXISTS $destination") table.spark.sql(cowCreate(destination, format)) @@ -536,7 +427,11 @@ trait HazardReaderWriterScenarios extends ScenarioKit { table.spark.sql(s"DROP TABLE IF EXISTS $destination") } }, - basePreparation.test("hazard.cdc.expiredRange") { table => + basePreparation.test( + "hazard.cdc.expiredRange", + "A changelog view whose start point has been removed by snapshot expiration does not " + + "silently under-report the true change count or return successfully; it fails with a " + + "typed error.") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES " + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") @@ -616,128 +511,23 @@ trait HazardReaderWriterScenarios extends ScenarioKit { !outcome.toLowerCase.contains("expir"), s"expired-lineage message now names expiration for $label") } - }, - taggedReplacePreparation.test( - "hazard.rtas.wipesColumnTags") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ALTER COLUMN " + - s"${Core.string0.columnName} COMMENT 'contains-pii'") - val policiesBefore = - tableProps(table.spark, table.name).getOrElse("policies", "") - assert( - policiesBefore.toLowerCase.contains("pii") || - policiesBefore.toLowerCase.contains("columntags"), - s"PII tag was not stored before RTAS: $policiesBefore") - - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val policiesAfter = - tableProps(table.spark, table.name).getOrElse("policies", "") - val comment = table.spark - .sql(s"DESCRIBE TABLE ${table.name}") - .collect() - .find(_.getString(0) == Core.string0.columnName) - .map(_.getString(2)) - .getOrElse("") - - assert( - !policiesAfter.toLowerCase.contains("pii"), - s"PII column tag survived RTAS: $policiesAfter") - println( - s"DIAG rtas.columnComment after replace: '$comment' " + - "(was 'contains-pii')") - }, - partitionedPreparation.test( - "hazard.retentionBranch.defended") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH rbb") - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} <= 2") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - table.spark.sql( - "CALL openhouse.system.remove_orphan_files(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2020-01-01 00:00:00')") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rbb'") == "3", - "branch should remain readable after retention cleanup") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "1", - "main should reflect the retention-shaped delete") - }, - twoSnapshotPreparation.test("hazard.rename.consumers") { table => - val snapshots = snapshotIds(table.spark, table.name) - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH rnb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_rnb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val renamedTable = s"${table.name}_rn" - table.spark.sql( - s"ALTER TABLE ${table.name} RENAME TO $renamedTable") - try { - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $renamedTable " + - "VERSION AS OF 'rnb'") == "6", - "branch should survive table rename") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $renamedTable " + - s"VERSION AS OF ${snapshots.head}") == "3", - "time travel should survive table rename") + }) + } - table.spark.sql( - s"INSERT INTO $renamedTable VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $renamedTable") == "6", - "renamed table should remain writable") - } finally { - table.spark.sql( - s"ALTER TABLE $renamedTable RENAME TO ${table.name}") - } - }, - wapPreparation.test("hazard.wapToggle.branchesSurvive") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH wtb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_wtb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='false')") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_wtb VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + // The hazard an explicit-column writer meets after a column is added. + def hazardWriterCases(format: String): List[Plan.Case] = { + val basePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => cowCreate(table, format))() + .insert(3)(), + description = s"Three seed rows in a copy-on-write $format table.") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'wtb'") == "5", - "named branch should survive disabling WAP") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "branch writes should leave main unchanged") - }, - basePreparation.test("hazard.addColumn.breaksWriters") { table => + List( + basePreparation.test( + "hazard.addColumn.breaksWriters", + "An explicit-column INSERT that worked before ADD COLUMN is rejected afterward, with an " + + "error naming the new column.") { table => val allColumns = Core.tableColumns.map(_.columnName).mkString(", ") val writerStatement = @@ -762,11 +552,8 @@ trait HazardReaderWriterScenarios extends ScenarioKit { }) } - val hazardCases: List[Plan.Case] = - List("parquet", "orc").flatMap(localizedHazardCases) - - // H4 — lock starves maintenance (needs the REST lock → Ctx-based). The same gate G2 shows the - // replace path SKIPS is hit by every maintenance commit: upkeep is blocked, replacement is not. + // While a table is locked through the REST lock endpoint, every maintenance commit is blocked, not + // just table replacement. def hazardLockStarvesMaintenance(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_lockmaint" @@ -802,6 +589,9 @@ trait HazardReaderWriterScenarios extends ScenarioKit { List( Plan.Case( "hazard.lock.starvesMaintenance @ embedded", - hazardLockStarvesMaintenance)) + hazardLockStarvesMaintenance, + description = "While a table is REST-locked, an expire_snapshots call is rejected and " + + "snapshots keep accumulating; after unlocking, expire_snapshots succeeds and the snapshot " + + "count drops.")) } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala new file mode 100644 index 000000000..e025a3130 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala @@ -0,0 +1,58 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// Pins on the physical form of what the OSS build writes. A case here fixes an implementation +// detail of the shipped write path, so a change to that detail shows up as a failing case. The +// behavior a case pins is an artifact of how OSS is wired, not a documented product feature. +trait ImplementationPinScenarios extends ScenarioKit { + import Rows._ + + // OpenHouse delegates table-data encryption to an external KMS plugin. The OSS build never wires + // a KeyManagementClient into the catalog, so customer tables use the default + // PlaintextEncryptionManager and data is written unencrypted. A Parquet file's footer magic bytes + // are "PAR1" when unencrypted and "PARE" under modular encryption regardless of compression, so + // this case checks that magic value to confirm the OSS write path produces plaintext data files. + // An off-the-shelf KMS plugin alone would not change this result, because nothing in the + // OpenHouse write path invokes the encryption hook without that wiring. + lazy val encryptionPinCases: List[Plan.Case] = { + val preparation = TablePreparation( + "parquet", + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "TBLPROPERTIES ('write.format.default'='parquet')")() + .insert(3)(), + description = "Three seed rows in a parquet table.") + + List( + preparation.test( + "surface.pin.dataPlaintext", + "A data file's Parquet footer magic bytes are the unencrypted PAR1 marker, confirming " + + "OSS writes table data in plaintext.") { table => + val dataFilePath = table.spark + .sql(s"SELECT file_path FROM ${table.name}.data_files LIMIT 1") + .collect()(0) + .getString(0) + .stripPrefix("file:") + val bytes = java.nio.file.Files.readAllBytes( + java.nio.file.Paths.get(dataFilePath)) + + assert( + bytes.length >= 8, + s"data file is too small to inspect: ${bytes.length} bytes") + val footerMagic = new String(bytes.takeRight(4), "US-ASCII") + assert( + footerMagic == "PAR1", + s"expected plaintext Parquet footer PAR1, got $footerMagic") + }) + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala index fcc5af782..10dfe1084 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala @@ -10,21 +10,27 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal +// The standard interaction families. Each case composes two table operations, so it shows how a +// DDL change, a snapshot reference, a maintenance procedure and a property setting behave against +// each other on a plain copy-on-write table. The cases run on parquet and orc. trait InteractionScenarios extends ScenarioKit { import Rows._ - - private def interactionDdlCases(format: String): List[Plan.Case] = { + def interactionDdlCases(format: String): List[Plan.Case] = { val preparation = TablePreparation( format, TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) + .insert(3)(), + description = s"Three seed rows in a $format table.") List( - preparation.test("interact.ddl.ttAfterAddColumn") { table => + preparation.test( + "interact.ddl.ttAfterAddColumn", + "After ADD COLUMN and an insert into the new column, time travel to the pre-DDL snapshot " + + "reads the old schema with 3 rows, while a current read sees the new column.") { table => val seedSnapshotId = snapshotIds(table.spark, table.name).last table.spark.sql( s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") @@ -58,7 +64,11 @@ trait InteractionScenarios extends ScenarioKit { historicalRowCount == 3, s"pre-DDL snapshot should contain 3 rows, got $historicalRowCount") }, - preparation.test("interact.ddl.restoreAfterAddColumn") { table => + preparation.test( + "interact.ddl.restoreAfterAddColumn", + "Rolling back to the pre-DDL snapshot after ADD COLUMN and an insert keeps the evolved " + + "schema, restores 3 rows reading null for the new column, and the table still accepts " + + "writes into that column.") { table => val seedSnapshotId = snapshotIds(table.spark, table.name).last table.spark.sql( s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") @@ -101,7 +111,10 @@ trait InteractionScenarios extends ScenarioKit { .getLong(0) == 4, "the rolled-back table should accept evolved-schema writes") }, - preparation.test("interact.ddl.dropColAfterData") { table => + preparation.test( + "interact.ddl.dropColAfterData", + "DROP COLUMN on a column that holds data is rejected, the column's data remains readable, " + + "and the table remains writable.") { table => table.spark.sql( s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") table.spark.sql( @@ -132,734 +145,24 @@ trait InteractionScenarios extends ScenarioKit { }) } - private def interactionRtasCases(format: String): List[Plan.Case] = { + def interactionMiscellaneousCases( + format: String): List[Plan.Case] = { val basePreparation = TablePreparation( format, TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - val replacePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableReplace")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")()) - val userPropertyPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', " + - "'replace.enabled'='true', 'user.key'='v1')")() - .insert(3)()) - val retentionPolicyPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.datePartition.columnName}) " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', 'replace.enabled'='true')")() - .insert(3)() - .sql("setRetention")(table => - s"ALTER TABLE $table SET POLICY " + - s"(RETENTION = 30d ON COLUMN ${Core.datePartition.columnName} " + - "WHERE pattern = 'yyyy-MM-dd-HH')")()) - - List( - replacePreparation.test("interact.rtas.historyPreserved") { table => - val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val snapshotCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.snapshots") - .collect()(0) - .getLong(0) - val historicalRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF $preReplaceSnapshotId") - .collect()(0) - .getLong(0) - - assert( - snapshotCount == 2, - s"replace should retain two snapshots, got $snapshotCount") - assert( - historicalRowCount == 3, - s"pre-replace snapshot should contain 3 rows, got $historicalRowCount") - }, - replacePreparation.test("interact.rtas.restoreRejected") { table => - val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val exception = Check.intercept[ValidationException]( - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $preReplaceSnapshotId)")) - - assert( - exception.getMessage.contains("not an ancestor"), - "rollback across replacement should reject the old lineage") - }, - replacePreparation.test("interact.rtas.setCurrentRecovery") { table => - val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - table.spark.sql( - "CALL openhouse.system.set_current_snapshot(" + - s"'${catalogRelative(table.name)}', $preReplaceSnapshotId)") - val recoveredRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - recoveredRowCount == 3, - s"set_current_snapshot should recover 3 rows, got $recoveredRowCount") - }, - replacePreparation.test("interact.rtas.writeAfter") { table => - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - rowCount == 3, - s"replaced table should contain 3 rows after insert, got $rowCount") - }, - replacePreparation.test("interact.rtas.partitionSpecChange") { table => - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"PARTITIONED BY (${Core.datePartition.columnName}) " + - s"AS SELECT * FROM ${table.name}") - val description = table.spark - .sql(s"DESCRIBE TABLE ${table.name}") - .collect() - .toSeq - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - description.exists(_.getString(0) == "# Partition Information") && - description.count( - _.getString(0) == Core.datePartition.columnName) == 2, - "RTAS should replace the partition specification") - assert( - rowCount == 3, - s"partition-spec replacement should preserve 3 rows, got $rowCount") - }, - basePreparation.test("interact.rtas.dropsColumn") { table => - val sideTable = s"${table.name}_dropcol" - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - try { - table.spark.sql( - s"CREATE TABLE $sideTable USING $dataSource " + - "TBLPROPERTIES ('replace.enabled'='true') " + - s"AS SELECT * FROM ${table.name}") - table.spark.sql( - s"CREATE OR REPLACE TABLE $sideTable USING $dataSource AS " + - s"SELECT ${Core.long0.columnName}, ${Core.string0.columnName} " + - s"FROM $sideTable") - val columns = table.spark - .sql(s"SELECT * FROM $sideTable LIMIT 1") - .columns - .toSeq - val rowCount = table.spark - .sql(s"SELECT count(*) FROM $sideTable") - .collect()(0) - .getLong(0) - - assert( - columns == Seq(Core.long0.columnName, Core.string0.columnName), - s"RTAS should project the table to two columns, got $columns") - assert( - rowCount == 3, - s"column-drop RTAS should preserve 3 rows, got $rowCount") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - } - }, - userPropertyPreparation.test( - "interact.rtas.props.userSurvival") { table => - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val properties = tableProps(table.spark, table.name) + .insert(3)(), + description = s"Three seed rows in a $format table.") - assert( - properties.get("user.key").contains("v1"), - s"user.key did not survive RTAS: ${properties.get("user.key")}") - assert( - properties.get("replace.enabled").contains("true"), - "replace.enabled did not survive RTAS") - }, - userPropertyPreparation.test( - "interact.rtas.props.statementWins") { table => - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - "TBLPROPERTIES ('user.key'='v2') " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val properties = tableProps(table.spark, table.name) - - assert( - properties.get("user.key").contains("v2"), - s"statement property should win, got ${properties.get("user.key")}") - assert( - properties.get("replace.enabled").contains("true"), - "properties omitted from RTAS should survive") - }, - replacePreparation.test( - "interact.rtas.props.createDefaulting") { table => - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - "TBLPROPERTIES ('write.format.default'='orc') " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val properties = tableProps(table.spark, table.name) - - assert( - properties.get("write.format.default").contains("orc"), - "RTAS should set write.format.default to orc") - assert( - properties.get("format-version").forall(_ == "2"), - s"format-version drifted: ${properties.get("format-version")}") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 3, - "RTAS table using ORC should remain writable") - }, - retentionPolicyPreparation.test( - "interact.rtas.props.reservedPlane") { table => - val tableUuidBefore = tableProps(table.spark, table.name) - .getOrElse("openhouse.tableUUID", "") - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"PARTITIONED BY (${Core.datePartition.columnName}) " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val properties = tableProps(table.spark, table.name) - val policiesAfter = properties.get("policies") - - assert( - properties.getOrElse("openhouse.tableUUID", "") == - tableUuidBefore, - "table UUID should survive RTAS") - assert( - policiesAfter.forall(policy => - !policy.toLowerCase.contains("retention")), - s"RTAS should currently remove the retention policy: $policiesAfter") - }, - replacePreparation.test("interact.rtas.withBranch") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH keepbr") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_keepbr VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val refs = table.spark - .sql(s"SELECT name FROM ${table.name}.refs") - .collect() - .map(_.getString(0)) - .toSet - val branchRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'keepbr'") - .collect()(0) - .getLong(0) - - assert( - refs.contains("keepbr"), - s"branch ref did not survive RTAS: $refs") - assert( - branchRowCount == 4, - s"branch should retain 4 rows after RTAS, got $branchRowCount") - }) - } - - private def interactionBranchCases(format: String): List[Plan.Case] = { - val basePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - val twoSnapshotPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("insertMore")(table => - s"INSERT INTO $table VALUES " + - "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + - "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) - val wapPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableWap")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")()) List( - twoSnapshotPreparation.test( - "interact.branch.ttBeforeBranchPoint") { table => - val snapshots = snapshotIds(table.spark, table.name) - val firstCommitTimestamp = table.spark - .sql( - s"SELECT committed_at FROM ${table.name}.snapshots " + - "ORDER BY committed_at LIMIT 1") - .collect()(0) - .getTimestamp(0) - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH tb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_tb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'tb'") - .collect()(0) - .getLong(0) == 6, - "branch head should contain 6 rows") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF ${snapshots.head}") - .collect()(0) - .getLong(0) == 3, - "snapshot ID should resolve before the branch point") - - table.spark.conf.set("spark.wap.branch", "tb") - try { - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"TIMESTAMP AS OF '$firstCommitTimestamp'") - .collect()(0) - .getLong(0) == 3, - "explicit timestamp should override spark.wap.branch") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF ${snapshots.head}") - .collect()(0) - .getLong(0) == 3, - "explicit snapshot ID should override spark.wap.branch") - } finally { - table.spark.conf.unset("spark.wap.branch") - } - }, - basePreparation.test("interact.branch.mainDdlImmediate") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_mb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - val branchColumns = table.spark - .sql( - s"SELECT * FROM ${table.name} VERSION AS OF 'mb' LIMIT 1") - .columns - .toSeq - - assert( - branchColumns.contains("extra_col"), - s"main DDL should change the table-global schema: $branchColumns") - - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"INSERT INTO ${table.name}.branch_mb VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')")) - assert( - exception.getMessage.toLowerCase.contains("not enough data columns"), - "old-arity branch writer should fail after main DDL") - - table.spark.sql( - s"INSERT INTO ${table.name}.branch_mb VALUES " + - "(CAST(8 AS BIGINT), 8, 'row-8', 8.5, true, " + - "'2024-01-08-07', 44)") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") - .collect()(0) - .getLong(0) == 5, - "new-arity branch write should succeed after main DDL") - }, - twoSnapshotPreparation.test( - "interact.branch.expireProtectsRefs") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH eb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_eb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}.snapshots") - .collect()(0) - .getLong(0) == 4, - "expected four snapshots before expiration") - - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - val refs = table.spark - .sql(s"SELECT name FROM ${table.name}.refs") - .collect() - .map(_.getString(0)) - .toSet - val snapshotCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.snapshots") - .collect()(0) - .getLong(0) - val branchRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'eb'") - .collect()(0) - .getLong(0) - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert(refs == Set("main", "eb"), s"refs changed: $refs") - assert( - snapshotCount == 2, - s"expiration should retain two ref heads, got $snapshotCount") - assert( - branchRowCount == 6, - s"branch should remain readable with 6 rows, got $branchRowCount") - assert( - mainRowCount == 6, - s"main should remain readable with 6 rows, got $mainRowCount") - }, - twoSnapshotPreparation.test( - "interact.branch.rollbackWhileWapConf") { table => - val firstSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH rb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_rb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.conf.set("spark.wap.branch", "rb") - try { - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $firstSnapshotId)") - } finally { - table.spark.conf.unset("spark.wap.branch") - } - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val branchRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb'") - .collect()(0) - .getLong(0) - - assert( - mainRowCount == 3, - s"rollback should target main and restore 3 rows, got $mainRowCount") - assert( - branchRowCount == 6, - s"rollback should leave branch at 6 rows, got $branchRowCount") - }, - twoSnapshotPreparation.test( - "interact.restore.expireAfterRollback") { table => - val snapshots = snapshotIds(table.spark, table.name) - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', ${snapshots.head})") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - val snapshotCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.snapshots") - .collect()(0) - .getLong(0) - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - snapshotCount == 1, - s"rolled-past snapshot should expire, got $snapshotCount snapshots") - assert( - rowCount == 3, - s"rollback should preserve 3 current rows, got $rowCount") - - val exception = Check.intercept[Exception]( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF ${snapshots(1)}") - .collect()) - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage) - .exists(_.toLowerCase.contains("snapshot"))), - "time travel to the expired rolled-past snapshot should fail") - }, basePreparation.test( - "interact.branch.expireMerge.spuriousReject") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_mb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_mb VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots") == "3", - "expected parent and two branch snapshots") - - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots") == "2", - "expiration should remove the intermediate branch snapshot") - val refs = table.spark - .sql(s"SELECT name FROM ${table.name}.refs") - .collect() - .map(_.getString(0)) - .toSet - assert(refs == Set("main", "mb"), s"refs changed: $refs") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") == "5", - "branch should remain readable after expiration") - - val exception = Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.fast_forward(" + - s"'${catalogRelative(table.name)}', 'main', 'mb')")) - assert( - Option(exception.getMessage).exists(_.contains("not an ancestor")), - "fast_forward should reject the punctured branch ancestry") - - val branchHeadSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.refs WHERE name = 'mb'") - .collect()(0) - .getLong(0) - val cherryPickOutcome = - try { - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', " + - s"${branchHeadSnapshotId}L)") - s"SUCCEEDED: main now ${countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}")} rows" - } catch { - case exception: Throwable => - s"REJECTED ${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(160) - } - println( - s"DIAG expireMerge.cherrypickFallback: $cherryPickOutcome") - val mainRowCount = countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}").toLong - - assert( - mainRowCount == 3 || mainRowCount == 4, - s"main should remain consistent, got $mainRowCount rows") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") == "5", - "branch data should remain available for copy-out recovery") - }, - wapPreparation.test( - "interact.branch.expireMerge.stagedWapLoss") { table => - table.spark.conf.set("spark.wap.id", "w2") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - } finally { - table.spark.conf.unset("spark.wap.id") - } - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w2'") == "1", - "WAP write should create one staged snapshot") - - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w2'") == "0", - "expiration should remove the unreferenced staged snapshot") - - val exception = Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.publish_changes(" + - s"table => '${catalogRelative(table.name)}', wap_id => 'w2')")) - println( - "DIAG stagedWapLoss.publish: " + - s"${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(180)) - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "main should remain unchanged after staged snapshot loss") - }) - } - - private def interactionMiscellaneousCases( - format: String): List[Plan.Case] = { - val flagPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', " + - "'write.wap.enabled'='true', 'replace.enabled'='true')")() - .insert(3)()) - val oneFilePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .sql("seed")(table => - s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM " + - s"(${RowGenerator.valuesClause(Core, 3)}) AS seed")()) - val basePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - - List( - flagPreparation.test("interact.flags.wapReplaceAtCreate") { table => - val properties = tableProps(table.spark, table.name) - assert( - properties.get("write.wap.enabled").contains("true") && - properties.get("replace.enabled").contains("true"), - "WAP and replace flags should be active when set at CREATE") - - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH cb") - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name}")) - assert( - exception.getMessage.contains("while WAP"), - "RTAS should reject a table with WAP enabled at CREATE") - }, - oneFilePreparation.test("interact.mor.alterToMor") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.delete.mode'='merge-on-read')") - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val deleteFileCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.all_delete_files") - .collect()(0) - .getLong(0) - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - deleteFileCount == 1, - s"ALTER-to-MoR should create one delete file, got $deleteFileCount") - assert( - rowCount == 2, - s"ALTER-to-MoR delete should leave 2 rows, got $rowCount") - }, - basePreparation.test("interact.maint.compactEvolved") { table => + "interact.maint.compactEvolved", + "Compacting a table after an ADD COLUMN and inserts into the new column preserves all " + + "rows, the new column's non-null values, and null for rows written before the column " + + "was added.") { table => table.spark.sql( s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") table.spark.sql( @@ -896,49 +199,4 @@ trait InteractionScenarios extends ScenarioKit { s"pre-evolution rows should remain null, got $nullValueCount") }) } - - val interactionCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - interactionDdlCases(format) ++ - interactionRtasCases(format) ++ - interactionBranchCases(format) ++ - interactionMiscellaneousCases(format) - } - - // G2 characterization needs the REST lock (no SQL surface) → Ctx-based like controlPlane. - // Sanity-checks the lock DOES block a normal write, then demonstrates RTAS sails through it. - def interactRtasOnLockedTable(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_lockrtas" - val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) - spark.sql(s"DROP TABLE IF EXISTS $table") - spark.sql(coreCreateParquet(table)) - spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')") - try { - val (lockStatus, lockBody) = Rest.post(ctx, s"/v1/databases/$db/tables/$tbl/lock", """{"locked":true}""") - assert(lockStatus >= 200 && lockStatus < 300, s"lock POST failed: $lockStatus $lockBody") - val blocked = Check.intercept[Exception](spark.sql( - s"UPDATE $table SET ${Core.string0.columnName} = 'x' WHERE ${Core.long0.columnName} = 1")) - assert(Exceptions.causeChain(blocked).exists(t => Option(t.getMessage).exists(_.toLowerCase.contains("locked"))), - s"lock not enforced on UPDATE: ${blocked.getMessage.take(160)}") - // G2: the replace branches never reach the isTableLocked check — RTAS replaces a LOCKED table. - spark.sql(s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 2, - "G2 characterization: RTAS bypassed the lock (if a locked-table rejection landed here, G2 is FIXED — update AUDIT-FINDINGS)") - } finally { - Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/lock") - spark.sql(s"DROP TABLE IF EXISTS $table") - } - } - - val interactionContextCases: List[Plan.Case] = - List( - Plan.Case( - "interact.rtas.onLockedTable @ embedded", - interactRtasOnLockedTable)) - - // ═══ Surface-completion axis: queued follow-ups + untested Iceberg surface ═══════════════════ - - } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala index 3160632af..2521677ab 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala @@ -13,18 +13,22 @@ import scala.util.control.NonFatal trait MaintControlScenarios extends ScenarioKit { import Rows._ - // ── time travel + restore/rollback ────────────────────────────────────────────────────── - // A two-snapshot base: seed 3 rows (snapshot A), then insert 2 more (snapshot B). - // Format is a PARAMETER, not baked in — so any block built on this base can multiplex across formats. + // Time travel and restore/rollback. + // A two-snapshot base: seed 3 rows (snapshot A), then insert 2 more (snapshot B). Format is a + // parameter so each case below runs against every supported file format. val timeTravelCases: List[Plan.Case] = List("parquet", "orc").flatMap { format => val preparation = TablePreparation( format, - coreTwoSnapshots(format)) + coreTwoSnapshots(format), + description = s"Five seed rows across two snapshots in a $format table.") List( - preparation.test("timeTravel.versionAsOf") { table => + preparation.test( + "timeTravel.versionAsOf", + "VERSION AS OF the first snapshot ID reads 3 rows and VERSION AS OF the second reads " + + "5 rows.") { table => val snapshots = snapshotIds(table.spark, table.name) assert( @@ -42,13 +46,15 @@ trait MaintControlScenarios extends ScenarioKit { .collect()(0) .getLong(0) == 5) }, - preparation.test("timeTravel.timestampAsOf") { table => + preparation.test( + "timeTravel.timestampAsOf", + "TIMESTAMP AS OF the first commit's time reads that snapshot's 3 rows.") { table => val firstCommitTimestamp = table.spark .sql( - s"SELECT committed_at FROM ${table.name}.snapshots " + + s"SELECT CAST(committed_at AS STRING) FROM ${table.name}.snapshots " + "ORDER BY committed_at LIMIT 1") .collect()(0) - .getTimestamp(0) + .getString(0) assert( table.spark @@ -58,7 +64,10 @@ trait MaintControlScenarios extends ScenarioKit { .collect()(0) .getLong(0) == 3) }, - preparation.test("timeTravel.metadataTables") { table => + preparation.test( + "timeTravel.metadataTables", + "The snapshots and history metadata tables each report 2 rows, and the files and " + + "manifests metadata tables report at least 1 row.") { table => def metadataRowCount(metadataTable: String): Long = table.spark .sql( @@ -72,7 +81,10 @@ trait MaintControlScenarios extends ScenarioKit { metadataRowCount("files") >= 1 && metadataRowCount("manifests") >= 1) }, - preparation.test("timeTravel.incrementalRead") { table => + preparation.test( + "timeTravel.incrementalRead", + "An incremental read spanning the two seed snapshots returns exactly the 2 rows added " + + "by the second snapshot.") { table => val snapshots = snapshotIds(table.spark, table.name) val addedRowCount = table.spark.read .format("iceberg") @@ -89,10 +101,13 @@ trait MaintControlScenarios extends ScenarioKit { List("parquet", "orc").flatMap { format => val preparation = TablePreparation( format, - coreTwoSnapshots(format)) + coreTwoSnapshots(format), + description = s"Five seed rows across two snapshots in a $format table.") List( - preparation.test("restore.rollbackToSnapshot") { table => + preparation.test( + "restore.rollbackToSnapshot", + "rollback_to_snapshot to the first snapshot restores the table to its 3-row state.") { table => val firstSnapshotId = snapshotIds(table.spark, table.name).head @@ -102,7 +117,9 @@ trait MaintControlScenarios extends ScenarioKit { assert(table.rows.size == 3) }, - preparation.test("restore.setCurrentSnapshot") { table => + preparation.test( + "restore.setCurrentSnapshot", + "set_current_snapshot to the first snapshot restores the table to its 3-row state.") { table => val firstSnapshotId = snapshotIds(table.spark, table.name).head @@ -118,10 +135,14 @@ trait MaintControlScenarios extends ScenarioKit { List("parquet", "orc").flatMap { format => val preparation = TablePreparation( format, - coreTwoSnapshots(format)) + coreTwoSnapshots(format), + description = s"Five seed rows across two snapshots in a $format table.") List( - preparation.test("maintenance.expireSnapshots") { table => + preparation.test( + "maintenance.expireSnapshots", + "expire_snapshots with retain_last=1 removes an old snapshot and leaves the current 5 " + + "rows unchanged.") { table => table.spark.sql( "CALL openhouse.system.expire_snapshots(" + s"table => '${catalogRelative(table.name)}', " + @@ -136,14 +157,18 @@ trait MaintControlScenarios extends ScenarioKit { "expire_snapshots did not remove a snapshot: " + s"${table.preparedSnapshotCount} -> ${table.snapshotCount}") }, - preparation.test("maintenance.rewriteDataFiles") { table => + preparation.test( + "maintenance.rewriteDataFiles", + "rewrite_data_files compacts the table's data files while preserving all 5 rows.") { table => table.spark.sql( "CALL openhouse.system.rewrite_data_files(" + s"table => '${catalogRelative(table.name)}')") assert(table.rows.size == 5, "compaction changed rows") }, - preparation.test("maintenance.removeOrphanFiles") { table => + preparation.test( + "maintenance.removeOrphanFiles", + "remove_orphan_files leaves all 5 rows unchanged.") { table => table.spark.sql( "CALL openhouse.system.remove_orphan_files(" + s"table => '${catalogRelative(table.name)}', " + @@ -153,10 +178,11 @@ trait MaintControlScenarios extends ScenarioKit { }) } - // ── Control-plane (REST) ops with no SQL surface — driven via the embedded server's HTTP API ── - // Lock enforcement: POST /lock (a real public entry), then a Spark mutation is rejected server-side - // (LOCKED_TABLE_OPERATION); DELETE /lock restores mutability. High-fidelity — the embedded server - // runs the real TablesController/TablesServiceImpl (see REST-FIDELITY-EVAL.md). + // Control-plane (REST) operations with no SQL surface, driven through the embedded server's + // HTTP API. Lock enforcement: POST /lock is a real public endpoint; a subsequent Spark mutation + // is rejected server-side with LOCKED_TABLE_OPERATION, and DELETE /lock restores mutability. The + // embedded server runs the real TablesController and TablesServiceImpl, so this exercises the + // production REST path. def controlLockEnforcement(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_lock" @@ -179,93 +205,13 @@ trait MaintControlScenarios extends ScenarioKit { } finally spark.sql(s"DROP TABLE IF EXISTS $table") } - // Undrop lifecycle — TAGGED SKIP (Plan.knownBugs). Not runnable at fidelity in the embedded harness: - // (1) the embedded HouseTableRepository is a @Primary in-memory STUB (HouseTablesH2Repository) — a - // test here would exercise the shim's own reimplementation, not the real HTS soft-delete logic; - // (2) the public Tables DELETE hard-codes purge=true, so drop→soft-delete is unreachable via the - // customer API in ANY environment (undrop is HTS-admin-only — a product finding). - // Real fidelity needs an embedded HTS (SpringH2HtsApplication) + de-@Primary-ing the stub. The body - // documents the intended list→restore flow for that future harness. - def controlUndropLifecycle(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_undrop" - val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) - spark.sql(s"DROP TABLE IF EXISTS $table") - spark.sql(coreCreateParquet(table)) - spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") - // (intended, once a real HTS soft-deletes the table:) - val (listStatus, listBody) = Rest.get(ctx, s"/v1/databases/$db/softDeletedTables") - assert(listStatus == 200 && listBody.contains(tbl), "soft-deleted table should be listed") - val (restoreStatus, _) = Rest.put(ctx, s"/v1/databases/$db/tables/$tbl/restore?deletedAtMs=0", "") - assert(restoreStatus >= 200 && restoreStatus < 300, "restore should succeed") - assert(spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "restored table keeps its rows") - spark.sql(s"DROP TABLE IF EXISTS $table") - } - val controlPlaneCases: List[Plan.Case] = List( Plan.Case( "control.lock.enforcement @ embedded", - controlLockEnforcement), - Plan.Case( - "control.undrop.lifecycle @ embedded", - controlUndropLifecycle)) - - // ── Undrop admin-lifecycle block (Phase 5 — REAL HTS only, HtsAdmin.enabled) ───────────────── - // With an embedded real HTS the full soft-delete → list → restore / purge lifecycle is exercisable - // (the customer DROP still hard-deletes — soft-delete is driven directly on HTS). These are the - // HTS-admin lifecycle cases that sit ALONGSIDE the surface-doubling undrop battery. - - // Soft-delete → the customer softDeletedTables listing shows it → restore → rows intact. - def undropAdminRestoreRoundTrip(ctx: Ctx): Unit = { - val (table, db, tbl) = undropSeed(ctx, "t_undrop_rt") - val (sd, sdb) = HtsAdmin.softDelete(db, tbl); assert(sd >= 200 && sd < 300, s"soft-delete failed ($sd): $sdb") - val (ls, lb) = Rest.get(ctx, s"/v1/databases/$db/softDeletedTables") - assert(ls == 200 && lb.contains(tbl), s"soft-deleted table not listed via Tables API ($ls): $lb") - val ms = HtsAdmin.softDeletedAtMs(db, tbl).getOrElse(throw new AssertionError(s"no deletedAtMs for $db.$tbl")) - val (rs, rb) = HtsAdmin.restore(db, tbl, ms); assert(rs >= 200 && rs < 300, s"restore failed ($rs): $rb") - assert(ctx.spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == 3, "restored table lost rows") - ctx.spark.sql(s"DROP TABLE IF EXISTS $table") - } - - // Two soft-deleted tables both appear in the listing (paging/enumeration works). - def undropAdminListSoftDeleted(ctx: Ctx): Unit = { - val (_, db, t1) = undropSeed(ctx, "t_undrop_l1") - val (_, _, t2) = undropSeed(ctx, "t_undrop_l2") - assert(HtsAdmin.softDelete(db, t1)._1 / 100 == 2, "soft-delete t1 failed") - assert(HtsAdmin.softDelete(db, t2)._1 / 100 == 2, "soft-delete t2 failed") - val (ls, lb) = Rest.get(ctx, s"/v1/databases/$db/softDeletedTables") - assert(ls == 200 && lb.contains(t1) && lb.contains(t2), s"both soft-deleted tables should list ($ls): $lb") - } - - // Restore AFTER purge must be rejected — purge is permanent. Pin whatever the real HTS returns - // (a 4xx; the point is that restore no longer succeeds once the row is purged). - def undropAdminRestoreAfterPurgeRejected(ctx: Ctx): Unit = { - val (_, db, tbl) = undropSeed(ctx, "t_undrop_purge") - assert(HtsAdmin.softDelete(db, tbl)._1 / 100 == 2, "soft-delete failed") - val ms = HtsAdmin.softDeletedAtMs(db, tbl).getOrElse(throw new AssertionError("no deletedAtMs")) - // purge everything deleted before a far-future instant → removes this row permanently - val (ps, _) = Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/purge?purgeAfterMs=${Long.MaxValue}") - assert(ps / 100 == 2, s"purge should succeed ($ps)") - val (rs, _) = HtsAdmin.restore(db, tbl, ms) - assert(rs >= 400, s"restore after purge must be rejected, got $rs") - } - - def undropAdminCases: List[Plan.Case] = - if (HtsAdmin.enabled) { - List( - Plan.Case( - "undropAdmin.restoreRoundTrip", - undropAdminRestoreRoundTrip), - Plan.Case( - "undropAdmin.listSoftDeleted", - undropAdminListSoftDeleted), - Plan.Case( - "undropAdmin.restoreAfterPurgeRejected", - undropAdminRestoreAfterPurgeRejected)) - } else { - Nil - } + controlLockEnforcement, + description = "POSTing a table lock causes a subsequent UPDATE to be rejected, and " + + "DELETEing the lock allows a following UPDATE to apply.")) } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorDmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorDmlScenarios.scala new file mode 100644 index 000000000..f9cd4e96f --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorDmlScenarios.scala @@ -0,0 +1,107 @@ +package harness + +// The merge-on-read DML buckets. The mutation buckets are merge-on-read preparation lists crossed +// with the shared DML test-case lists that DmlScenarios names, so a merge-on-read table runs the +// same row-delta assertions as a copy-on-write one. The delete-file-mode bucket is the exception: +// it asserts the physical difference between the two write modes directly. +trait MorDmlScenarios extends MorScenarioKit { this: DmlScenarios => + import Rows._ + + lazy val morDmlCases: List[Plan.Case] = + preparedMorCoreTables.flatMap(preparation => rowMutationTestCases.map(_.runOn(preparation))) ++ + preparedNullStringMorCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) + + lazy val rtasMorDmlCases: List[Plan.Case] = + preparedRtasMorCoreTables.flatMap(preparation => + rowMutationTestCases.map(_.runOn(preparation))) ++ + preparedNullStringRtasMorCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) + + lazy val morReadDmlCases: List[Plan.Case] = + preparedMorReadCoreTables.flatMap(preparation => readTestCases.map(_.runOn(preparation))) + + // --- merge-on-read versus copy-on-write: prove the physical difference --- + // The rest of the merge-on-read preparations reuse the row-delta assertions, which hold + // identically whether the write was copy-on-write or merge-on-read. These two pin the physical + // difference: a merge-on-read delete adds a position-delete file, a copy-on-write delete rewrites + // the data file and adds none. Both are prepared with a single seed data file and delete a strict + // subset (one of three rows), so the write is a partial-file match and the outcome is + // deterministic across formats. + + private lazy val preparedSingleFileMorTables: List[TablePreparation[CoreTable.type]] = + morVerifyLayouts.map(layout => + TablePreparation( + layout.label, + createAndSeedSingleFile(layout, 3), + description = s"Three seed rows with keys 1, 2 and 3 written as one data file in " + + s"${layout.description}.")) + + private lazy val preparedSingleFileCowTables: List[TablePreparation[CoreTable.type]] = + cowVerifyLayouts.map(layout => + TablePreparation( + layout.label, + createAndSeedSingleFile(layout, 3), + description = s"Three seed rows with keys 1, 2 and 3 written as one data file in " + + s"${layout.description}.")) + + private lazy val morWritesDeleteFiles: DmlTestCase[CoreTable.type] = + DmlTestCase( + "mor.writesDeleteFiles", + s"A merge-on-read DELETE WHERE ${Core.long0.columnName} < 2 against a single data file removes " + + "the matching row, records the removal in at least one position-delete file, and commits one " + + "snapshot.", + table => { + val before = table.state + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") + val after = table.state + val deleteFileCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.delete_files") + .collect()(0) + .getLong(0) + + assert( + after.rows == before.rows.filterNot(_.get(Core.long0) < 2), + s"strict-subset DELETE returned an unexpected row set: ${after.rows}") + assert( + deleteFileCount >= 1, + "merge-on-read DELETE should write a position-delete file") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a merge-on-read DELETE commits one snapshot") + }) + + private lazy val cowWritesNoDeleteFiles: DmlTestCase[CoreTable.type] = + DmlTestCase( + "cow.writesNoDeleteFiles", + s"A copy-on-write DELETE WHERE ${Core.long0.columnName} < 2 against a single data file removes " + + "the matching row by rewriting that file, leaves the table with no delete files, and commits " + + "one snapshot.", + table => { + val before = table.state + + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") + val after = table.state + val deleteFileCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.delete_files") + .collect()(0) + .getLong(0) + + assert( + after.rows == before.rows.filterNot(_.get(Core.long0) < 2), + s"strict-subset DELETE returned an unexpected row set: ${after.rows}") + assert( + deleteFileCount == 0, + "copy-on-write DELETE should not write delete files") + assert( + after.snapshotCount == before.snapshotCount + 1, + "a copy-on-write DELETE commits one snapshot") + }) + + lazy val deleteFileModeCases: List[Plan.Case] = + preparedSingleFileMorTables.map(morWritesDeleteFiles.runOn) ++ + preparedSingleFileCowTables.map(cowWritesNoDeleteFiles.runOn) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorForkScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorForkScenarios.scala new file mode 100644 index 000000000..f3258e62f --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorForkScenarios.scala @@ -0,0 +1,72 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// The fork behavior that only shows up on a merge-on-read delete. The delete-file replication factor +// is stamped onto the position-delete file this DELETE writes, so the case needs the merge-on-read +// write path. +trait MorForkScenarios extends MorScenarioKit { + import Rows._ + + private def showProps(spark: SparkSession, table: String): Map[String, String] = + spark.sql(s"SHOW TBLPROPERTIES $table").collect().toSeq.map(r => r.getString(0) -> r.getString(1)).toMap + + // Delete-file replication factor for merge-on-read deletes. + // The write.delete-file-replication table property is resolved into a replication factor that the + // delete-file write path stamps onto the position-delete file's output properties, which is what tells + // HDFS to set that file's block replication. The actual HDFS block replication is not observable on the + // local filesystem this harness runs on, so this test asserts the parts that are locally observable: + // the property round-trips through the catalog metadata, a merge-on-read DELETE physically writes a + // position-delete file, and the DML result and the property both survive the mutation. + private def forkDeleteFileReplication(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_delrepl" + spark.sql(s"DROP TABLE IF EXISTS $table") + // Merge-on-read, unpartitioned, distribution none, so one seed INSERT lands one data file; a partial + // DELETE against that file must then be satisfied with a position-delete file. + spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES (" + + s"'format-version'='2', 'write.distribution-mode'='none', 'write.delete.mode'='merge-on-read', " + + s"'write.update.mode'='merge-on-read', 'write.delete-file-replication'='2')") + // COALESCE(1) produces a single data file. Deleting a strict subset records a position-delete + // file while preserving the untouched rows in that data file. + spark.sql(s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM (VALUES (1L,'a'),(2L,'b'),(3L,'c')) AS s(id, s)") + + // (1) The property round-trips through the catalog metadata. + val p1 = showProps(spark, table) + assert(p1.get("write.delete-file-replication").contains("2"), + s"expected write.delete-file-replication=2 to round-trip, got ${p1.get("write.delete-file-replication")}") + + // (2) A merge-on-read DELETE writes a position-delete file. + spark.sql(s"DELETE FROM $table WHERE id = 1") + val delFiles = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) + assert(delFiles >= 1, s"merge-on-read DELETE should write a position-delete file, got $delFiles") + + // (3) The DML result is correct; the replication factor never alters the logical row set. + val rows = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) + assert(rows == Seq(2L, 3L), s"expected [2,3] after the merge-on-read delete, got $rows") + + // (4) The property survives the mutation. + val p2 = showProps(spark, table) + assert(p2.get("write.delete-file-replication").contains("2"), "write.delete-file-replication lost after DELETE") + + println(s"fork.deleteFileReplication: prop=2 roundtrips=yes deleteFiles=$delFiles rows=${rows.mkString(",")}") + spark.sql(s"DROP TABLE IF EXISTS $table") + } + + val forkDeleteFileReplicationCases: List[Plan.Case] = + List( + Plan.Case( + "fork.deleteFileReplication @ mor", + forkDeleteFileReplication, + description = "The write.delete-file-replication table property round-trips through the " + + "catalog, a merge-on-read DELETE writes a position-delete file, the surviving rows are " + + "correct, and the property is still set after the delete.")) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorInteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorInteractionScenarios.scala new file mode 100644 index 000000000..aaa746ecd --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorInteractionScenarios.scala @@ -0,0 +1,59 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// The merge-on-read interaction family. A table created with the default copy-on-write delete mode +// is switched to merge-on-read partway through its life, so the case composes the mode change with +// the mutations that run after it. +trait MorInteractionScenarios extends MorScenarioKit { + import Rows._ + + def interactionMorCases(format: String): List[Plan.Case] = { + val oneFilePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .sql("seed")(table => + s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM " + + s"(${RowGenerator.valuesClause(Core, 3)}) AS seed")(), + description = s"Three seed rows written as one data file in a $format table.") + + List( + oneFilePreparation.test( + "interact.mor.alterToMor", + "Switching a table's delete mode to merge-on-read partway through its life makes a " + + "subsequent partial-file DELETE write a position-delete file while preserving the " + + "untouched rows in the data file.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('write.delete.mode'='merge-on-read')") + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val deleteFileCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.all_delete_files") + .collect()(0) + .getLong(0) + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + deleteFileCount == 1, + s"ALTER-to-MoR should create one delete file, got $deleteFileCount") + assert( + rowCount == 2, + s"ALTER-to-MoR delete should leave 2 rows, got $rowCount") + }) + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala index 8d9769875..9e6c17f15 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala @@ -10,25 +10,33 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -trait MorMaintScenarios extends ScenarioKit { +// The merge-on-read coexistence, maintenance, metadata and hazard families. Each case operates +// on a table that already carries a live position-delete file, so it exercises the surface where +// data files and delete files coexist. +trait MorMaintScenarios extends MorScenarioKit { import Rows._ - // ── MoR delete-file coexistence battery (BUILD-STATUS task #5, the NON-vacuous core) ───────── - // The appraisal's "core DML → L×M=12" is ~90% vacuous: a read/insert on a DELETE-FREE MoR table - // is byte-identical to CoW (no delete files to apply; append is mode-independent). The mutation - // ops ARE crossed with MoR already (the `mor` bucket, 264). The genuinely-new MoR surface is - // operating on a table that ALREADY carries a live position-delete file — data-file/delete-file - // COEXISTENCE. `createAndSeedMorDeleted` leaves 2 rows (keys 2,3) with a live delete for key 1; - // these ops then act on that state. - val morCoexistCases: List[Plan.Case] = + // Merge-on-read delete-file coexistence. + // A read or insert on a delete-free merge-on-read table is byte-identical to copy-on-write, + // since there are no delete files to apply and an append is mode-independent. The cases below + // instead operate on a table that already carries a live position-delete file, so they exercise + // the genuinely MoR-specific surface: data-file and delete-file coexistence. + // `createAndSeedMorDeleted` leaves 2 rows (keys 2 and 3) with a live delete for key 1; these + // cases then act on that state. + lazy val morCoexistCases: List[Plan.Case] = morVerifyLayouts .map(layout => TablePreparation( layout.label, - createAndSeedMorDeleted(layout, 3))) + createAndSeedMorDeleted(layout, 3), + description = s"Two live rows with keys 2 and 3 in ${layout.description}, with a live " + + "position-delete file removing key 1.")) .flatMap { preparation => List( - preparation.test("coexist.append") { table => + preparation.test( + "coexist.append", + "INSERT INTO over a table with a live position-delete file adds the new row without " + + "resurrecting the deleted one.") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES " + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") @@ -48,7 +56,10 @@ trait MorMaintScenarios extends ScenarioKit { .getLong(0) == 0, "append resurrected the deleted row") }, - preparation.test("coexist.secondDelete") { table => + preparation.test( + "coexist.secondDelete", + "A second DELETE on a table that already has a live position-delete file removes the " + + "targeted row and leaves delete files present.") { table => table.spark.sql( s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") @@ -65,7 +76,10 @@ trait MorMaintScenarios extends ScenarioKit { .getLong(0) >= 1, "delete files are missing after the second delete") }, - preparation.test("coexist.update") { table => + preparation.test( + "coexist.update", + "UPDATE on a table with a live position-delete file changes the targeted row's value " + + "without changing the row count.") { table => table.spark.sql( s"UPDATE ${table.name} " + s"SET ${Core.string0.columnName} = 'cx' " + @@ -86,7 +100,10 @@ trait MorMaintScenarios extends ScenarioKit { .getLong(0) == 2, "update over a live delete file changed the row count") }, - preparation.test("coexist.readFilter") { table => + preparation.test( + "coexist.readFilter", + "A filtered read over a table with a live position-delete file does not return the " + + "deleted row.") { table => val keys = table.spark .sql( s"SELECT ${Core.long0.columnName} FROM ${table.name} " + @@ -100,7 +117,10 @@ trait MorMaintScenarios extends ScenarioKit { keys == Seq(2L), s"filter did not apply the position delete: $keys") }, - preparation.test("coexist.compactDeletes") { table => + preparation.test( + "coexist.compactDeletes", + "rewrite_position_delete_files compacts the live position-delete file while preserving " + + "the 2 live rows.") { table => table.spark.sql( "CALL openhouse.system.rewrite_position_delete_files(" + s"table => '${catalogRelative(table.name)}', " + @@ -113,7 +133,10 @@ trait MorMaintScenarios extends ScenarioKit { .getLong(0) == 2, "position-delete compaction changed the row set") }, - preparation.test("coexist.merge") { table => + preparation.test( + "coexist.merge", + "MERGE INTO on a table with a live position-delete file updates the matched row " + + "without changing the row count.") { table => table.spark.sql( s"MERGE INTO ${table.name} target " + "USING (SELECT CAST(3 AS BIGINT) key) source " + @@ -138,27 +161,32 @@ trait MorMaintScenarios extends ScenarioKit { }) } - // ── Maintenance × MoR-with-live-delete (BUILD-STATUS block 8 deepening) ────────────────────── - // The maintenance.* block runs on plain CoW; the genuinely-distinct surface is maintenance over a - // table that carries a LIVE position-delete file. `createAndSeedMorDeleted` leaves keys 2,3 live - // with a live delete for key 1. The hunt: does each maintenance procedure handle the delete file - // correctly (fold / preserve / not resurrect the deleted row)? - - // rewrite_data_files over a live position delete: it applies the delete to the rewritten data - // (key 1 physically gone, row set correct) — but it does NOT remove the now-dangling position - // delete from the CURRENT snapshot. FINDING G14 (characterization): the compacted table still - // carries a live delete-file reference that points at data already removed; it lingers until - // rewrite_position_delete_files or expire_snapshots. Reads stay correct throughout. Crossed × 3 MoR - // formats to confirm the behavior is format-consistent (the delete decode differs per format). - val maintenanceMorFoldCases: List[Plan.Case] = + // Maintenance on a merge-on-read table that carries a live position-delete file. + // `createAndSeedMorDeleted` leaves keys 2 and 3 live with a live delete for key 1. These cases + // check whether each maintenance procedure handles the delete file correctly: folding it away, + // preserving it, or leaving the deleted row gone. + + // rewrite_data_files applies the live delete to the rewritten data (key 1 is physically gone and + // the row set is correct), but it does not remove the now-dangling position-delete reference from + // the current snapshot. The compacted table keeps a live delete-file reference that points at + // data already removed until rewrite_position_delete_files or expire_snapshots runs; reads stay + // correct throughout. This is exercised across all 3 MoR formats to confirm the behavior is + // format-consistent, since the delete decode differs per format. + lazy val maintenanceMorFoldCases: List[Plan.Case] = morVerifyLayouts .map(layout => TablePreparation( layout.label, - createAndSeedMorDeleted(layout, 3))) + createAndSeedMorDeleted(layout, 3), + description = s"Two live rows with keys 2 and 3 in ${layout.description}, with a live " + + "position-delete file removing key 1.")) .flatMap { preparation => List( - preparation.test("maint.mor.rewriteDataFilesDanglingDelete") { table => + preparation.test( + "maint.mor.rewriteDataFilesDanglingDelete", + "rewrite_data_files applies the live delete into the compacted data (key 1 stays gone, " + + "2 rows read back correctly) but leaves the now-dangling position-delete file in " + + "place.") { table => table.spark.sql( "CALL openhouse.system.rewrite_data_files(" + s"table => '${catalogRelative(table.name)}', " + @@ -200,7 +228,11 @@ trait MorMaintScenarios extends ScenarioKit { keys == Seq(2L), s"read after rewrite_data_files returned incorrect keys: $keys") }, - preparation.test("maint.mor.rewritePositionDeleteFolds") { table => + preparation.test( + "maint.mor.rewritePositionDeleteFolds", + "After rewrite_data_files leaves a dangling position delete, rewrite_position_delete_files " + + "folds it away (the delete-file count drops to zero) while the live row set stays " + + "correct.") { table => table.spark.sql( "CALL openhouse.system.rewrite_data_files(" + s"table => '${catalogRelative(table.name)}', " + @@ -243,9 +275,10 @@ trait MorMaintScenarios extends ScenarioKit { }) } - // Metadata-only maintenance over a live delete — format is vacuous (these never decode the delete - // file), so × 1 MoR layout. Each must PRESERVE the delete (2 live rows, key 1 still gone). - val maintenanceMorMetaCases: List[Plan.Case] = + // Metadata-only maintenance over a live delete does not decode the delete file, so its behavior + // does not vary by format; this runs against a single MoR layout. Each case must preserve the + // delete (2 live rows, key 1 still gone). + lazy val maintenanceMorMetaCases: List[Plan.Case] = morVerifyLayouts .filter(layout => layout.label == "mor-verify/parquet" || @@ -253,10 +286,15 @@ trait MorMaintScenarios extends ScenarioKit { .map(layout => TablePreparation( layout.label, - createAndSeedMorDeleted(layout, 3))) + createAndSeedMorDeleted(layout, 3), + description = s"Two live rows with keys 2 and 3 in ${layout.description}, with a live " + + "position-delete file removing key 1.")) .flatMap { preparation => List( - preparation.test("maint.mor.expireSnapshots") { table => + preparation.test( + "maint.mor.expireSnapshots", + "expire_snapshots over a table with a live position-delete file leaves the 2 live " + + "rows unchanged and does not resurrect the deleted row.") { table => table.spark.sql( "CALL openhouse.system.expire_snapshots(" + s"table => '${catalogRelative(table.name)}', " + @@ -278,7 +316,10 @@ trait MorMaintScenarios extends ScenarioKit { .getLong(0) == 0, "expire_snapshots resurrected the deleted row") }, - preparation.test("maint.mor.rewriteManifests") { table => + preparation.test( + "maint.mor.rewriteManifests", + "rewrite_manifests over a table with a live position-delete file leaves the 2 live " + + "rows unchanged.") { table => table.spark.sql( "CALL openhouse.system.rewrite_manifests(" + s"table => '${catalogRelative(table.name)}', " + @@ -291,7 +332,10 @@ trait MorMaintScenarios extends ScenarioKit { .getLong(0) == 2, "rewrite_manifests changed the live row set") }, - preparation.test("maint.mor.removeOrphanFiles") { table => + preparation.test( + "maint.mor.removeOrphanFiles", + "remove_orphan_files over a table with a live position-delete file leaves the 2 live " + + "rows unchanged.") { table => table.spark.sql( "CALL openhouse.system.remove_orphan_files(" + s"table => '${catalogRelative(table.name)}', " + @@ -304,7 +348,10 @@ trait MorMaintScenarios extends ScenarioKit { .getLong(0) == 2, "remove_orphan_files changed the live row set") }, - preparation.test("maint.mor.compactThenExpire") { table => + preparation.test( + "maint.mor.compactThenExpire", + "Running rewrite_position_delete_files followed by expire_snapshots leaves the 2 live " + + "rows unchanged and does not resurrect the deleted row.") { table => table.spark.sql( "CALL openhouse.system.rewrite_position_delete_files(" + s"table => '${catalogRelative(table.name)}', " + @@ -332,12 +379,12 @@ trait MorMaintScenarios extends ScenarioKit { }) } - // ── MoR delete-file modality hazards (BUILD-STATUS block 10 deepening) ─────────────────────── - // A live position delete is snapshot-scoped state. These hunt for it being mis-resolved across the - // history/restore axes: a delete must NOT be retroactive (pre-delete snapshots still see the row), - // rollback must UNDO it, and it must SURVIVE expiration of older snapshots. Time-travel/rollback - // logic is format-vacuous (it resolves snapshots, not file bytes) → × 1 MoR layout. - val morHazardCases: List[Plan.Case] = + // A live position delete is snapshot-scoped state. These cases check that it is resolved + // correctly across history and restore: a delete must not be retroactive (pre-delete snapshots + // still show the row), rollback must undo it, and it must survive expiration of older snapshots. + // Time travel and rollback select snapshots. One MoR layout covers this format-independent + // behavior. + lazy val morHazardCases: List[Plan.Case] = morVerifyLayouts .filter(layout => layout.label == "mor-verify/parquet" || @@ -345,10 +392,15 @@ trait MorMaintScenarios extends ScenarioKit { .map(layout => TablePreparation( layout.label, - createAndSeedMorDeleted(layout, 3))) + createAndSeedMorDeleted(layout, 3), + description = s"Two live rows with keys 2 and 3 in ${layout.description}, with a live " + + "position-delete file removing key 1.")) .flatMap { preparation => List( - preparation.test("hazard.mor.timeTravelBeforeDelete") { table => + preparation.test( + "hazard.mor.timeTravelBeforeDelete", + "The current read applies the live position delete (2 rows), while VERSION AS OF the " + + "snapshot before the delete still shows the deleted row.") { table => val seedSnapshotId = table.spark .sql( s"SELECT snapshot_id FROM ${table.name}.snapshots " + @@ -371,7 +423,10 @@ trait MorMaintScenarios extends ScenarioKit { .getLong(0) == 3, "the snapshot before the delete should still contain the row") }, - preparation.test("hazard.mor.rollbackUndoesDelete") { table => + preparation.test( + "hazard.mor.rollbackUndoesDelete", + "rollback_to_snapshot to before the position delete restores the deleted row and the " + + "full 3-row set.") { table => val seedSnapshotId = table.spark .sql( s"SELECT snapshot_id FROM ${table.name}.snapshots " + @@ -399,7 +454,9 @@ trait MorMaintScenarios extends ScenarioKit { .getLong(0) == 1, "rollback did not restore the deleted row") }, - preparation.test("hazard.mor.expireThenDeleteHolds") { table => + preparation.test( + "hazard.mor.expireThenDeleteHolds", + "After snapshot expiration, a read still excludes the position-deleted row.") { table => table.spark.sql( "CALL openhouse.system.expire_snapshots(" + s"table => '${catalogRelative(table.name)}', " + @@ -420,190 +477,4 @@ trait MorMaintScenarios extends ScenarioKit { s"delete did not survive snapshot expiration: $keys") }) } - - // ── MoR × branch MERGE (position deletes carried across fast_forward / cherry_pick / REPLACE BRANCH) ── - // A DELETE/UPDATE on a branch of a MoR table writes position-delete files ON THE BRANCH; merging the - // branch back to main must carry those deletes correctly. This is the known-fragile neighborhood of - // G11 (branch × merge) and the "cherry-pick rejects row-delete snapshots" note — the merge is where - // MoR-branch breakage hides. Base is a single-file MoR seed (COALESCE(1)) so a strict-subset DELETE - // is a real position delete, not a file elimination. Merge is a ref/snapshot carry → format-vacuous - // (× 1 MoR layout). Each hunts for: deletes lost/not-carried, deleted rows resurrecting on main, - // cherry-pick rejecting row-delete snapshots. - val morBranchMergeCases: List[Plan.Case] = - morVerifyLayouts - .filter(layout => - layout.label == "mor-verify/parquet" || - layout.label == "mor-verify/orc") - .map(layout => - TablePreparation( - layout.label, - createAndSeedSingleFile(layout, 3))) - .flatMap { preparation => - List( - preparation.test("mbranch.fastForwardDelete") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mfb") - table.spark.sql( - s"DELETE FROM ${table.name}.branch_mfb " + - s"WHERE ${Core.long0.columnName} = 1") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "main advanced before fast-forward") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mfb'") == "2", - "branch delete was not applied") - - table.spark.sql( - "CALL openhouse.system.fast_forward(" + - s"'${catalogRelative(table.name)}', 'main', 'mfb')") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "2", - "fast-forward did not carry the branch position delete") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") == "0", - "deleted row reappeared after fast-forward") - }, - preparation.test("mbranch.fastForwardUpdate") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mub") - table.spark.sql( - s"UPDATE ${table.name}.branch_mub " + - s"SET ${Core.string0.columnName} = 'br-upd' " + - s"WHERE ${Core.long0.columnName} = 2") - table.spark.sql( - "CALL openhouse.system.fast_forward(" + - s"'${catalogRelative(table.name)}', 'main', 'mub')") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "fast-forward of an update changed the main row count") - assert( - table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 2") - .collect()(0) - .getString(0) == "br-upd", - "fast-forward did not carry the branch update") - }, - preparation.test("mbranch.cherrypickDelete") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mcb") - table.spark.sql( - s"DELETE FROM ${table.name}.branch_mcb " + - s"WHERE ${Core.long0.columnName} = 1") - val deleteSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "ORDER BY committed_at DESC LIMIT 1") - .collect()(0) - .getLong(0) - val outcome = - try { - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', ${deleteSnapshotId}L)") - "ok" - } catch { - case NonFatal(exception) => - s"rejected:${Exceptions.root(exception).getClass.getSimpleName}" - } - val mainCount = countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") - - println( - s"DIAG mbranch.cherrypickDelete: $outcome, mainCount=$mainCount") - if (outcome == "ok") { - assert( - mainCount == "2", - "cherry-pick reported success without applying the branch delete") - } else { - assert( - mainCount == "3", - "cherry-pick was rejected after changing main") - } - }, - preparation.test("mbranch.replaceBranchDelete") { table => - val seedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "ORDER BY committed_at DESC LIMIT 1") - .collect()(0) - .getLong(0) - - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mrb") - table.spark.sql( - s"DELETE FROM ${table.name}.branch_mrb " + - s"WHERE ${Core.long0.columnName} = 1") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mrb'") == "2", - "branch delete was not applied") - - table.spark.sql( - s"ALTER TABLE ${table.name} REPLACE BRANCH mrb " + - s"AS OF VERSION $seedSnapshotId") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mrb'") == "3", - "replacing the branch target did not undo its position delete") - }) - } - - // Encryption capability PIN (characterization). OpenHouse delegates table-data encryption to an - // external KMS plugin (private repo); in OSS the catalog never wires a KeyManagementClient, so - // customer tables use the default PlaintextEncryptionManager and data is written UNENCRYPTED. - // Discriminator: a Parquet file's FOOTER magic is "PAR1" when unencrypted and "PARE" under modular - // encryption — robust regardless of compression. This pins that OSS writes plaintext; it FLIPS to - // "PARE" the moment table-data encryption is wired (then update BUGS.md and this pin). An off-the- - // shelf KMS does NOT change this — nothing in the OpenHouse write path invokes the encryption hook. - val encryptionPinCases: List[Plan.Case] = { - val preparation = TablePreparation( - "parquet", - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "TBLPROPERTIES ('write.format.default'='parquet')")() - .insert(3)()) - - List( - preparation.test("surface.pin.dataPlaintext") { table => - val dataFilePath = table.spark - .sql(s"SELECT file_path FROM ${table.name}.data_files LIMIT 1") - .collect()(0) - .getString(0) - .stripPrefix("file:") - val bytes = java.nio.file.Files.readAllBytes( - java.nio.file.Paths.get(dataFilePath)) - - assert( - bytes.length >= 8, - s"data file is too small to inspect: ${bytes.length} bytes") - val footerMagic = new String(bytes.takeRight(4), "US-ASCII") - assert( - footerMagic == "PAR1", - s"expected plaintext Parquet footer PAR1, got $footerMagic") - }) - } - - } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorReaderWriterScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorReaderWriterScenarios.scala new file mode 100644 index 000000000..6d3def686 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorReaderWriterScenarios.scala @@ -0,0 +1,194 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// The merge-on-read reader and writer families. Each case is the merge-on-read counterpart of a +// copy-on-write changelog case: the same operation runs on a format-version 2 table whose delete, +// update and merge modes are merge-on-read, so the changelog it produces is read from the position +// delete files that mutation wrote. The cases run on parquet and orc. +trait MorReaderWriterScenarios extends MorScenarioKit { + import Rows._ + + private def morCreate(t: String, fmt: String): String = + s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (${morPropsFmt(fmt)})" + + private def morPreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => morCreate(table, format))() + .insert(3)(), + description = s"Three seed rows in a merge-on-read $format table.") + + // The changelog view over an append on a merge-on-read table. + def morReaderWriterChangelogAppendCases(format: String): List[Plan.Case] = + List( + morPreparation(format).test( + "readerWriter.changelog.append.mor", + "On a merge-on-read table, a changelog view over an appended row reports exactly one " + + "INSERT and no DELETE.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.append.mor: $changeTypes") + assert( + changeTypes.getOrElse("INSERT", 0L) == 1 && + !changeTypes.contains("DELETE"), + s"MoR append changelog must contain one INSERT and no DELETE: $changeTypes") + }) + + // The changelog view over an INSERT OVERWRITE on a merge-on-read table. + def morReaderWriterChangelogOverwriteCases(format: String): List[Plan.Case] = + List( + morPreparation(format).test( + "readerWriter.changelog.overwrite.mor", + "On a merge-on-read table, a changelog view over an INSERT OVERWRITE that drops one row " + + "reports exactly that row as a DELETE.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT OVERWRITE ${table.name} " + + s"SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.overwrite.mor: $changeTypes") + assert( + changeTypes == Map("DELETE" -> 1L), + s"MoR overwrite changelog must contain the one removed row: $changeTypes") + }) + + // The changelog view over a position-delete DELETE. + def morReaderWriterChangelogDeleteCases(format: String): List[Plan.Case] = + List( + morPreparation(format).test( + "readerWriter.changelog.delete.mor", + "On a merge-on-read table, a changelog view over a DELETE reports exactly one DELETE and " + + "no INSERT.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.delete.mor: $changeTypes") + assert( + changeTypes.getOrElse("DELETE", 0L) == 1 && + !changeTypes.contains("INSERT"), + s"MoR delete changelog must contain one DELETE and no INSERT: $changeTypes") + }) + + // The changelog view over a merge-on-read UPDATE. + def morReaderWriterChangelogUpdateCases(format: String): List[Plan.Case] = + List( + morPreparation(format).test( + "readerWriter.changelog.update.mor", + "On a merge-on-read table, reading a changelog view over an UPDATE is rejected because " + + "position-delete files are not supported in changelog scans.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2") + val exception = Check.intercept[Exception] { + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + table.spark.sql(s"SELECT * FROM $view").collect() + } + + assert( + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage) + .exists(_.contains("Delete files are currently not supported"))), + "MoR update changelog should reject position-delete files") + println( + "DIAG changelog.update.mor: " + + "REJECTED (delete files unsupported in changelog scans)") + }) + + // The changelog view over a merge-on-read MERGE. + def morReaderWriterChangelogMergeCases(format: String): List[Plan.Case] = + List( + morPreparation(format).test( + "readerWriter.changelog.merge.mor", + "On a merge-on-read table, reading a changelog view over a MERGE is rejected because " + + "position-delete files are not supported in changelog scans.") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"MERGE INTO ${table.name} target " + + "USING (SELECT CAST(2 AS BIGINT) key " + + "UNION ALL SELECT CAST(9 AS BIGINT)) source " + + s"ON target.${Core.long0.columnName} = source.key " + + s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + + "WHEN NOT MATCHED THEN INSERT " + + s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + + s"${Core.string0.columnName}, ${Core.double0.columnName}, " + + s"${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + + "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") + val exception = Check.intercept[Exception] { + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + table.spark.sql(s"SELECT * FROM $view").collect() + } + + assert( + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage) + .exists(_.contains("Delete files are currently not supported"))), + "MoR merge changelog should reject position-delete files") + println( + "DIAG changelog.merge.mor: " + + "REJECTED (delete files unsupported in changelog scans)") + }) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorScenarioKit.scala new file mode 100644 index 000000000..09d14c2a5 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorScenarioKit.scala @@ -0,0 +1,137 @@ +package harness + +// The merge-on-read preparation kit. A merge-on-read table is format version 2 with the delete, +// update and merge modes set to merge-on-read, so a mutation records position-delete files while +// preserving the untouched data files. This layer sits above RTAS, so it also owns the replace-lineage +// merge-on-read preparations. The members are lazy so they initialize on first read, after every +// trait mixed into `object Scenarios` has been constructed. +trait MorScenarioKit extends RtasScenarioKit { + + // Merge-on-read layouts use the standard shapes and record DELETE, UPDATE and MERGE changes in + // position-delete files. Only mutation operations run against these format-version 2 layouts. + private def morLayout(partitioning: Partitioning, format: String): Layout = + Layout( + s"mor-${partitioning.label}/$format", + s"a merge-on-read format-version 2 $format table ${partitioning.description}", + table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES ('write.format.default'='$format', 'format-version'='2', " + + s"'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read', 'write.merge.mode'='merge-on-read')") + + lazy val morLayouts: List[Layout] = + for { + format <- fileFormats + partitioning <- partitionings + } yield morLayout(partitioning, format) + + lazy val unpartitionedMorLayouts: List[Layout] = + fileFormats.map(format => morLayout(unpartitioned, format)) + + // Layouts that pin how a DELETE is written physically. Both set `write.distribution-mode=none` + // and stay unpartitioned so a single seed INSERT lands every row in ONE data file. Deleting a + // strict subset is then a partial-file match, which Iceberg satisfies by writing a position + // delete under merge-on-read and by rewriting the data file under copy-on-write. The general + // `morLayouts` seed spreads rows over several files, where a delete aligned with a file boundary + // is satisfied by dropping that whole file, so these layouts are what make the physical outcome + // deterministic across formats. + lazy val morVerifyLayouts: List[Layout] = + fileFormats.map(format => Layout( + s"mor-verify/$format", + s"a merge-on-read format-version 2 $format table with no partitioning that writes one data file per insert", + table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'format-version'='2', 'write.distribution-mode'='none', " + + s"'write.delete.mode'='merge-on-read')")) + + lazy val cowVerifyLayouts: List[Layout] = + fileFormats.map(format => Layout( + s"cow-verify/$format", + s"a copy-on-write format-version 2 $format table with no partitioning that writes one data file per insert", + table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'format-version'='2', 'write.distribution-mode'='none', " + + s"'write.delete.mode'='copy-on-write')")) + + lazy val preparedMorCoreTables: List[TablePreparation[CoreTable.type]] = + morLayouts.map(layout => + TablePreparation( + layout.label, + createAndSeed(layout, 3), + description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, " + + "so a mutation writes position-delete files.")) + + // Seed every row into ONE data file. A plain seed INSERT spreads the rows over a couple of files, + // where a delete aligned with a file boundary is satisfied by dropping that whole file. The + // `COALESCE(1)` hint forces a single write task and so a single data file, which makes a + // strict-subset delete a partial-file match: merge-on-read writes a position delete for it, and + // copy-on-write rewrites the data file. + def createAndSeedSingleFile(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(layout.create)() + .sql(s"seed($numberOfRows, one-file)")(table => + s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM (${RowGenerator.valuesClause(Core, numberOfRows)}) AS seed")( + view => assert(view.after.size == numberOfRows, + s"single-file seed expected $numberOfRows rows, got ${view.after.size}")) + + // Seed one data file on a merge-on-read layout, then delete a strict subset, which leaves a live + // position-delete file. A table in this state exercises the scan path where the reader applies a + // position delete, so the read cases run against rows that survive that filtering. + def createAndSeedMorDeleted(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = + createAndSeedSingleFile(layout, numberOfRows) + .step("prep.morDelete") { (spark, table) => + spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") // a strict subset, so Iceberg writes a position delete + } { view => + assert(view.after.size == numberOfRows - 1, s"MoR prep delete failed: ${view.after.size}") + val deleteFiles = view.spark.sql(s"SELECT count(*) FROM ${view.table}.all_delete_files").collect()(0).getLong(0) + assert(deleteFiles == 1, s"MoR prep must leave a live position-delete file, got $deleteFiles") + } + + lazy val preparedMorReadCoreTables: List[TablePreparation[CoreTable.type]] = + morVerifyLayouts.map { layout => + TablePreparation( + layout.label, + createAndSeedMorDeleted(layout, 3), + "prep.morRead:", + description = s"Three seed rows written as one data file in ${layout.description}, then the " + + "row with key 1 deleted merge-on-read, so keys 2 and 3 remain behind a live position-delete " + + "file that the reader applies at scan time.") + } + + // RTAS preparation on a MERGE-ON-READ table: the replace re-specifies the MoR delete, update, and + // merge modes, so the mutation cases exercise the MoR write path on a replace-lineage table. + protected def morPropsFmt(format: String) = s"'write.format.default'='$format', 'format-version'='2', " + + "'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read', 'write.merge.mode'='merge-on-read'" + + def createAndSeedRtasMor(partitioning: Partitioning, numberOfRows: Int, format: String): TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES (${morPropsFmt(format)}, 'replace.enabled'='true')")() + .insert(numberOfRows)() + .sql("prep.rtasMor")(t => s"CREATE OR REPLACE TABLE $t USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES (${morPropsFmt(format)}) AS SELECT * FROM $t")() + // The OpenHouse user guide requires REFRESH TABLE after a replace, so the Spark session + // reads the committed metadata pointer before the preparation returns. + .sql("prep.rtasMor.refresh")(t => s"REFRESH TABLE $t")() + + lazy val preparedRtasMorCoreTables: List[TablePreparation[CoreTable.type]] = + fileFormats.map { format => + TablePreparation( + s"mor-${unpartitioned.label}/$format", + createAndSeedRtasMor(unpartitioned, 3, format), + "prep.rtasMor:", + description = s"Three seed rows with keys 1, 2 and 3 in a merge-on-read format-version 2 " + + s"$format table ${unpartitioned.description}, then replaced by CREATE OR REPLACE TABLE AS " + + "SELECT re-specifying the merge-on-read modes, so mutations run on replace lineage.") + } + + lazy val preparedNullStringMorCoreTables: List[TablePreparation[CoreTable.type]] = + preparedMorCoreTables.map(withNullStringRow) + + lazy val preparedNullStringRtasMorCoreTables: List[TablePreparation[CoreTable.type]] = + preparedRtasMorCoreTables.map(withNullStringRow) + + lazy val morReadLayoutFormatPreparations: List[TablePreparation[CoreTable.type]] = + preparedMorReadCoreTables + + def morReadLayoutFormatCases: List[Plan.Case] = + layoutFormatCasesFor(morReadLayoutFormatPreparations) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorSurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorSurfaceScenarios.scala new file mode 100644 index 000000000..5fb0cdc3d --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorSurfaceScenarios.scala @@ -0,0 +1,76 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// The merge-on-read surface families. Both cases start from a table whose delete mode is +// merge-on-read and whose seed is one data file, so a strict-subset DELETE leaves a live +// position-delete file. One compacts that file through rewrite_position_delete_files, the other +// reads it back through the position_deletes metadata table. The cases run on parquet and orc. +trait MorSurfaceScenarios extends MorScenarioKit { + import Rows._ + + private def surfaceMergeOnReadPreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', " + + "'write.delete.mode'='merge-on-read')")() + .sql("seed")(table => + s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM " + + s"(${RowGenerator.valuesClause(Core, 3)}) AS seed")(), + description = s"Three seed rows written as one data file in a merge-on-read $format " + + "table.") + + // The rewrite procedure that compacts position-delete files. + def morSurfaceRewriteProcedureCases(format: String): List[Plan.Case] = + List( + surfaceMergeOnReadPreparation(format).test( + "surface.proc.rewritePositionDeletes", + "After a MoR DELETE creates one position-delete file, rewrite_position_delete_files " + + "compacts it while the 2 surviving rows remain readable.") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.all_delete_files") == "1", + "MoR delete should create one position-delete file") + + table.spark.sql( + "CALL openhouse.system.rewrite_position_delete_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "2", + "rewrite_position_delete_files should preserve live rows") + }) + + // The position_deletes metadata table. + def morSurfaceMetadataCases(format: String): List[Plan.Case] = + List( + surfaceMergeOnReadPreparation(format).test( + "surface.meta.positionDeletes", + "After a MoR DELETE, the position_deletes metadata table reports exactly the one " + + "position-delete entry it created.") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}.position_deletes") == "1", + "position_deletes should expose the MoR position delete") + }) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala index 0799084d9..a81a42bd3 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala @@ -18,14 +18,20 @@ trait NegativeDdlScenarios extends ScenarioKit { val negativeCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => List( - preparation.test("negative.nonExistentColumn") { table => + preparation.test( + "negative.nonExistentColumn", + "DELETE with a WHERE clause on a nonexistent column is rejected with an " + + "AnalysisException naming that column.") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( s"DELETE FROM ${table.name} WHERE no_such_column = 1")) assert(exception.getMessage.contains("no_such_column")) }, - preparation.test("negative.nonDeterministicDelete") { table => + preparation.test( + "negative.nonDeterministicDelete", + "DELETE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an " + + "AnalysisException about determinism.") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( s"DELETE FROM ${table.name} WHERE rand() < 0.5")) @@ -33,7 +39,10 @@ trait NegativeDdlScenarios extends ScenarioKit { assert( exception.getMessage.toLowerCase.contains("deterministic")) }, - preparation.test("negative.nonDeterministicUpdate") { table => + preparation.test( + "negative.nonDeterministicUpdate", + "UPDATE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an " + + "AnalysisException about determinism.") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( s"UPDATE ${table.name} SET $S = 'x' WHERE rand() < 0.5")) @@ -41,7 +50,10 @@ trait NegativeDdlScenarios extends ScenarioKit { assert( exception.getMessage.toLowerCase.contains("deterministic")) }, - preparation.test("negative.insertArity") { table => + preparation.test( + "negative.insertArity", + "INSERT INTO with too few values for the table's columns is rejected with an " + + "AnalysisException about the missing data columns.") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( s"INSERT INTO ${table.name} VALUES (CAST(1 AS BIGINT), 1)")) @@ -50,7 +62,10 @@ trait NegativeDdlScenarios extends ScenarioKit { exception.getMessage.toLowerCase.contains( "not enough data columns")) }, - preparation.test("negative.mergeConflictingUpdates") { table => + preparation.test( + "negative.mergeConflictingUpdates", + "A MERGE whose UPDATE SET assigns the same target column twice is rejected with an " + + "AnalysisException about multiple assignments.") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( s"""MERGE INTO ${table.name} target USING ( @@ -62,7 +77,10 @@ trait NegativeDdlScenarios extends ScenarioKit { assert(exception.getMessage.contains("Multiple assignments")) }, - preparation.test("negative.mergeCardinalityViolation") { table => + preparation.test( + "negative.mergeCardinalityViolation", + "A MERGE whose source has two rows matching the same target row fails with a " + + "cardinality-violation error naming the multi-row match.") { table => val exception = Check.intercept[Exception]( table.spark.sql( s"""MERGE INTO ${table.name} target USING ( @@ -82,7 +100,10 @@ trait NegativeDdlScenarios extends ScenarioKit { "expected a MERGE cardinality-violation message, got: " + exception.getMessage) }, - preparation.test("negative.partitionByNonExistent") { table => + preparation.test( + "negative.partitionByNonExistent", + "CREATE TABLE PARTITIONED BY a nonexistent column is rejected with an " + + "AnalysisException naming that column, and no scratch table is left behind.") { table => val scratchTable = table.name + "_x" val exception = Check.intercept[AnalysisException]( table.spark.sql( @@ -97,7 +118,10 @@ trait NegativeDdlScenarios extends ScenarioKit { val ddlNegativeCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => List( - preparation.test("ddl.neg.dropColumn") { table => + preparation.test( + "ddl.neg.dropColumn", + "ALTER TABLE DROP COLUMN is rejected with a BadRequestException naming the column that " + + "would be dropped.") { table => val exception = Check.intercept[BadRequestException]( table.spark.sql( s"ALTER TABLE ${table.name} DROP COLUMN ${Core.int0.columnName}")) @@ -109,7 +133,10 @@ trait NegativeDdlScenarios extends ScenarioKit { exception.getMessage.contains(Core.int0.columnName), s"message should name the dropped column: ${exception.getMessage.take(160)}") }, - preparation.test("ddl.neg.narrowType") { table => + preparation.test( + "ddl.neg.narrowType", + "ALTER TABLE ALTER COLUMN to a narrower type (bigint to int) is rejected with an " + + "AnalysisException about the unsupported column change.") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.long0.columnName} TYPE int")) @@ -118,7 +145,10 @@ trait NegativeDdlScenarios extends ScenarioKit { exception.getMessage.contains("NOT_SUPPORTED_CHANGE_COLUMN"), s"unexpected message: ${exception.getMessage.take(160)}") }, - preparation.test("ddl.neg.setNotNull") { table => + preparation.test( + "ddl.neg.setNotNull", + "ALTER TABLE ALTER COLUMN SET NOT NULL on a nullable column is rejected with an " + + "AnalysisException about the nullable-to-non-nullable change.") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.string0.columnName} SET NOT NULL")) @@ -137,15 +167,20 @@ trait NegativeDdlScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + s"'write.format.default'='$format', 'format-version'='1')")() - .insert(3)()) + .insert(3)(), + description = "Three seed rows in a table created with format-version=1 requested.") val previousVersionsPreparation = TablePreparation( format, TableTest(Core).sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'write.metadata.previous-versions-max'='7')")()) + s"'write.format.default'='$format', 'write.metadata.previous-versions-max'='7')")(), + description = "An unseeded table created with write.metadata.previous-versions-max=7.") List( - preparation.test("ddl.props.userRoundTrip") { table => + preparation.test( + "ddl.props.userRoundTrip", + "SET TBLPROPERTIES adds a user property that reads back, and UNSET TBLPROPERTIES " + + "removes it.") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('my_key'='my_val')") assert( @@ -157,7 +192,10 @@ trait NegativeDdlScenarios extends ScenarioKit { !tableProps(table.spark, table.name).contains("my_key"), "user prop not removed") }, - preparation.test("ddl.props.reservedOpenhouse") { table => + preparation.test( + "ddl.props.reservedOpenhouse", + "SET TBLPROPERTIES on the reserved openhouse.tableUUID property is rejected with a " + + "BadRequestException about the restriction.") { table => val exception = Check.intercept[BadRequestException]( table.spark.sql( s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + @@ -167,7 +205,10 @@ trait NegativeDdlScenarios extends ScenarioKit { exception.getMessage.toLowerCase.contains("restriction"), s"msg: ${exception.getMessage.take(200)}") }, - formatVersionPreparation.test("ddl.props.formatVersionForced") { table => + formatVersionPreparation.test( + "ddl.props.formatVersionForced", + "Even though format-version=1 was requested at creation, the table is forced to " + + "format-version=2 and remains writable.") { table => val formatVersion = tableProps(table.spark, table.name).get("format-version") assert( @@ -177,7 +218,10 @@ trait NegativeDdlScenarios extends ScenarioKit { table.rows.size == 3, "table not writable at the forced format-version") }, - previousVersionsPreparation.test("ddl.props.previousVersionsHonored") { table => + previousVersionsPreparation.test( + "ddl.props.previousVersionsHonored", + "The write.metadata.previous-versions-max property requested at creation is honored " + + "and reads back as 7.") { table => val previousVersions = tableProps(table.spark, table.name).get("write.metadata.previous-versions-max") @@ -187,16 +231,14 @@ trait NegativeDdlScenarios extends ScenarioKit { }) } - // Per-case "current seed format" (default parquet). The assembly's `crossFmt` sets it around each case - // so a block multiplexes across formats WITHOUT every builder taking an explicit fmt param. Safe because - // each case runs sequentially on its own worker thread (session-per-worker, parallel runner). This is - // how format-INERT-by-hypothesis blocks (DDL/props/policy/branch/surface/negatives) get run on ORC too — - + // Each preparation carries its format directly into the cases assembled below. val ddlMiscellaneousCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => val format = preparation.label List( - preparation.test("ddl.sortOrder.orderedBy") { table => + preparation.test( + "ddl.sortOrder.orderedBy", + "ALTER TABLE WRITE ORDERED BY a single column sets write.distribution-mode to range.") { table => table.spark.sql( s"ALTER TABLE ${table.name} WRITE ORDERED BY ${Core.long0.columnName}") @@ -207,7 +249,10 @@ trait NegativeDdlScenarios extends ScenarioKit { distributionMode.contains("range"), s"distribution-mode not range: $distributionMode") }, - preparation.test("ddl.sortOrder.orderedByMulti") { table => + preparation.test( + "ddl.sortOrder.orderedByMulti", + "ALTER TABLE WRITE ORDERED BY multiple columns sets range distribution and the table " + + "remains writable, growing from 3 to 5 rows after a follow-up insert.") { table => table.spark.sql( s"ALTER TABLE ${table.name} WRITE ORDERED BY " + s"${Core.string0.columnName} DESC NULLS FIRST, ${Core.long0.columnName}") @@ -221,7 +266,10 @@ trait NegativeDdlScenarios extends ScenarioKit { assert(table.rows.size == 5, "multi-col ordered write path failed") }, - preparation.test("ddl.renameTable") { table => + preparation.test( + "ddl.renameTable", + "ALTER TABLE RENAME TO moves the table to the new name with its 3 rows intact and the " + + "old name stops resolving; the test restores the original name afterward.") { table => val renamedTable = s"${table.name}_ren" table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $renamedTable") @@ -232,7 +280,10 @@ trait NegativeDdlScenarios extends ScenarioKit { table.spark.sql(s"SELECT 1 FROM ${table.name} LIMIT 1")) table.spark.sql(s"ALTER TABLE $renamedTable RENAME TO ${table.name}") }, - preparation.test("ddl.renameTable.conflict") { table => + preparation.test( + "ddl.renameTable.conflict", + "ALTER TABLE RENAME TO a name that already exists is rejected with an error naming the " + + "conflict.") { table => val conflictingTable = s"${table.name}_other" table.spark.sql(s"DROP TABLE IF EXISTS $conflictingTable") @@ -247,7 +298,10 @@ trait NegativeDdlScenarios extends ScenarioKit { s"msg: ${exception.getMessage.take(160)}") table.spark.sql(s"DROP TABLE IF EXISTS $conflictingTable") }, - preparation.test("ddl.ns.createRejected") { table => + preparation.test( + "ddl.ns.createRejected", + "CREATE NAMESPACE is rejected with an UnsupportedOperationException, since this " + + "catalog does not support creating namespaces.") { table => val exception = Check.intercept[UnsupportedOperationException]( table.spark.sql("CREATE NAMESPACE openhouse.a_new_db")) @@ -255,7 +309,10 @@ trait NegativeDdlScenarios extends ScenarioKit { exception.getMessage.contains("not supported"), s"msg: ${exception.getMessage.take(160)}") }, - preparation.test("ddl.ns.dropRejected") { table => + preparation.test( + "ddl.ns.dropRejected", + "DROP NAMESPACE is rejected with an UnsupportedOperationException, since this catalog " + + "does not support dropping namespaces.") { table => val exception = Check.intercept[UnsupportedOperationException]( table.spark.sql("DROP NAMESPACE openhouse.dbMatrix")) @@ -274,10 +331,14 @@ trait NegativeDdlScenarios extends ScenarioKit { s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + "PARTITIONED BY (datepartition) " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) + .insert(3)(), + description = "Three seed rows in a table partitioned by datepartition.") List( - preparation.test("ddl.policy.sharing") { table => + preparation.test( + "ddl.policy.sharing", + "SET POLICY (SHARING=TRUE) records the sharing policy and the table remains queryable.") { + table => table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") val policies = tableProps(table.spark, table.name).getOrElse("policies", "") @@ -290,7 +351,10 @@ trait NegativeDdlScenarios extends ScenarioKit { table.rows.size == 3, "table not queryable after SET POLICY (SHARING)") }, - preparation.test("ddl.policy.history") { table => + preparation.test( + "ddl.policy.history", + "SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20) records the history policy and the table " + + "remains queryable.") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20)") @@ -303,7 +367,10 @@ trait NegativeDdlScenarios extends ScenarioKit { table.rows.size == 3, "table not queryable after SET POLICY (HISTORY)") }, - preparation.test("ddl.policy.replication") { table => + preparation.test( + "ddl.policy.replication", + "SET POLICY (REPLICATION) followed by UNSET POLICY (REPLICATION) leaves the table " + + "queryable with its 3 rows intact.") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") table.spark.sql( @@ -311,7 +378,10 @@ trait NegativeDdlScenarios extends ScenarioKit { assert(table.rows.size == 3) }, - retentionPreparation.test("ddl.policy.retention") { table => + retentionPreparation.test( + "ddl.policy.retention", + "SET POLICY (RETENTION = 30d ON COLUMN datepartition ...) records the retention policy " + + "and the table remains queryable.") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET POLICY (" + "RETENTION = 30d ON COLUMN datepartition WHERE pattern = 'yyyy-MM-dd-HH')") @@ -325,7 +395,10 @@ trait NegativeDdlScenarios extends ScenarioKit { table.rows.size == 3, "table not queryable after SET POLICY (RETENTION)") }, - preparation.test("ddl.policy.neg.historyMaxAge") { table => + preparation.test( + "ddl.policy.neg.historyMaxAge", + "SET POLICY (HISTORY MAX_AGE=5D) exceeds the allowed range and is rejected with a " + + "BadRequestException stating the 1-to-3-day limit.") { table => val exception = Check.intercept[BadRequestException]( table.spark.sql( s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=5D)")) @@ -334,7 +407,10 @@ trait NegativeDdlScenarios extends ScenarioKit { exception.getMessage.contains("max age must be between 1 to 3 days"), s"msg: ${exception.getMessage.take(160)}") }, - preparation.test("ddl.policy.neg.historyVersions") { table => + preparation.test( + "ddl.policy.neg.historyVersions", + "SET POLICY (HISTORY VERSIONS=200) exceeds the allowed range and is rejected with a " + + "BadRequestException stating the 2-to-100-version limit.") { table => val exception = Check.intercept[BadRequestException]( table.spark.sql( s"ALTER TABLE ${table.name} SET POLICY (HISTORY VERSIONS=200)")) @@ -345,59 +421,6 @@ trait NegativeDdlScenarios extends ScenarioKit { }) } - val ddlCtasRtasCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => - List( - preparation.test("ddl.ctas") { table => - val targetTable = s"${table.name}_ctas" - - table.spark.sql(s"DROP TABLE IF EXISTS $targetTable") - table.spark.sql( - s"CREATE TABLE $targetTable USING $dataSource AS SELECT * FROM ${table.name}") - - assert( - table.spark.sql(s"SELECT count(*) FROM $targetTable").collect()(0).getLong(0) == 3, - "CTAS lost rows") - - table.spark.sql(s"DROP TABLE IF EXISTS $targetTable") - }, - preparation.test("ddl.rtas.enabled") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('replace.enabled'='true')") - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} WHERE ${Core.long0.columnName} <= 2") - - assert( - table.spark.sql(s"SELECT count(*) FROM ${table.name}").collect()(0).getLong(0) == 2, - "RTAS did not replace") - }, - preparation.test("ddl.rtas.disabled") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name}")) - - assert( - exception.getMessage.contains("REPLACE TABLE AS SELECT is not enabled"), - s"msg: ${exception.getMessage.take(160)}") - }, - preparation.test("ddl.rtas.replicationConflict") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('replace.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") - - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name}")) - - assert( - exception.getMessage.contains("while replication is enabled"), - s"msg: ${exception.getMessage.take(160)}") - }) - } - val ddlTagAclFeatureCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => val format = preparation.label val distributionModePreparation = TablePreparation( @@ -406,10 +429,14 @@ trait NegativeDdlScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + s"'write.format.default'='$format', 'write.distribution-mode'='none')")() - .insert(3)()) + .insert(3)(), + description = "Three seed rows in a table created with write.distribution-mode=none.") List( - preparation.test("ddl.colTag") { table => + preparation.test( + "ddl.colTag", + "ALTER TABLE MODIFY COLUMN SET TAG = (PII) tags a column without masking or changing " + + "the values that queries return.") { table => table.spark.sql( s"ALTER TABLE ${table.name} MODIFY COLUMN " + s"${Core.string0.columnName} SET TAG = (PII)") @@ -426,7 +453,10 @@ trait NegativeDdlScenarios extends ScenarioKit { values == Seq("row-1", "row-2", "row-3"), s"SET TAG changed query results (should not mask): $values") }, - preparation.test("ddl.acl.grantUnshared") { table => + preparation.test( + "ddl.acl.grantUnshared", + "GRANT SELECT on a table that is not marked shared is rejected with an " + + "IllegalArgumentException stating the table is not shared.") { table => val exception = Check.intercept[IllegalArgumentException]( table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC")) @@ -434,13 +464,45 @@ trait NegativeDdlScenarios extends ScenarioKit { exception.getMessage.contains("is not a shared table"), s"msg: ${exception.getMessage.take(160)}") }, - preparation.test("ddl.acl.grantShared") { table => - table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") - table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC") - - assert(table.rows.size == 3, "shared/granted table not queryable") - }, - distributionModePreparation.test("ddl.featureFlag.distributionMode") { table => + preparation + .test( + "ddl.acl.grantShared", + "On a shared table, GRANT SELECT TO PUBLIC makes SHOW GRANTS list SELECT for PUBLIC " + + "and the table stays queryable; REVOKE SELECT then removes that grant from SHOW " + + "GRANTS.") { table => + table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") + table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC") + + val grantsAfterGrant = table.spark + .sql(s"SHOW GRANTS ON TABLE ${table.name}") + .collect() + .map(row => (row.getString(0), row.getString(1))) + .toSet + assert( + grantsAfterGrant.contains(("SELECT", "PUBLIC")), + s"SHOW GRANTS did not include SELECT for PUBLIC: $grantsAfterGrant") + assert(table.rows.size == 3, "shared/granted table not queryable") + + table.spark.sql(s"REVOKE SELECT ON TABLE ${table.name} FROM PUBLIC") + val grantsAfterRevoke = table.spark + .sql(s"SHOW GRANTS ON TABLE ${table.name}") + .collect() + .map(row => (row.getString(0), row.getString(1))) + .toSet + assert( + !grantsAfterRevoke.contains(("SELECT", "PUBLIC")), + s"SHOW GRANTS retained SELECT for PUBLIC: $grantsAfterRevoke") + } + .copy(embeddedSkipReason = Some( + "The embedded test server has no OPA endpoint configured, so grantRole and " + + "listAclPolicies are no-ops that always report an empty ACL list. GRANT and REVOKE " + + "succeed without error, while SHOW GRANTS always returns an empty ACL list. The " + + "li-openhouse acceptance environment runs the assertions against its configured " + + "authorization service.")), + distributionModePreparation.test( + "ddl.featureFlag.distributionMode", + "The write.distribution-mode=none property requested at creation is honored and the " + + "table remains writable under it.") { table => val distributionMode = tableProps(table.spark, table.name).get("write.distribution-mode") @@ -451,7 +513,10 @@ trait NegativeDdlScenarios extends ScenarioKit { table.rows.size == 3, "table not writable under distribution-mode=none") }, - preparation.test("ddl.repl.tableTypeImmutable") { table => + preparation.test( + "ddl.repl.tableTypeImmutable", + "ALTER TABLE SET TBLPROPERTIES ('openhouse.tableType'='REPLICA_TABLE') is rejected with " + + "a BadRequestException, since table type cannot be changed after creation.") { table => val exception = Check.intercept[BadRequestException]( table.spark.sql( s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + @@ -463,42 +528,4 @@ trait NegativeDdlScenarios extends ScenarioKit { }) } - // ── DDL Phase 24b: encryption — asserts the INTENDED behavior, tagged SKIP in OSS ───────────── - // The KMS plugin is external/private (a repo-wide search finds no EncryptionManager / - // KeyManagementClient / crypto factory / interface / mock). This test asserts what SHOULD happen — - // with encryption configured, the data file must NOT be readable as plaintext parquet. In OSS the - // hook is un-wired so files are plaintext and this would fail; it is tagged in Plan.knownBugs and - // reports SKIP until the private plugin is present (then unskip to validate encryption-ON). - val ddlEncryptionCases: List[Plan.Case] = { - val preparation = TablePreparation( - "parquet", - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - "'write.format.default'='parquet', 'encryption.key-id'='k1', " + - "'write.metadata.encryption.gcm-key-id'='k1')")() - .insert(3)()) - - List(preparation.test("ddl.encryption.active") { table => - val filePath = table.spark - .sql(s"SELECT file_path FROM ${table.name}.files LIMIT 1") - .collect()(0) - .getString(0) - .stripPrefix("file:") - val fileHeader = new String( - java.nio.file.Files.readAllBytes(java.nio.file.Paths.get(filePath)).take(4)) - - assert( - fileHeader != "PAR1", - s"encryption not in force: data file is plaintext parquet (magic=$fileHeader); " + - "requires the private KMS plugin") - }) - } - - // ═══ Feature-INTERACTION axis (INTERACTION-AUDIT.md) — behaviors, single layout ══════════════ - // Characterization stance: rejections are PINS of current behavior (tripwires), not contracts; - // a pin that starts failing means the product changed — update the pin and activate the dormant - // coverage it gates (see the pin inventory in INTERACTION-AUDIT.md §2b). - - } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala index 8de27468f..4470a7f70 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala @@ -13,7 +13,7 @@ import scala.util.control.NonFatal trait NestedTypesScenarios extends ScenarioKit { import Rows._ - // ── nested / complex types (NestedTable) ─────────────────────────────────────────────── + // Nested and complex types (NestedTable). val nestedLayouts: List[Layout] = List("parquet", "orc", "avro").map(format => Layout(s"nested-unpartitioned/$format", table => s"CREATE TABLE $table (${NestedTable.columnDefinitions}) USING $dataSource TBLPROPERTIES ('write.format.default'='$format')")) @@ -26,10 +26,15 @@ trait NestedTypesScenarios extends ScenarioKit { .map(layout => TablePreparation( layout.label, - createAndSeedNested(layout, 3))) + createAndSeedNested(layout, 3), + description = s"Three seed rows with nested struct, array, map and doubly-nested " + + s"struct fields in an unpartitioned ${layout.label.split('/').last} table.")) .flatMap { preparation => List( - preparation.test("nested.roundtrip") { table => + preparation.test( + "nested.roundtrip", + "Selecting the top-level id alongside struct, array, map and nested-struct fields " + + "reads back exactly the seeded values for all 3 rows.") { table => val actual = table.spark .sql( s"SELECT id, s.x, s.y, arr, m['k'], nested.inner.z " + @@ -56,7 +61,10 @@ trait NestedTypesScenarios extends ScenarioKit { assert(actual == expected) }, - preparation.test("nested.projectField") { table => + preparation.test( + "nested.projectField", + "Selecting only a nested struct field (s.x) returns just that field's values for " + + "all 3 rows, in id order.") { table => val actual = table.spark .sql(s"SELECT s.x FROM ${table.name} ORDER BY id") .collect() @@ -65,7 +73,10 @@ trait NestedTypesScenarios extends ScenarioKit { assert(actual == Seq(1, 2, 3)) }, - preparation.test("nested.filterNestedField") { table => + preparation.test( + "nested.filterNestedField", + "Filtering WHERE s.x = 2 on a nested struct field returns only the matching row's " + + "id.") { table => val actual = table.spark .sql(s"SELECT id FROM ${table.name} WHERE s.x = 2 ORDER BY id") .collect() @@ -74,7 +85,10 @@ trait NestedTypesScenarios extends ScenarioKit { assert(actual == Seq(2L)) }, - preparation.test("nested.updateStructField") { table => + preparation.test( + "nested.updateStructField", + "UPDATE SET s.x = 99 WHERE id = 2 changes only that row's nested field, leaving " + + "other rows' nested fields untouched.") { table => table.spark.sql( s"UPDATE ${table.name} SET s.x = 99 WHERE id = 2") @@ -89,7 +103,10 @@ trait NestedTypesScenarios extends ScenarioKit { .collect()(0) .getInt(0) == 1) }, - preparation.test("nested.mergeInsert") { table => + preparation.test( + "nested.mergeInsert", + "MERGE WHEN NOT MATCHED THEN INSERT with a fully nested source row adds a 4th row " + + "whose nested struct field reads back as inserted.") { table => table.spark.sql( s"""MERGE INTO ${table.name} target USING ( SELECT * FROM VALUES @@ -116,19 +133,29 @@ trait NestedTypesScenarios extends ScenarioKit { .collect()(0) .getInt(0) == 4) }, - preparation.test("nested.deleteByNestedField") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE s.x = 2") + preparation + .test( + "nested.deleteByNestedField", + "DELETE WHERE s.x = 2 filtering on a nested struct field removes only the matching " + + "row, leaving ids 1 and 3.") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE s.x = 2") - val ids = table.spark - .sql(s"SELECT id FROM ${table.name} ORDER BY id") - .collect() - .toSeq - .map(_.getLong(0)) + val ids = table.spark + .sql(s"SELECT id FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(_.getLong(0)) - assert(ids == Seq(1L, 3L)) - }, - preparation.test("nested.nullValues") { table => + assert(ids == Seq(1L, 3L)) + } + .copy(knownBugReason = Some( + "DELETE on a nested struct field crashes in the Spark and Iceberg row-level " + + "rewrite.")), + preparation.test( + "nested.nullValues", + "Inserting a row with NULL struct, empty array and empty map reads back a null " + + "struct and an empty array for that row.") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES (" + "CAST(4 AS BIGINT), " + @@ -146,7 +173,7 @@ trait NestedTypesScenarios extends ScenarioKit { }) } - // ── type-edge coverage (TypesTable) ───────────────────────────────────────────────────── + // Type-edge coverage (TypesTable). val typesLayouts: List[Layout] = List("parquet", "orc", "avro").map(format => Layout(s"types-unpartitioned/$format", table => s"CREATE TABLE $table (${TypesTable.columnDefinitions}) USING $dataSource TBLPROPERTIES ('write.format.default'='$format')")) @@ -159,15 +186,25 @@ trait NestedTypesScenarios extends ScenarioKit { s"(CAST($id AS BIGINT), $n, $x, $dec, $str, CAST('b' AS binary), DATE '2024-01-01', " + s"TIMESTAMP '2024-01-01 00:00:00', TIMESTAMP_NTZ '2024-01-01 00:00:00')" + private def partitionRow(id: Long, str: String, timestamp: String): String = + s"(CAST($id AS BIGINT), ${id.toInt}, ${id}.5, " + + s"CAST(${id}.50 AS decimal(10,2)), '$str', CAST('bin-$id' AS binary), " + + s"DATE '${timestamp.take(10)}', TIMESTAMP '$timestamp', TIMESTAMP_NTZ '$timestamp')" + val typesCases: List[Plan.Case] = typesLayouts .map(layout => TablePreparation( layout.label, - createAndSeedTypes(layout, 3))) + createAndSeedTypes(layout, 3), + description = "Three seed rows covering int, double, decimal, string, binary, date, " + + s"timestamp and timestamp_ntz columns in an unpartitioned ${layout.label.split('/').last} table.")) .flatMap { preparation => List( - preparation.test("types.roundtrip") { table => + preparation.test( + "types.roundtrip", + "Selecting id, n, x, dec and str for the first seeded row reads back the exact " + + "long, int, double, decimal and string values that were seeded.") { table => val row = table.spark .sql( s"SELECT id, n, x, dec, str FROM ${table.name} WHERE id = 1") @@ -182,7 +219,10 @@ trait NestedTypesScenarios extends ScenarioKit { new java.math.BigDecimal("1.50")) == 0) assert(row.getString(4) == "row-1") }, - preparation.test("types.nulls") { table => + preparation.test( + "types.nulls", + "Inserting a row with every non-key column NULL reads back as null for the int, " + + "double, string, timestamp and timestamp_ntz columns.") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES (" + "CAST(10 AS BIGINT), NULL, NULL, NULL, NULL, " + @@ -195,7 +235,10 @@ trait NestedTypesScenarios extends ScenarioKit { assert((0 to 4).forall(row.isNullAt)) }, - preparation.test("types.specialFloats") { table => + preparation.test( + "types.specialFloats", + "Inserting rows with double('NaN') and double('Infinity') reads back as NaN and " + + "positive infinity respectively.") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES " + s"${typesRow(11, "0", "double('NaN')", "CAST(0 AS decimal(10,2))", "'x'")}, " + @@ -214,7 +257,10 @@ trait NestedTypesScenarios extends ScenarioKit { .getDouble(0) .isInfinite) }, - preparation.test("types.boundaries") { table => + preparation.test( + "types.boundaries", + "Inserting a row at Long.MaxValue, Int.MaxValue and a max-precision decimal reads " + + "those boundary values back unchanged.") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES " + typesRow( @@ -236,17 +282,20 @@ trait NestedTypesScenarios extends ScenarioKit { row.getDecimal(2).compareTo( new java.math.BigDecimal("99999999.99")) == 0) }, - preparation.test("types.unicodeAndEmpty") { table => + preparation.test( + "types.unicodeAndEmpty", + "Inserting rows with a unicode string and an empty string reads each back " + + "unchanged.") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES " + - s"${typesRow(13, "0", "0.0", "CAST(0 AS decimal(10,2))", "'日本語 🎉'")}, " + + s"${typesRow(13, "0", "0.0", "CAST(0 AS decimal(10,2))", "'\u65e5\u672c\u8a9e \uD83C\uDF89'")}, " + s"${typesRow(14, "0", "0.0", "CAST(0 AS decimal(10,2))", "''")}") assert( table.spark .sql(s"SELECT str FROM ${table.name} WHERE id = 13") .collect()(0) - .getString(0) == "日本語 🎉") + .getString(0) == "\u65e5\u672c\u8a9e \uD83C\uDF89") assert( table.spark .sql(s"SELECT str FROM ${table.name} WHERE id = 14") @@ -255,21 +304,19 @@ trait NestedTypesScenarios extends ScenarioKit { }) } - // ── partition transforms + evolution ──────────────────────────────────────────────────── - // Each transform test is self-contained: create partitioned by the transform, seed, and verify - // the rows roundtrip and a partition spec is registered. + // Partition transforms and evolution. val partitionTransformCases: List[Plan.Case] = List("parquet", "orc").flatMap { format => val supported = List( - "partition.identity" -> "id", - "partition.bucket" -> "bucket(4, id)", - "partition.truncate" -> "truncate(2, str)", - "partition.years" -> "years(ts)", - "partition.months" -> "months(ts)", - "partition.days" -> "days(ts)", - "partition.hours" -> "hours(ts)") + ("partition.identity", "id", "id", 3), + ("partition.bucket", "bucket(4, id)", "id_bucket", 2), + ("partition.truncate", "truncate(2, str)", "str_trunc", 3), + ("partition.years", "years(ts)", "ts_year", 2), + ("partition.months", "months(ts)", "ts_month", 3), + ("partition.days", "days(ts)", "ts_day", 3), + ("partition.hours", "hours(ts)", "ts_hour", 3)) .map { - case (caseName, transform) => + case (caseName, transform, partitionField, expectedPartitionCount) => TablePreparation( format, TableTest(TypesTable) @@ -277,14 +324,33 @@ trait NestedTypesScenarios extends ScenarioKit { s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + s"USING $dataSource PARTITIONED BY ($transform) " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - .test(caseName) { table => - assert(table.rows.size == 3) + .sql("insertPartitionRows")(table => + s"INSERT INTO $table VALUES " + + partitionRow(1, "aa-1", "2023-12-31 23:00:00") + ", " + + partitionRow(2, "bb-2", "2024-01-01 00:00:00") + ", " + + partitionRow(3, "cc-3", "2024-02-01 01:00:00"))(view => + assert( + view.after.size == view.before.size + 3, + s"expected three partition test rows, got ${view.after.size}")), + description = s"Three rows in a $format table partitioned by $transform.") + .test( + caseName, + s"With PARTITIONED BY ($transform), the partitions metadata table reports a " + + s"single partition field named $partitionField and $expectedPartitionCount " + + "distinct partitions for the seeded rows.") { table => + val partitionTable = + table.spark.table(s"${table.name}.partitions") + val partitionFields = partitionTable.schema("partition").dataType + .asInstanceOf[org.apache.spark.sql.types.StructType] + .fieldNames + .toSeq + + assert( + partitionFields == Seq(partitionField), + s"expected partition field $partitionField, got ${partitionFields.mkString(", ")}") assert( - table.spark - .sql(s"SELECT * FROM ${table.name}.partitions") - .collect() - .nonEmpty) + partitionTable.count() == expectedPartitionCount, + s"expected $expectedPartitionCount partitions for $transform") } } val rejected = List( @@ -301,8 +367,13 @@ trait NestedTypesScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + s"USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")()) - .test(caseName) { table => + s"TBLPROPERTIES ('write.format.default'='$format')")(), + description = s"An unpartitioned, unseeded $format table.") + .test( + caseName, + s"CREATE TABLE PARTITIONED BY ($transform) is rejected with a RuntimeException " + + s"whose message contains '$expectedMessage', and no scratch table is left " + + "behind.") { table => val scratchTable = table.name + "_x" val exception = Check.intercept[RuntimeException]( table.spark.sql( @@ -319,8 +390,8 @@ trait NestedTypesScenarios extends ScenarioKit { supported ++ rejected } - // OpenHouse contract: partition evolution is NOT supported — ALTER … ADD/DROP PARTITION FIELD is - // rejected with a 400 telling you to recreate the table. Captured as negative tests. + // Partition evolution is not supported: ALTER TABLE ADD or DROP PARTITION FIELD is rejected with + // a 400 response telling the caller to recreate the table. These cases capture that rejection. val partitionEvolutionCases: List[Plan.Case] = List("parquet", "orc").flatMap { format => List( @@ -330,8 +401,12 @@ trait NestedTypesScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - .test("partition.evolutionAdd.rejected") { table => + .insert(3)(), + description = s"Three seed rows in an unpartitioned $format table.") + .test( + "partition.evolutionAdd.rejected", + "ALTER TABLE ADD PARTITION FIELD is rejected with an exception stating partition " + + "evolution is not supported.") { table => val exception = Check.intercept[Exception]( table.spark.sql( s"ALTER TABLE ${table.name} ADD PARTITION FIELD datepartition")) @@ -346,8 +421,12 @@ trait NestedTypesScenarios extends ScenarioKit { s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + "PARTITIONED BY (datepartition) " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - .test("partition.evolutionDrop.rejected") { table => + .insert(3)(), + description = s"Three seed rows in a $format table partitioned by datepartition.") + .test( + "partition.evolutionDrop.rejected", + "ALTER TABLE DROP PARTITION FIELD is rejected with an exception stating partition " + + "evolution is not supported.") { table => val exception = Check.intercept[Exception]( table.spark.sql( s"ALTER TABLE ${table.name} DROP PARTITION FIELD datepartition")) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala index 4ec1e77dd..ae16fef80 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala @@ -1,14 +1,35 @@ package harness -/** Mixes the scenario-owned case lists and shared preparation kit into one catalog source. */ +/** Mixes the scenario-owned case lists and shared preparation kits into one catalog source. + * + * The traits are listed bottom-up in the feature stack: the standard framework first, then RTAS, + * then merge-on-read, then branch and write-audit-publish. A feature layer's traits extend that + * layer's kit, so removing a layer's files and the traits below removes the layer entirely. + */ object Scenarios - extends MorMaintScenarios - with DmlScenarios + extends DmlScenarios with NestedTypesScenarios with MaintControlScenarios with ForkScenarios - with BranchWapScenarios with NegativeDdlScenarios with InteractionScenarios with SurfaceScenarios with HazardReaderWriterScenarios + with ImplementationPinScenarios + with RtasDmlScenarios + with RtasDdlScenarios + with RtasInteractionScenarios + with RtasSurfaceScenarios + with RtasHazardScenarios + with MorDmlScenarios + with MorMaintScenarios + with MorReaderWriterScenarios + with MorInteractionScenarios + with MorSurfaceScenarios + with MorForkScenarios + with BranchDmlScenarios + with BranchWapScenarios + with BranchInteractionScenarios + with BranchSurfaceScenarios + with BranchHazardScenarios + with BranchMorScenarios diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala index 879038667..47a99eafc 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala @@ -2,31 +2,99 @@ package harness /** Defines the ordered catalog of scenario-owned test cases. */ object Plan { - final case class Case(id: String, run: Ctx => Unit) + final case class Case( + id: String, + run: Ctx => Unit, + description: String, + preparationDescription: String = "", + knownBugReason: Option[String] = None, + embeddedSkipReason: Option[String] = None + ) { + require(description.trim.nonEmpty, s"test case $id needs a description") + } /** The deterministic ordered case catalog. Reading it does not execute a case or start Spark. */ def caseIds: List[String] = cases.map(_.id) - // Known PRODUCT bugs: any case whose id contains the key is reported SKIP (bug: reason) instead - // of failing the suite, and is tracked in BUGS.md. This is how we "tag a failing test and filter - // it": a genuine bug is tagged here, deferred for follow-up, and never plowed past silently. - val knownBugs: List[(String, String)] = List( - // insert.explicitColumns is NO LONGER a bug tag — reclassified to a negative PIN (engine limitation, - // not OpenHouse; code-verified). See insertExplicitColumns above and BUGS.md. - "nested.deleteByNestedField" -> - "DELETE WHERE crashes with an internal optimizer NPE (SELECT/UPDATE on the same field work). Code-verified UPSTREAM: OpenHouse contributes no code to the row-level DELETE rewrite (owned by IcebergSparkSessionExtensions + Spark optimizer); the NPE is in the nested-field DELETE-rewrite plan. Needs a full stack capture before filing — see BUGS.md", - "prep.ordered:delete.byPartitionPredicate" -> - "DELETE by a partition predicate against a table created WITH a WRITE ORDERED BY clause throws an internal analyzer NPE, while the same DELETE on an unordered table (delete.byPartitionPredicate) succeeds. Code-verified UPSTREAM and the same family as nested.deleteByNestedField: OpenHouse contributes no code to the row-level DELETE rewrite (owned by IcebergSparkSessionExtensions plus the Spark optimizer), so the NPE lives in the ORDERED-BY DELETE-rewrite plan on Spark 3.5.2 / Iceberg 1.5.2. It reproduces identically on the embedded catalog and on the remote cluster, so it is gated in both environments. Needs a full stack capture before filing — see BUGS.md", - "ddl.renameColumn" -> - "RENAME COLUMN is a silent no-op. Code-verified GENUINE OpenHouse regression from #558 (commit 0ad4914): server-side normalizeSchemaCasingToTable rewrites every field's name to the table's spelling BY FIELD ID (BaseIcebergSchemaValidator:60-73), reverting the rename, and it runs BEFORE the sameSchema gate so validateWriteSchema (which would reject loudly) never fires. Fix: guard the normalizer with equalsIgnoreCase. Silent failure worse than the pre-#558 clean rejection — see BUGS.md", - "ddl.encryption" -> - "encryption KMS plugin is external/private (no impl/interface/mock in-repo); OSS leaves the encryption() hook un-wired and writes plaintext, so the intended-behavior assertion is deferred until the plugin is present — see DDL-TEST-PLAN.md / AUDIT-FINDINGS.md", - "control.undrop" -> - "undrop is SKIP under the DEFAULT stub path (HouseTableRepository is a @Primary in-memory stub; the public Tables DELETE hard-codes purge=true). Under HARNESS_REAL_HTS=1 the real embedded HTS is booted and undrop runs for real as the undrop:* battery + undropAdmin.* lifecycle (NOT SKIP) — see HTS-EMBED-PLAN.md / HTS-EMBED-IMPL.md / REST-FIDELITY-EVAL.md" - ) + def bugReason(testCase: Case): Option[String] = + testCase.knownBugReason.map(reason => s"bug: $reason") - def bugReason(id: String): Option[String] = - knownBugs.collectFirst { case (key, reason) if id.contains(key) => s"bug: $reason" } + // The interaction, surface, reader/writer and hazard families are crossed with these two file + // formats. Each family runs on one format before the next format starts, so the format loop is the + // outer one and every contribution below keeps the catalog position it holds today. + private val crossedFormats: List[String] = List("parquet", "orc") + + private def interactionContributions: List[Case] = + crossedFormats.flatMap { format => + List( + Scenarios.interactionDdlCases(format), + Scenarios.interactionRtasCases(format), + Scenarios.interactionBranchCases(format), + Scenarios.interactionBranchFlagCases(format), + Scenarios.interactionMorCases(format), + Scenarios.interactionMiscellaneousCases(format) + ).flatten + } + + private def surfaceContributions: List[Case] = + crossedFormats.flatMap { format => + List( + Scenarios.surfaceBranchMaintenanceCases(format), + Scenarios.surfaceMessageCases(format), + Scenarios.surfaceBranchCases(format), + Scenarios.surfaceReaderCases(format), + Scenarios.surfaceRewriteProcedureCases(format), + Scenarios.morSurfaceRewriteProcedureCases(format), + Scenarios.surfaceBranchPublishCases(format), + Scenarios.surfaceSnapshotProcedureCases(format), + Scenarios.surfaceMetadataCases(format), + Scenarios.morSurfaceMetadataCases(format), + Scenarios.surfaceConcurrencyCases(format), + Scenarios.surfaceRtasConcurrencyCases(format), + Scenarios.surfaceSchemaCases(format), + Scenarios.surfaceWriteCases(format), + Scenarios.surfaceBranchWriteCases(format), + Scenarios.surfacePinCases(format) + ).flatten + } + + private def hazardContributions: List[Case] = + crossedFormats.flatMap { format => + List( + Scenarios.hazardReaderCases(format), + Scenarios.hazardRtasCases(format), + Scenarios.hazardBranchCases(format), + Scenarios.hazardWriterCases(format) + ).flatten + } + + private def readerWriterContributions: List[Case] = + crossedFormats.flatMap { format => + List( + Scenarios.readerWriterChangelogAppendCases(format), + Scenarios.morReaderWriterChangelogAppendCases(format), + Scenarios.readerWriterChangelogOverwriteCases(format), + Scenarios.morReaderWriterChangelogOverwriteCases(format), + Scenarios.readerWriterChangelogDeleteCases(format), + Scenarios.morReaderWriterChangelogDeleteCases(format), + Scenarios.readerWriterChangelogUpdateCases(format), + Scenarios.morReaderWriterChangelogUpdateCases(format), + Scenarios.readerWriterChangelogMergeCases(format), + Scenarios.morReaderWriterChangelogMergeCases(format), + Scenarios.readerWriterIncrementalAndStreamCases(format) + ).flatten + } + + // Every DDL-consumer family runs against one evolved preparation before the next preparation + // starts, so the preparation loop is the outer one here. + private def ddlConsumerContributions: List[Case] = + Scenarios.ddlConsumerPreparations.flatMap { preparation => + List( + Scenarios.ddlConsumerDataCases(preparation), + Scenarios.branchDdlConsumerCases(preparation), + Scenarios.ddlConsumerCompactionCases(preparation) + ).flatten + } def cases: List[Case] = List( @@ -42,6 +110,10 @@ object Plan { Scenarios.restoreRollbackCases, Scenarios.negativeCases, Scenarios.createSchemaCases, + Scenarios.layoutFormatCases, + Scenarios.rtasLayoutFormatCases, + Scenarios.branchLayoutFormatCases, + Scenarios.morReadLayoutFormatCases, Scenarios.ddlSchemaCases, Scenarios.ddlNegativeCases, Scenarios.ddlPropertyCases, @@ -49,37 +121,39 @@ object Plan { Scenarios.ddlPolicyCases, Scenarios.ddlCtasRtasCases, Scenarios.ddlTagAclFeatureCases, - Scenarios.ddlEncryptionCases, Scenarios.maintenanceCases, Scenarios.controlPlaneCases, - Scenarios.branchingCases, - Scenarios.interactionCases, - Scenarios.interactionContextCases, - Scenarios.surfaceCases, - Scenarios.hazardCases, - Scenarios.hazardContextCases, - Scenarios.branchDmlCases, - Scenarios.branchDdlCases, - Scenarios.wapStagedCases, - Scenarios.branchPartitionedDmlCases, - Scenarios.branchMorDmlCases, - Scenarios.rtasDmlCases, - Scenarios.rtasPartitionedDmlCases, - Scenarios.rtasMorDmlCases, - Scenarios.morReadDmlCases, - Scenarios.morCoexistCases, - Scenarios.ddlConsumerCases, - Scenarios.readerWriterCases, - Scenarios.orderedDmlCases, - Scenarios.evolvedDmlCases, - Scenarios.undroppedDmlCases, - Scenarios.undropAdminCases, - Scenarios.maintenanceMorFoldCases, - Scenarios.maintenanceMorMetaCases, - Scenarios.undropInteractionCases, - Scenarios.morHazardCases, - Scenarios.morBranchMergeCases, - Scenarios.encryptionPinCases, - Scenarios.forkCases - ).flatten + Scenarios.branchingCases + ).flatten ++ + interactionContributions ++ + Scenarios.interactionContextCases ++ + surfaceContributions ++ + hazardContributions ++ + Scenarios.hazardContextCases ++ + List( + Scenarios.branchDmlCases, + Scenarios.branchDdlCases, + Scenarios.wapStagedCases, + Scenarios.branchPartitionedDmlCases, + Scenarios.branchMorDmlCases, + Scenarios.rtasDmlCases, + Scenarios.rtasPartitionedDmlCases, + Scenarios.rtasMorDmlCases, + Scenarios.morReadDmlCases, + Scenarios.morCoexistCases + ).flatten ++ + ddlConsumerContributions ++ + readerWriterContributions ++ + List( + Scenarios.orderedDmlCases, + Scenarios.evolvedDmlCases, + Scenarios.maintenanceMorFoldCases, + Scenarios.maintenanceMorMetaCases, + Scenarios.morHazardCases, + Scenarios.morBranchMergeCases, + Scenarios.encryptionPinCases, + Scenarios.forkColumnDefaultAndDistributionCases, + Scenarios.forkDeleteFileReplicationCases, + Scenarios.forkFileAndCompactionCases + ).flatten } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDdlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDdlScenarios.scala new file mode 100644 index 000000000..ea07d3d26 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDdlScenarios.scala @@ -0,0 +1,76 @@ +package harness + +import org.apache.iceberg.exceptions.BadRequestException + +// The CTAS and RTAS DDL family. CREATE TABLE AS SELECT copies a seeded table into a new one, and +// CREATE OR REPLACE TABLE AS SELECT replaces a table's content in place. The replace path is gated +// on the replace.enabled table property and is rejected outright while a replication policy is set, +// so both the enabled and the rejected outcomes are pinned here. +trait RtasDdlScenarios extends RtasScenarioKit { + + lazy val ddlCtasRtasCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => + List( + preparation.test( + "ddl.ctas", + "CREATE TABLE AS SELECT from the seeded table produces a new table with the same 3 " + + "rows.") { table => + val targetTable = s"${table.name}_ctas" + + table.spark.sql(s"DROP TABLE IF EXISTS $targetTable") + table.spark.sql( + s"CREATE TABLE $targetTable USING $dataSource AS SELECT * FROM ${table.name}") + + assert( + table.spark.sql(s"SELECT count(*) FROM $targetTable").collect()(0).getLong(0) == 3, + "CTAS lost rows") + + table.spark.sql(s"DROP TABLE IF EXISTS $targetTable") + }, + preparation.test( + "ddl.rtas.enabled", + "With replace.enabled=true, CREATE OR REPLACE TABLE AS SELECT replaces the table's " + + "content, leaving only the 2 rows selected by the replacement query.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('replace.enabled'='true')") + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} WHERE ${Core.long0.columnName} <= 2") + + assert( + table.spark.sql(s"SELECT count(*) FROM ${table.name}").collect()(0).getLong(0) == 2, + "RTAS did not replace") + }, + preparation.test( + "ddl.rtas.disabled", + "Without replace.enabled set, CREATE OR REPLACE TABLE AS SELECT is rejected with a " + + "BadRequestException stating RTAS is not enabled.") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name}")) + + assert( + exception.getMessage.contains("REPLACE TABLE AS SELECT is not enabled"), + s"msg: ${exception.getMessage.take(160)}") + }, + preparation.test( + "ddl.rtas.replicationConflict", + "With replace.enabled=true but a replication policy also set, CREATE OR REPLACE TABLE " + + "AS SELECT is rejected with a BadRequestException about replication being enabled.") { + table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('replace.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") + + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name}")) + + assert( + exception.getMessage.contains("while replication is enabled"), + s"msg: ${exception.getMessage.take(160)}") + }) + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDmlScenarios.scala new file mode 100644 index 000000000..b225e557f --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDmlScenarios.scala @@ -0,0 +1,16 @@ +package harness + +// The RTAS DML buckets. Each bucket is a replace-lineage preparation list crossed with one of the +// shared DML test-case lists that DmlScenarios names, so a replaced table runs the same operations +// and the same assertions as a freshly created one. +trait RtasDmlScenarios extends RtasScenarioKit { this: DmlScenarios => + + lazy val rtasDmlCases: List[Plan.Case] = + preparedRtasCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) ++ + preparedNullStringRtasCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) + + lazy val rtasPartitionedDmlCases: List[Plan.Case] = + preparedRtasPartitionedCoreTables.flatMap(preparation => + partitionedTableTestCases.map(_.runOn(preparation))) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasHazardScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasHazardScenarios.scala new file mode 100644 index 000000000..1d7291dc7 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasHazardScenarios.scala @@ -0,0 +1,57 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// The RTAS hazard family. A column tag policy is set on a table, the table is then replaced through +// CREATE OR REPLACE TABLE AS SELECT, and the case reads the policy back. The cases run on parquet +// and orc. +trait RtasHazardScenarios extends RtasScenarioKit { this: HazardReaderWriterScenarios => + import Rows._ + + def hazardRtasCases(format: String): List[Plan.Case] = { + val taggedReplacePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => cowCreate(table, format))() + .insert(3)() + .sql("enableReplace")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")() + .sql("tagPii")(table => + s"ALTER TABLE $table MODIFY COLUMN " + + s"${Core.string0.columnName} SET TAG = (PII)")(), + description = s"Three seed rows in a $format table with replace.enabled set and the string " + + "column tagged PII.") + + List( + taggedReplacePreparation.test( + "hazard.rtas.preservesColumnTags", + "CREATE OR REPLACE TABLE AS SELECT preserves the PII column tag policy that was set " + + "before the replace.") { table => + val policiesBefore = + tableProps(table.spark, table.name).getOrElse("policies", "") + assert( + policiesBefore.toLowerCase.contains("pii") || + policiesBefore.toLowerCase.contains("columntags"), + s"PII tag was not stored before RTAS: $policiesBefore") + + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val policiesAfter = + tableProps(table.spark, table.name).getOrElse("policies", "") + + assert( + policiesAfter == policiesBefore, + s"RTAS should preserve the PII column tag: $policiesAfter") + }) + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasInteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasInteractionScenarios.scala new file mode 100644 index 000000000..60557fa13 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasInteractionScenarios.scala @@ -0,0 +1,390 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// The RTAS interaction family. Each case composes CREATE OR REPLACE TABLE AS SELECT with another +// table state or another operation: an evolved schema, a snapshot reference, a maintenance +// procedure, or a REST lock. The cases run on parquet and orc. +trait RtasInteractionScenarios extends RtasScenarioKit { + import Rows._ + + def interactionRtasCases(format: String): List[Plan.Case] = { + val basePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)(), + description = s"Three seed rows in a $format table.") + val replacePreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableReplace")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")(), + description = s"Three seed rows in a $format table with replace.enabled set to true.") + val userPropertyPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', " + + "'replace.enabled'='true', 'user.key'='v1')")() + .insert(3)(), + description = s"Three seed rows in a $format table with replace.enabled set to true and a " + + "user property user.key=v1.") + val retentionPolicyPreparation = TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"PARTITIONED BY (${Core.datePartition.columnName}) " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', 'replace.enabled'='true')")() + .insert(3)() + .sql("setRetention")(table => + s"ALTER TABLE $table SET POLICY " + + s"(RETENTION = 30d ON COLUMN ${Core.datePartition.columnName} " + + "WHERE pattern = 'yyyy-MM-dd-HH')")(), + description = s"Three seed rows in a $format table partitioned by datepartition, with " + + "replace.enabled set to true and a 30-day retention policy on datepartition.") + + List( + replacePreparation.test( + "interact.rtas.historyPreserved", + "CREATE OR REPLACE TABLE AS SELECT keeps the pre-replace snapshot in history: two " + + "snapshots exist afterward and the pre-replace one still reads 3 rows.") { table => + val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val snapshotCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.snapshots") + .collect()(0) + .getLong(0) + val historicalRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF $preReplaceSnapshotId") + .collect()(0) + .getLong(0) + + assert( + snapshotCount == 2, + s"replace should retain two snapshots, got $snapshotCount") + assert( + historicalRowCount == 3, + s"pre-replace snapshot should contain 3 rows, got $historicalRowCount") + }, + replacePreparation.test( + "interact.rtas.restoreRejected", + "Rolling back to a snapshot from before CREATE OR REPLACE TABLE AS SELECT is rejected " + + "because it is not an ancestor of the current snapshot.") { table => + val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val exception = Check.intercept[ValidationException]( + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $preReplaceSnapshotId)")) + + assert( + exception.getMessage.contains("not an ancestor"), + "rollback across replacement should reject the old lineage") + }, + replacePreparation.test( + "interact.rtas.setCurrentRecovery", + "set_current_snapshot to a pre-replace snapshot recovers the pre-replace 3 rows.") { table => + val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + table.spark.sql( + "CALL openhouse.system.set_current_snapshot(" + + s"'${catalogRelative(table.name)}', $preReplaceSnapshotId)") + val recoveredRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + recoveredRowCount == 3, + s"set_current_snapshot should recover 3 rows, got $recoveredRowCount") + }, + replacePreparation.test( + "interact.rtas.writeAfter", + "A table replaced by CREATE OR REPLACE TABLE AS SELECT accepts an insert immediately " + + "afterward, and the row count reflects both the replacement's rows and the new insert.") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + rowCount == 3, + s"replaced table should contain 3 rows after insert, got $rowCount") + }, + replacePreparation.test( + "interact.rtas.partitionSpecChange", + "CREATE OR REPLACE TABLE AS SELECT with a new PARTITIONED BY clause replaces the " + + "partition specification and preserves all 3 rows.") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"PARTITIONED BY (${Core.datePartition.columnName}) " + + s"AS SELECT * FROM ${table.name}") + val description = table.spark + .sql(s"DESCRIBE TABLE ${table.name}") + .collect() + .toSeq + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + description.exists(_.getString(0) == "# Partition Information") && + description.count( + _.getString(0) == Core.datePartition.columnName) == 2, + "RTAS should replace the partition specification") + assert( + rowCount == 3, + s"partition-spec replacement should preserve 3 rows, got $rowCount") + }, + basePreparation.test( + "interact.rtas.dropsColumn", + "CREATE OR REPLACE TABLE AS SELECT with a narrower column list projects a separate table " + + "down to those two columns while preserving all 3 rows.") { table => + val sideTable = s"${table.name}_dropcol" + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + try { + table.spark.sql( + s"CREATE TABLE $sideTable USING $dataSource " + + "TBLPROPERTIES ('replace.enabled'='true') " + + s"AS SELECT * FROM ${table.name}") + table.spark.sql( + s"CREATE OR REPLACE TABLE $sideTable USING $dataSource AS " + + s"SELECT ${Core.long0.columnName}, ${Core.string0.columnName} " + + s"FROM $sideTable") + val columns = table.spark + .sql(s"SELECT * FROM $sideTable LIMIT 1") + .columns + .toSeq + val rowCount = table.spark + .sql(s"SELECT count(*) FROM $sideTable") + .collect()(0) + .getLong(0) + + assert( + columns == Seq(Core.long0.columnName, Core.string0.columnName), + s"RTAS should project the table to two columns, got $columns") + assert( + rowCount == 3, + s"column-drop RTAS should preserve 3 rows, got $rowCount") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + } + }, + userPropertyPreparation.test( + "interact.rtas.props.userSurvival", + "CREATE OR REPLACE TABLE AS SELECT with no TBLPROPERTIES clause preserves the existing " + + "user.key and replace.enabled properties.") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val properties = tableProps(table.spark, table.name) + + assert( + properties.get("user.key").contains("v1"), + s"user.key did not survive RTAS: ${properties.get("user.key")}") + assert( + properties.get("replace.enabled").contains("true"), + "replace.enabled did not survive RTAS") + }, + userPropertyPreparation.test( + "interact.rtas.props.statementWins", + "CREATE OR REPLACE TABLE AS SELECT with a TBLPROPERTIES clause overrides the matching " + + "existing property while properties absent from the statement survive unchanged.") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + "TBLPROPERTIES ('user.key'='v2') " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val properties = tableProps(table.spark, table.name) + + assert( + properties.get("user.key").contains("v2"), + s"statement property should win, got ${properties.get("user.key")}") + assert( + properties.get("replace.enabled").contains("true"), + "properties omitted from RTAS should survive") + }, + replacePreparation.test( + "interact.rtas.props.createDefaulting", + "CREATE OR REPLACE TABLE AS SELECT with write.format.default=orc sets that property, " + + "keeps format-version at 2, and the replaced table remains writable.") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + "TBLPROPERTIES ('write.format.default'='orc') " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val properties = tableProps(table.spark, table.name) + + assert( + properties.get("write.format.default").contains("orc"), + "RTAS should set write.format.default to orc") + assert( + properties.get("format-version").forall(_ == "2"), + s"format-version drifted: ${properties.get("format-version")}") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 3, + "RTAS table using ORC should remain writable") + }, + retentionPolicyPreparation.test( + "interact.rtas.props.preservesRetentionPolicy", + "CREATE OR REPLACE TABLE AS SELECT with a new partition spec preserves the table's UUID " + + "and its retention policy.") { table => + val tableUuidBefore = tableProps(table.spark, table.name) + .getOrElse("openhouse.tableUUID", "") + val policiesBefore = tableProps(table.spark, table.name) + .getOrElse("policies", "") + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"PARTITIONED BY (${Core.datePartition.columnName}) " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val properties = tableProps(table.spark, table.name) + + assert( + properties.getOrElse("openhouse.tableUUID", "") == + tableUuidBefore, + "table UUID should survive RTAS") + assert( + properties.getOrElse("policies", "") == policiesBefore, + "RTAS should preserve the retention policy") + }, + replacePreparation + .test( + "interact.rtas.withBranch", + "CREATE OR REPLACE TABLE AS SELECT while a branch exists is rejected because branching " + + "is enabled, and both main and the branch remain exactly as they were before the " + + "attempt.") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} CREATE BRANCH keepbr") + table.spark.sql( + s"INSERT INTO ${table.name}.branch_keepbr VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val mainStateBefore = table.state + val branchRowCountBefore = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'keepbr'") + .collect()(0) + .getLong(0) + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2")) + val branchRowCountAfter = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} VERSION AS OF 'keepbr'") + .collect()(0) + .getLong(0) + + assert( + exception.getMessage.contains("while branching is enabled"), + s"msg: ${exception.getMessage.take(160)}") + assert( + table.state == mainStateBefore, + "rejected RTAS should not change the main table") + assert( + branchRowCountAfter == branchRowCountBefore, + "rejected RTAS should not change the branch") + } + .copy(knownBugReason = Some( + "The guide documents CREATE OR REPLACE TABLE AS SELECT as incompatible with an " + + "existing branch. The current product accepts the statement. This case keeps the " + + "documented rejection as the contract so the gap is visible; it is skipped until the " + + "product enforces the rejection."))) + } + + // The REST lock has no SQL surface, so this case runs directly against a Ctx like the other + // control-plane cases. The lock must reject both a normal write and RTAS. + def interactRtasOnLockedTable(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = s"${ctx.namespace}.t_lockrtas" + val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) + spark.sql(s"DROP TABLE IF EXISTS $table") + spark.sql(coreCreateParquet(table)) + spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") + spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')") + try { + val (lockStatus, lockBody) = Rest.post(ctx, s"/v1/databases/$db/tables/$tbl/lock", """{"locked":true}""") + assert(lockStatus >= 200 && lockStatus < 300, s"lock POST failed: $lockStatus $lockBody") + val blocked = Check.intercept[Exception](spark.sql( + s"UPDATE $table SET ${Core.string0.columnName} = 'x' WHERE ${Core.long0.columnName} = 1")) + assert(Exceptions.causeChain(blocked).exists(t => Option(t.getMessage).exists(_.toLowerCase.contains("locked"))), + s"lock not enforced on UPDATE: ${blocked.getMessage.take(160)}") + val rowCountBefore = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) + val snapshotCountBefore = + spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) + val replaceFailure = Check.intercept[BadRequestException]( + spark.sql( + s"CREATE OR REPLACE TABLE $table USING $dataSource " + + s"AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2")) + + assert( + replaceFailure.getMessage.toLowerCase.contains("locked"), + s"RTAS rejection did not identify the lock: ${replaceFailure.getMessage.take(160)}") + assert( + spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == rowCountBefore, + "rejected RTAS changed the table rows") + assert( + spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) == + snapshotCountBefore, + "rejected RTAS committed a snapshot") + } finally { + Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/lock") + spark.sql(s"DROP TABLE IF EXISTS $table") + } + } + + val interactionContextCases: List[Plan.Case] = + List( + Plan.Case( + "interact.rtas.onLockedTable @ embedded", + interactRtasOnLockedTable, + description = "While a table is REST-locked, both UPDATE and CREATE OR REPLACE TABLE AS " + + "SELECT are rejected, and the table keeps the same rows and snapshots.")) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasScenarioKit.scala new file mode 100644 index 000000000..faf85a98e --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasScenarioKit.scala @@ -0,0 +1,56 @@ +package harness + +// The RTAS preparation kit. A replace-lineage table is created, seeded, and then re-specified by +// CREATE OR REPLACE TABLE AS SELECT, so every case that runs on one of these preparations exercises +// the replace path. The members are lazy so they initialize on first read, after every trait mixed +// into `object Scenarios` has been constructed. +trait RtasScenarioKit extends ScenarioKit { + + def createAndSeedRtas(partitioning: Partitioning, numberOfRows: Int, format: String): TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES ('write.format.default'='$format', 'replace.enabled'='true')")() + .insert(numberOfRows)() + .sql("prep.rtas")(t => s"CREATE OR REPLACE TABLE $t USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES ('write.format.default'='$format') AS SELECT * FROM $t")() + // The OpenHouse user guide requires REFRESH TABLE after a replace: the Spark session caches + // the table state it read before the replace, and REFRESH re-reads the committed metadata + // pointer so later statements in the session see the replaced table. + .sql("prep.rtas.refresh")(t => s"REFRESH TABLE $t")() + + // Create and seed, then CREATE OR REPLACE ... AS SELECT * re-specifying the same shape, so the + // table holds the same three rows and was reached through the replace path. The cases run on all + // six layouts, so a replaced table supports the same operations as a freshly created one. + private def rtasPreparationDescription(partitioning: Partitioning, format: String): String = + s"Three seed rows with keys 1, 2 and 3 in a $format table ${partitioning.description}, then " + + "replaced by CREATE OR REPLACE TABLE AS SELECT re-specifying the same shape, so the table " + + "holds the same three rows on replace lineage." + + lazy val preparedRtasCoreTables: List[TablePreparation[CoreTable.type]] = + for { + partitioning <- partitionings + format <- fileFormats + } yield TablePreparation( + s"${partitioning.label}/$format", + createAndSeedRtas(partitioning, 3, format), + "prep.rtas:", + description = rtasPreparationDescription(partitioning, format)) + + lazy val preparedRtasPartitionedCoreTables: List[TablePreparation[CoreTable.type]] = + fileFormats.map { format => + TablePreparation( + s"${partitionedByDate.label}/$format", + createAndSeedRtas(partitionedByDate, 3, format), + "prep.rtas:", + description = rtasPreparationDescription(partitionedByDate, format)) + } + + lazy val preparedNullStringRtasCoreTables: List[TablePreparation[CoreTable.type]] = + preparedRtasCoreTables.map(withNullStringRow) + + lazy val rtasLayoutFormatPreparations: List[TablePreparation[CoreTable.type]] = + preparedRtasCoreTables + + def rtasLayoutFormatCases: List[Plan.Case] = + layoutFormatCasesFor(rtasLayoutFormatPreparations) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasSurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasSurfaceScenarios.scala new file mode 100644 index 000000000..470063891 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasSurfaceScenarios.scala @@ -0,0 +1,120 @@ +package harness + +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.iceberg.exceptions.BadRequestException +import org.apache.iceberg.exceptions.ValidationException +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import scala.annotation.tailrec +import scala.reflect.{ClassTag, classTag} +import scala.util.control.NonFatal + +// The RTAS surface families. One case reads back the message the catalog returns when replace is +// disabled, alongside the other rejection messages a user meets; the other races a replace against +// an append. Both drive CREATE OR REPLACE TABLE AS SELECT, so they belong to the RTAS layer. The +// seeded preparation and the concurrency helpers come from the standard surface trait. The cases run +// on parquet and orc. +trait RtasSurfaceScenarios extends RtasScenarioKit { this: SurfaceScenarios => + import Rows._ + + // A rejection a SQL user reads back is readable when the message is non-empty, carries no + // internal-error marker, carries no raw stack frames, and starts with something other than + // java.lang.NullPointerException. + private def assertReadableMessage(context: String)(e: Throwable): Unit = { + val m = Option(e.getMessage).getOrElse("") + assert(m.nonEmpty, s"$context: empty error message (worst possible readability)") + assert(!m.contains("[INTERNAL_ERROR]"), s"$context: internal error surfaced to the user: ${m.take(160)}") + assert(!m.contains("\n\tat ") && !m.contains("\tat java."), s"$context: stacktrace frames in the user-facing message: ${m.take(160)}") + assert(!m.startsWith("java.lang.NullPointerException"), s"$context: bare NPE surfaced: ${m.take(160)}") + } + + private def surfaceReplacePreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)() + .sql("enableReplace")(table => + s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")(), + description = s"Three seed rows in an unpartitioned $format table with " + + "replace.enabled=true.") + + // The rejection messages a user reads back from the catalog. + def surfaceMessageCases(format: String): List[Plan.Case] = + List( + surfaceBasePreparation(format).test( + "surface.msg.readabilityGuard", + "Rejection messages for a dropped column, a reserved property, disabled RTAS and " + + "CREATE NAMESPACE are all non-empty, free of internal-error markers, free of raw " + + "stack frames, and not a bare NullPointerException.") { table => + assertReadableMessage("dropColumn")( + Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} " + + s"DROP COLUMN ${Core.int0.columnName}"))) + assertReadableMessage("reservedProp")( + Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + + "('openhouse.tableUUID'='x')"))) + assertReadableMessage("rtasDisabled")( + Check.intercept[Exception]( + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name}"))) + assertReadableMessage("createNamespace")( + Check.intercept[Exception]( + table.spark.sql("CREATE NAMESPACE openhouse.nope_ns"))) + }) + + // A replace racing an append. The outcome is either a commit or a typed commit conflict. + def surfaceRtasConcurrencyCases(format: String): List[Plan.Case] = + List( + surfaceReplacePreparation(format).test( + "surface.conc.rtasVsAppend", + "A concurrent CREATE OR REPLACE TABLE AS SELECT racing an INSERT settles at either 2 " + + "rows (replace won) or 3 rows (append also landed), with any failure being a typed " + + "commit conflict.") { table => + def replaceTable(): Unit = + try { + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"AS SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + } catch { + case exception: Throwable => + assert( + isTypedCommitConflict(exception), + s"RTAS race failed with ${exception.getClass.getName}") + } + def appendRow(): Unit = + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(30 AS BIGINT), 30, 'row-30', 30.5, " + + "true, '2024-01-09-01')") + } catch { + case exception: Throwable => + assert( + isTypedCommitConflict(exception), + s"append race failed with ${exception.getClass.getName}") + } + val threadErrors = + runConcurrently(Seq(() => replaceTable(), () => appendRow())) + + assert( + threadErrors.isEmpty, + s"racing thread failed with a non-conflict error: $threadErrors") + table.spark.sql(s"REFRESH TABLE ${table.name}") + val rowCount = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}").toLong + assert( + rowCount == 2 || rowCount == 3, + s"RTAS and append race settled at $rowCount rows") + println(s"DIAG conc.rtasVsAppend: settled at $rowCount rows") + }) +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala index 2e2f37bbb..412a2dd58 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala @@ -10,299 +10,236 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -// Shared foundation for every Scenario trait: the table/layout/prep "kit". All domain traits -// (DmlScenarios, ForkScenarios, ...) extend this, so mixing them into `object Scenarios` puts -// ScenarioKit first in the linearization → its vals initialize before any domain's, exactly as -// in the original single object. `protected` members are the shared kit; `public` ones are also -// consumed by `object Plan`. +// Shared foundation for every Scenario trait: the standard table/layout/prep "kit". All domain +// traits (DmlScenarios, ForkScenarios, ...) extend this, so mixing them into `object Scenarios` puts +// ScenarioKit first in the linearization, so its vals initialize before any domain's, exactly as +// in the original single object. It holds the copy-on-write layouts and preparations only; each +// feature layer carries its own kit that extends this one. `protected` members are the shared kit; +// `public` ones are also consumed by `object Plan`. trait ScenarioKit { import Rows._ protected val Core = CoreTable // brevity in the typed column references below protected val cols = Core.columnNames.mkString(", ") // source column list, so renames propagate - // Short typed views of the current rows, keyed by the long column, for incremental assertions. - protected def keyed(rows: Seq[Row]): Seq[Long] = rows.map(_.get(Core.long0)).sorted - protected def longToString(rows: Seq[Row]): Map[Long, String] = - rows.map(row => row.get(Core.long0) -> row.get(Core.string0)).toMap + // The rows a case reads back are ordered by the long column and carry exactly the core columns in + // their declared order, so an expected row set is written as the rows the case started from, + // filtered, mapped through `withColumnValue`, extended with literal rows, and re-sorted. Both + // helpers address columns by position so they also work on the literal rows a case writes out. + private def columnPosition(column: Column[_]): Int = Core.columnNames.indexOf(column.columnName) - // ── the layout axis: file format x partitioning, crossed with every operation ────────── - // Each layout is a plain literal CREATE statement (no dynamic assembly): the column list is one - // shared literal `columnDefinitions`, and format/partition are literal fragments. createSchema - // cross-checks the literal against CoreTable's declared columns, so the two can't silently drift. + protected def withColumnValue[T](row: Row, column: Column[T], value: T): Row = + Row.fromSeq(row.toSeq.updated(columnPosition(column), value)) + + protected def inKeyOrder(rows: Seq[Row]): Seq[Row] = + rows.sortBy(_.getLong(columnPosition(Core.long0))) + + // --- layouts: one file format and one partitioning per starting table shape --- + // A layout is one starting table shape. Each layout is a plain literal CREATE statement: the + // column list is one shared literal `columnDefinitions`, and format and partitioning are literal + // fragments. createSchema cross-checks the literal against CoreTable's declared columns, so the + // two stay in step. A layout belongs to the preparation, so one test case is written once and + // runs on every layout. protected val columnDefinitions = "foo_col_long bigint, foo_col_int int, foo_col_string string, foo_col_double double, foo_col_boolean boolean, datepartition string" - final case class Layout(label: String, create: String => String) + /** One starting table shape: the label that names it in a case id, a human description of the + * resulting table, and the CREATE statement that builds it. */ + final case class Layout(label: String, description: String, create: String => String) + + object Layout { + /** A layout whose label already reads as its description. */ + def apply(label: String, create: String => String): Layout = Layout(label, label, create) + } + + /** One partitioning choice: the label that names it in a case id, a human description, and the + * CREATE clause that applies it. */ + final case class Partitioning(label: String, description: String, clause: String) + + protected val unpartitioned = Partitioning("unpartitioned", "with no partitioning", "") + + protected val partitionedByDate = + Partitioning("partitioned", "partitioned by datepartition", "PARTITIONED BY (datepartition)") - protected val partitionVariants = List("unpartitioned" -> "", "partitioned" -> "PARTITIONED BY (datepartition)") + protected val partitionings: List[Partitioning] = List(unpartitioned, partitionedByDate) + + protected val fileFormats: List[String] = List("parquet", "orc", "avro") + + private def coreLayout(partitioning: Partitioning, format: String): Layout = + Layout( + s"${partitioning.label}/$format", + s"a copy-on-write $format table ${partitioning.description}", + table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES ('write.format.default'='$format')") val layouts: List[Layout] = for { - format <- List("parquet", "orc", "avro") - (partitionLabel, partitionClause) <- partitionVariants - } yield Layout(s"$partitionLabel/$format", table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource $partitionClause " + - s"TBLPROPERTIES ('write.format.default'='$format')") - - // Merge-on-read layouts: same shapes, but DELETE/UPDATE/MERGE write position-delete files - // (format v2) instead of rewriting data files. Crossed with the mutation operations only. - val morLayouts: List[Layout] = + format <- fileFormats + partitioning <- partitionings + } yield coreLayout(partitioning, format) + + val partitionedLayouts: List[Layout] = + fileFormats.map(format => coreLayout(partitionedByDate, format)) + + // Parquet and ORC layouts for bespoke DDL cases that do not need the full format cross. + val parquetAndOrcLayouts: List[Layout] = for { - format <- List("parquet", "orc", "avro") - (partitionLabel, partitionClause) <- partitionVariants - } yield Layout(s"mor-$partitionLabel/$format", table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource $partitionClause " + - s"TBLPROPERTIES ('write.format.default'='$format', 'format-version'='2', " + - s"'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read', 'write.merge.mode'='merge-on-read')") - - // Dedicated layouts for the CoW/MoR *physical* discriminator (below). Both pin - // `write.distribution-mode=none` and are unpartitioned so a single seed INSERT lands all rows in - // ONE data file; deleting a strict subset is then necessarily a PARTIAL-file match, which Iceberg - // cannot satisfy by whole-file elimination. That makes the physical outcome deterministic: MoR - // must add a position-delete file, CoW must rewrite the data file and add none. (The general - // `morLayouts` seed splits across files, so a boundary-aligned delete can legitimately drop a - // whole file with no position delete — correct Iceberg behaviour, but not what we want to pin.) - val morVerifyLayouts: List[Layout] = - List("parquet", "orc", "avro").map(format => Layout(s"mor-verify/$format", table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'format-version'='2', 'write.distribution-mode'='none', " + - s"'write.delete.mode'='merge-on-read')")) - - val cowVerifyLayouts: List[Layout] = - List("parquet", "orc", "avro").map(format => Layout(s"cow-verify/$format", table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'format-version'='2', 'write.distribution-mode'='none', " + - s"'write.delete.mode'='copy-on-write')")) - - // Preparation: create under `layout` and seed `numberOfRows` deterministic rows. Interchangeable - // with RTAS / drop+undrop preparations later — same resulting state. + format <- List("parquet", "orc") + partitioning <- partitionings + } yield coreLayout(partitioning, format) + + // Create the table under `layout`, then seed deterministic rows as a second visible step. def createAndSeed(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = TableTest(Core).sql("create")(layout.create)().insert(numberOfRows)() val preparedCoreTables: List[TablePreparation[CoreTable.type]] = - layouts.map(layout => TablePreparation(layout.label, createAndSeed(layout, 3))) + layouts.map(layout => + TablePreparation( + layout.label, + createAndSeed(layout, 3), + description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}.")) - val preparedMorCoreTables: List[TablePreparation[CoreTable.type]] = - morLayouts.map(layout => TablePreparation(layout.label, createAndSeed(layout, 3))) + val preparedPartitionedCoreTables: List[TablePreparation[CoreTable.type]] = + partitionedLayouts.map(layout => + TablePreparation( + layout.label, + createAndSeed(layout, 3), + description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, " + + "one row per datepartition value.")) val preparedOrderedCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation( layout.label, createAndSeedOrdered(layout, 3), - "prep.ordered:")) + "prep.ordered:", + description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + + s"ALTER TABLE WRITE ORDERED BY ${Core.long0.columnName}, so the table carries that write sort order.")) val preparedEvolvedCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation( layout.label, createAndSeedEvolved(layout, 3), - "prep.evolved:")) + "prep.evolved:", + description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + + "ADD COLUMN prep_extra int, so the table carries one column beyond the seed row shape " + + "and the seeded rows read null for it.")) val preparedEmptyCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => - TablePreparation(layout.label, TableTest(Core).sql("create")(layout.create)())) + TablePreparation( + layout.label, + TableTest(Core).sql("create")(layout.create)(), + description = s"${layout.description.capitalize} that is created and left unseeded, so it holds no rows.")) val preparedCoreFormats: List[TablePreparation[CoreTable.type]] = - layouts - .filter(layout => - layout.label == "unpartitioned/parquet" || layout.label == "unpartitioned/orc") - .map(layout => - TablePreparation(layout.label.stripPrefix("unpartitioned/"), createAndSeed(layout, 3))) - - // Preparation for the physical CoW/MoR discriminator: seed all rows into ONE data file. A plain - // seed INSERT fans the rows across a couple of files (writer-dependent), so a strict-subset delete - // can land on a whole file and be satisfied by file elimination rather than a position delete. The - // `COALESCE(1)` hint forces a single write task → a single data file, so deleting a strict subset - // is deterministically a PARTIAL-file match: MoR must add a position-delete file, CoW must rewrite. - def createAndSeedSingleFile(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = - TableTest(Core).sql("create")(layout.create)() - .sql(s"seed($numberOfRows, one-file)")(table => - s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM (${RowGenerator.valuesClause(Core, numberOfRows)}) AS seed")( - view => assert(view.after.size == numberOfRows, - s"single-file seed expected $numberOfRows rows, got ${view.after.size}")) - - // Phase 24 preparation multipliers: a DDL evolves the starting state, then a DML op runs on it. - // Ordered prep (sort order) is arity-neutral → crosses ALL operations. Evolved prep adds a column - // → INSERT arity changes, so it crosses only ops that don't re-insert all columns (delete/update/read). + List("parquet", "orc").map { format => + val layout = coreLayout(unpartitioned, format) + TablePreparation( + format, + createAndSeed(layout, 3), + description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}.") + } + + // A DDL step evolves the starting state, and the test case then runs on the evolved table. The + // ordered preparation adds a write sort order and leaves the column list intact, so every DML case + // runs on it. The evolved preparation adds a column, so it runs the cases that address columns by + // name: reads, deletes, and updates. def createAndSeedOrdered(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = createAndSeed(layout, numberOfRows).sql("prep.ordered")(t => s"ALTER TABLE $t WRITE ORDERED BY ${CoreTable.long0.columnName}")() def createAndSeedEvolved(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = createAndSeed(layout, numberOfRows).sql("prep.evolved")(t => s"ALTER TABLE $t ADD COLUMN prep_extra int")() - // Branch-routing prep (the T axis, wap-conf mechanism): seed on main, fork a branch, then set - // spark.wap.branch so the ENTIRE downstream operation (writes AND reads) routes to the branch — - // no per-op rewrite needed. The op's delta assertions are relative to view.before (also the - // branch), so they hold unchanged. Each case runs in its own spark.newSession() (parallel runner), - // so the conf never leaks across cases. This crosses the whole DML catalog onto a branch. - def createAndSeedOnBranch(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = - createAndSeed(layout, numberOfRows) - .sql("prep.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("prep.routeToBranch") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH b") - spark.conf.set("spark.wap.branch", "b") - }() - - private def assertBranchMainIsolation(table: PreparedTable[CoreTable.type]): Unit = { - table.spark.conf.unset("spark.wap.branch") - val mainCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - assert( - mainCount == 3, - s"branch operation leaked to main: expected 3 rows, got $mainCount") - } - - val preparedBranchCoreTables: List[TablePreparation[CoreTable.type]] = - layouts.map { layout => - TablePreparation( - layout.label, - createAndSeedOnBranch(layout, 3), - "branchWap:", - assertBranchMainIsolation) - } - - val preparedBranchMorCoreTables: List[TablePreparation[CoreTable.type]] = - morLayouts - .filter(_.label.startsWith("mor-unpartitioned/")) - .map { layout => - TablePreparation( - layout.label, - createAndSeedOnBranch(layout, 3), - "branchWap:", - assertBranchMainIsolation) - } - - // RTAS prep prefix (the P axis, replace-lineage leg — SURFACE-APPRAISAL step 2): create + seed, - // then CREATE OR REPLACE ... AS SELECT * re-specifying the SAME shape, so the table is - // functionally identical but reached via the replace path (the path G9/G10 showed misbehaves). - // Every downstream DML op then runs on a replace-lineage table. FULL CROSS (Phase 28): all 6 layouts - // ({unpartitioned,partitioned} × {parquet,orc,avro}) — mirrors the core `dml` block's layout coverage so - // the RTAS/replace-lineage substrate carries the same DML surface as the plain CREATE substrate. - // (label, partitionClause, format). - val rtasPrepShapes: List[(String, String, String)] = - for { (pl, pc) <- partitionVariants; fmt <- List("parquet", "orc", "avro") } yield (s"$pl/$fmt", pc, fmt) - - // MoR-read prep (closes the review's "reads on MoR with deletes is a distinct scan path" gap — - // SURFACE-APPRAISAL step 1). The current MoR bucket runs mutation ops (each reads back once), but - // never crosses the READ variants against a table carrying a LIVE position delete. Seed a single - // data file (COALESCE(1)) on a MoR layout, delete a strict subset → a position-delete file the - // reader must APPLY at scan time (not a whole-file elimination). Downstream read ops then assert - // the deleted row is excluded under each read shape (projection, filter-pushdown, ...). - def createAndSeedMorDeleted(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = - createAndSeedSingleFile(layout, numberOfRows) - .step("prep.morDelete") { (spark, table) => - spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") // strict subset → position delete - } { view => - assert(view.after.size == numberOfRows - 1, s"MoR prep delete failed: ${view.after.size}") - val deleteFiles = view.spark.sql(s"SELECT count(*) FROM ${view.table}.all_delete_files").collect()(0).getLong(0) - assert(deleteFiles == 1, s"MoR prep must leave a live position-delete file, got $deleteFiles") + // The same starting state with one more row appended, whose string column is null. A DELETE that + // selects rows by IS NULL is then written as one operation against a table that already holds a + // null string. + protected def withNullStringRow( + basePreparation: TablePreparation[CoreTable.type] + ): TablePreparation[CoreTable.type] = + basePreparation.copy( + preparation = basePreparation.preparation.sql("prep.nullStringRow")(table => + s"INSERT INTO $table VALUES (CAST(99 AS BIGINT), 99, NULL, 99.5, false, '2024-01-01-00')")(), + description = s"${basePreparation.description} A fourth row with key 99 is then appended " + + s"whose ${Core.string0.columnName} is null, so exactly one row of the table reads null for " + + "that column.") + + val preparedNullStringCoreTables: List[TablePreparation[CoreTable.type]] = + preparedCoreTables.map(withNullStringRow) + + val preparedNullStringOrderedCoreTables: List[TablePreparation[CoreTable.type]] = + preparedOrderedCoreTables.map(withNullStringRow) + + // This list validates that each preparation writes data files in its declared format. It runs on + // every preparation that leaves data files behind. Each feature layer owns the list for its + // preparations and builds it through this shared case body. + def layoutFormatCasesFor( + preparations: List[TablePreparation[CoreTable.type]] + ): List[Plan.Case] = + preparations.map { preparation => + preparation.test( + "format.materialization", + "Every data file the preparation wrote carries the extension of the table's declared " + + "write.format.default, and listing the files leaves the table state unchanged.") { table => + val before = table.state + val declaredFormat = table.spark + .sql(s"SHOW TBLPROPERTIES ${table.name} ('write.format.default')") + .collect()(0) + .getString(1) + val filePaths = table.spark + .sql(s"SELECT file_path FROM ${table.name}.files") + .collect() + .toSeq + .map(_.getString(0)) + val after = table.state + + assert( + filePaths.nonEmpty && filePaths.forall(_.toLowerCase.endsWith(s".$declaredFormat")), + s"data files are not all .$declaredFormat: $filePaths") + assert(after == before, "listing files leaves the rows and the snapshot count unchanged") } - - val preparedMorReadCoreTables: List[TablePreparation[CoreTable.type]] = - morVerifyLayouts.map { layout => - TablePreparation( - layout.label, - createAndSeedMorDeleted(layout, 3), - "prep.morRead:") } - // Undrop prep (the P axis, drop→undrop leg — SURFACE-APPRAISAL, requires embedded real HTS). Seed a - // plain table, then take it through the FULL soft-delete → restore round-trip on the real HTS, and - // hand the RESTORED table to the downstream op. The point is a modality audit: every feature's state - // (rows, snapshot lineage, refs, spec, sort order, properties, MoR delete files, schema) must survive - // the round-trip, so the whole DML/DDL catalog is crossed onto the restored table. Soft-delete is - // driven directly on HTS (customer DROP hard-deletes); restore uses the customer Tables API. - def createAndSeedUndropped(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = - createAndSeed(layout, numberOfRows) - .step("prep.undrop") { (spark, table) => - val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) - val (sdCode, sdBody) = HtsAdmin.softDelete(db, tbl) - assert(sdCode >= 200 && sdCode < 300, s"HTS soft-delete failed ($sdCode): $sdBody") - val deletedAtMs = HtsAdmin.softDeletedAtMs(db, tbl) - .getOrElse(throw new AssertionError(s"soft-deleted table $db.$tbl not found in querySoftDeleted")) - val (rCode, rBody) = HtsAdmin.restore(db, tbl, deletedAtMs) - assert(rCode >= 200 && rCode < 300, s"restore failed ($rCode): $rBody") - } { view => - assert(view.after.size == numberOfRows, - s"restored table must keep its $numberOfRows rows, got ${view.after.size}") - } + val layoutFormatPreparations: List[TablePreparation[CoreTable.type]] = + preparedCoreTables ++ preparedOrderedCoreTables - val preparedUndroppedCoreTables: List[TablePreparation[CoreTable.type]] = - layouts - .filter(layout => - layout.label.endsWith("/parquet") || - layout.label.endsWith("/orc")) - .map { layout => - TablePreparation( - layout.label, - createAndSeedUndropped(layout, 3), - "undrop:") - } + def layoutFormatCases: List[Plan.Case] = layoutFormatCasesFor(layoutFormatPreparations) - def createAndSeedRtas(partitionClause: String, numberOfRows: Int, format: String = "parquet"): TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource $partitionClause " + - s"TBLPROPERTIES ('write.format.default'='$format', 'replace.enabled'='true')")() - .insert(numberOfRows)() - .sql("prep.rtas")(t => s"CREATE OR REPLACE TABLE $t USING $dataSource $partitionClause " + - s"TBLPROPERTIES ('write.format.default'='$format') AS SELECT * FROM $t")() - // Iceberg documents CREATE OR REPLACE ... AS SELECT as ATOMIC on a SparkCatalog, so the client - // should observe a consistent table afterward with no manual refresh. On the embedded catalog it - // does; on the OpenHouse REST-backed catalog the client can retain a stale metadata pointer across - // the replace (surfacing downstream as a 400 "incorrect version" or "table not found after - // refresh"). REFRESH re-reads the committed pointer so the suite is robust to that catalog - // divergence; the divergence itself is filed as a product bug (see Remote Test Findings). - .sql("prep.rtas.refresh")(t => s"REFRESH TABLE $t")() - - // RTAS prep on a MERGE-ON-READ table (over-prune miss #1): the replace re-specifies the MoR delete/ - // update/merge modes, so downstream mutation ops exercise the MoR write path on a replace-lineage - // table. Non-vacuous per the appraisal — replace + MoR is a distinct combination. - protected def morPropsFmt(format: String) = s"'write.format.default'='$format', 'format-version'='2', " + - "'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read', 'write.merge.mode'='merge-on-read'" - - def createAndSeedRtasMor(partitionClause: String, numberOfRows: Int, format: String = "parquet"): TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource $partitionClause " + - s"TBLPROPERTIES (${morPropsFmt(format)}, 'replace.enabled'='true')")() - .insert(numberOfRows)() - .sql("prep.rtasMor")(t => s"CREATE OR REPLACE TABLE $t USING $dataSource $partitionClause " + - s"TBLPROPERTIES (${morPropsFmt(format)}) AS SELECT * FROM $t")() - // See createAndSeedRtas: REFRESH after the atomic replace guards the shared suite against the - // OpenHouse catalog's stale-pointer divergence (filed as a product bug). - .sql("prep.rtasMor.refresh")(t => s"REFRESH TABLE $t")() - - val preparedRtasCoreTables: List[TablePreparation[CoreTable.type]] = - rtasPrepShapes.map { - case (label, partitionClause, format) => - TablePreparation( - label, - createAndSeedRtas(partitionClause, 3, format), - "prep.rtas:") + private def waitForNextSnapshotTimestamp(spark: SparkSession, table: String): Unit = { + val previousTimestamp = spark + .sql( + s"SELECT committed_at FROM $table.snapshots " + + "ORDER BY committed_at DESC LIMIT 1") + .collect()(0) + .getTimestamp(0) + .getTime + val deadline = System.nanoTime() + java.util.concurrent.TimeUnit.SECONDS.toNanos(5) + + while ( + System.currentTimeMillis() <= previousTimestamp && + System.nanoTime() < deadline) { + Thread.sleep(1L) } - val preparedRtasMorCoreTables: List[TablePreparation[CoreTable.type]] = - List("parquet", "orc", "avro").map { format => - TablePreparation( - s"mor-unpartitioned/$format", - createAndSeedRtasMor("", 3, format), - "prep.rtasMor:") - } + assert( + System.currentTimeMillis() > previousTimestamp, + s"clock did not advance beyond snapshot timestamp $previousTimestamp") + } - // ── hoisted shared helpers (used across domain traits) ── + // Shared helpers used across domain traits. protected def coreTwoSnapshots(fmt: String): TableTest[CoreTable.type] = TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')")() .insert(3)() + .step("waitForNextSnapshotTimestamp")(waitForNextSnapshotTimestamp)() .sql("insertMore")(table => s"INSERT INTO $table VALUES " + s"(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")() - // No-arg overload (parquet) keeps the many existing single-format call sites unchanged. + protected def coreTwoSnapshots: TableTest[CoreTable.type] = coreTwoSnapshots("parquet") - // Snapshots in ancestry order (root first), following the parent_id chain — deterministic even + // Snapshots in ancestry order (root first), following the parent_id chain. This is deterministic even // if two commits happen to share a committed_at millisecond (which `ORDER BY committed_at` is not). protected def snapshotIds(spark: SparkSession, table: String): Seq[Long] = { val rows = spark.sql(s"SELECT snapshot_id, parent_id FROM $table.snapshots").collect().toSeq @@ -322,33 +259,13 @@ trait ScenarioKit { protected val L = CoreTable.long0.columnName - // The Spark datasource short-name for `CREATE TABLE ... USING `. Defaults to "iceberg" — the - // Apache Iceberg DataSourceRegister short-name that OSS OpenHouse registers and that every OpenHouse - // itest uses. A downstream environment whose shaded runtime relocates the Iceberg datasource to a - // different short-name (for example to let multiple Iceberg libraries coexist on one classpath) would - // find that `USING iceberg` does not resolve there. This is a plain `var` — not a runtime knob — that an - // environment adapter overrides purely in code (for example `Scenarios.dataSource = "openhouse"`) once, - // before it builds `Plan.cases`. The emitted SQL is otherwise byte-identical across environments. + // The Spark data source used by CREATE TABLE statements. The LinkedIn adapter overrides this before + // building Plan.cases. Catalog procedure calls still use the catalog name "openhouse". var dataSource: String = "iceberg" protected def coreCreateParquet(table: String): String = s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='parquet')" - protected def undropSeed(ctx: Ctx, name: String): (String, String, String) = { - val table = s"${ctx.namespace}.$name" - val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) - ctx.spark.sql(s"DROP TABLE IF EXISTS $table") - ctx.spark.sql(coreCreateParquet(table)) - ctx.spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") - (table, db, tbl) - } - - protected def softDeleteRestore(ctx: Ctx, db: String, tbl: String): Unit = { - assert(HtsAdmin.softDelete(db, tbl)._1 / 100 == 2, s"soft-delete $db.$tbl failed") - val ms = HtsAdmin.softDeletedAtMs(db, tbl).getOrElse(throw new AssertionError(s"no deletedAtMs for $db.$tbl")) - assert(HtsAdmin.restore(db, tbl, ms)._1 / 100 == 2, s"restore $db.$tbl failed") - } - protected def tableProps(spark: SparkSession, table: String): Map[String, String] = spark.sql(s"SHOW TBLPROPERTIES $table").collect().toSeq.map(r => r.getString(0) -> r.getString(1)).toMap diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala index 5348f8918..39abe1f75 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala @@ -10,32 +10,55 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal +// The standard surface families. A surface case pins one edge of what the catalog exposes on a +// plain copy-on-write table: a reader, a procedure, a metadata table, a concurrency outcome, a +// schema change, or a write property. The concurrency helpers below are feature neutral, so a +// feature layer reuses them through a self-type on this trait. The cases run on parquet and orc. trait SurfaceScenarios extends ScenarioKit { import Rows._ - - // Audit-B regression guard: a rejection message shown to a SQL user must not be a raw stacktrace, - // an [INTERNAL_ERROR], or a bare NPE. (It may still be MEH — jargony — that's tracked separately.) - private def assertReadableMessage(context: String)(e: Throwable): Unit = { - val m = Option(e.getMessage).getOrElse("") - assert(m.nonEmpty, s"$context: empty error message (worst possible readability)") - assert(!m.contains("[INTERNAL_ERROR]"), s"$context: internal error surfaced to the user: ${m.take(160)}") - assert(!m.contains("\n\tat ") && !m.contains("\tat java."), s"$context: stacktrace frames in the user-facing message: ${m.take(160)}") - assert(!m.startsWith("java.lang.NullPointerException"), s"$context: bare NPE surfaced: ${m.take(160)}") - } - - private def runConcurrently(functions: Seq[() => Unit]): Seq[Throwable] = { + protected def runConcurrently(functions: Seq[() => Unit]): Seq[Throwable] = { val errors = new java.util.concurrent.ConcurrentLinkedQueue[Throwable]() - val threads = functions.map(function => - new Thread(() => - try function() - catch { case throwable: Throwable => errors.add(throwable) })) + val start = new java.util.concurrent.CountDownLatch(1) + val threads = functions.zipWithIndex.map { case (function, index) => + val thread = new Thread( + () => + try { + start.await() + function() + } catch { + case interrupted: InterruptedException => + Thread.currentThread().interrupt() + errors.add(interrupted) + case throwable: Throwable => + errors.add(throwable) + }, + s"delta-harness-concurrent-$index") + thread.setDaemon(true) + thread + } threads.foreach(_.start()) - threads.foreach(_.join(180000)) + start.countDown() + + val deadline = + System.nanoTime() + java.util.concurrent.TimeUnit.MINUTES.toNanos(3) + threads.foreach { thread => + val remainingNanos = deadline - System.nanoTime() + if (remainingNanos > 0) { + java.util.concurrent.TimeUnit.NANOSECONDS.timedJoin(thread, remainingNanos) + } + } + + threads.filter(_.isAlive).foreach { thread => + errors.add( + new AssertionError( + s"${thread.getName} did not complete within 3 minutes")) + thread.interrupt() + } errors.toArray(Array.empty[Throwable]).toSeq } - private def isTypedCommitConflict(throwable: Throwable): Boolean = + protected def isTypedCommitConflict(throwable: Throwable): Boolean = Exceptions.causeChain(throwable).exists { cause => val className = cause.getClass.getName className.contains("CommitFailed") || @@ -45,327 +68,20 @@ trait SurfaceScenarios extends ScenarioKit { className.contains("WebClientResponse") } - private def surfaceBranchCases(format: String): List[Plan.Case] = { - val basePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - val twoSnapshotPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("insertMore")(table => - s"INSERT INTO $table VALUES " + - "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + - "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) - val wapPreparation = TablePreparation( + // Each surface family builds the starting states it needs, so a family reads on its own. The + // seeded table is the plainest of them, so the feature layers build their cases on it too. + protected def surfaceBasePreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( format, TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableWap")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")()) - - List( - twoSnapshotPreparation.test( - "surface.maint.compactWithBranch") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH cb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_cb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - val compactionResult = table.spark - .sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('min-input-files', '2'))") - .collect()(0) - - println( - "DIAG compactWithBranch: " + - s"mainCompaction rewritten=${compactionResult.get(0)} " + - s"added=${compactionResult.get(1)}") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "6", - "main compaction should preserve 6 rows") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") == "6", - "main compaction should preserve the branch") - - table.spark.conf.set("spark.wap.branch", "cb") - val branchRoutedOutcome = - try { - val result = table.spark - .sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}')") - .collect()(0) - s"RAN (rewritten=${result.get(0)}, added=${result.get(1)})" - } catch { - case exception: Throwable => - s"THREW ${exception.getClass.getSimpleName} :: " + - Option(exception.getMessage).getOrElse("").take(140) - } finally { - table.spark.conf.unset("spark.wap.branch") - } - println(s"DIAG compactUnderWapConf: $branchRoutedOutcome") - - table.spark.sql(s"REFRESH TABLE ${table.name}") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "6", - "branch-routed compaction attempt should preserve main") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") == "6", - "branch-routed compaction attempt should preserve the branch") - }, - basePreparation.test("surface.msg.readabilityGuard") { table => - assertReadableMessage("dropColumn")( - Check.intercept[Exception]( - table.spark.sql( - s"ALTER TABLE ${table.name} " + - s"DROP COLUMN ${Core.int0.columnName}"))) - assertReadableMessage("reservedProp")( - Check.intercept[Exception]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('openhouse.tableUUID'='x')"))) - assertReadableMessage("rtasDisabled")( - Check.intercept[Exception]( - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name}"))) - assertReadableMessage("createNamespace")( - Check.intercept[Exception]( - table.spark.sql("CREATE NAMESPACE openhouse.nope_ns"))) - }, - basePreparation.test("branch.leak.setProps") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH lb2") - table.spark.conf.set("spark.wap.branch", "lb2") - try { - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('user.leaked'='yes')") - } finally { - table.spark.conf.unset("spark.wap.branch") - } + .insert(3)(), + description = s"Three seed rows with keys 1, 2 and 3 in an unpartitioned $format table.") - assert( - tableProps(table.spark, table.name) - .get("user.leaked") - .contains("yes"), - "branch-routed property update should change table-global metadata") - }, - basePreparation.test("branch.leak.writeOrderedBy") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH lb3") - table.spark.conf.set("spark.wap.branch", "lb3") - try { - table.spark.sql( - s"ALTER TABLE ${table.name} " + - s"WRITE ORDERED BY ${Core.long0.columnName}") - } finally { - table.spark.conf.unset("spark.wap.branch") - } - - assert( - tableProps(table.spark, table.name) - .get("write.distribution-mode") - .contains("range"), - "branch-routed ordering should change table-global metadata") - }, - wapPreparation.test("branch.wapToggle.noGuard") { table => - table.spark.conf.set("spark.wap.id", "w9") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val stagedSnapshotCount = countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w9'") - assert( - stagedSnapshotCount == "1", - s"expected one staged snapshot, got $stagedSnapshotCount") - - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='false')") - val stagedAfterToggle = countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w9'") - - println(s"DIAG wapToggle: stagedAfterToggle=$stagedAfterToggle") - }, - wapPreparation.test("wap.neg.doubleCherrypick") { table => - table.spark.conf.set("spark.wap.id", "w1") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val stagedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w1'") - .collect()(0) - .getLong(0) - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', ${stagedSnapshotId}L)") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "4", - "first cherry-pick should publish the staged row") - - val exception = Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', ${stagedSnapshotId}L)")) - println( - "DIAG doubleCherrypick: " + - s"${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(180)) - assert( - Option(exception.getMessage).exists(message => - message.toLowerCase.contains("duplicate") || - message.toLowerCase.contains("already")), - "second cherry-pick should reject the duplicate WAP commit") - }, - basePreparation.test("wap.neg.expireRefTarget") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH eb2") - val branchHeadSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.refs " + - "WHERE name = 'eb2'") - .collect()(0) - .getLong(0) - val exception = Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - s"snapshot_ids => ARRAY(${branchHeadSnapshotId}L))")) - - println( - "DIAG expireRefTarget: " + - s"${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(180)) - }, - basePreparation.test("branch.fastForward.merge") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH fb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_fb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_fb VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "branch writes should not advance main") - - table.spark.sql( - "CALL openhouse.system.fast_forward(" + - s"'${catalogRelative(table.name)}', 'main', 'fb')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "5", - "fast_forward should move main to the branch head") - }, - basePreparation.test("branch.fastForward.divergent") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH db") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_db VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - val exception = Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.fast_forward(" + - s"'${catalogRelative(table.name)}', 'main', 'db')")) - - println( - "DIAG ffDivergent: " + - s"${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(180)) - assert( - Option(exception.getMessage).exists(message => - message.toLowerCase.contains("ancestor") || - message.toLowerCase.contains("fast-forward")), - "divergent fast_forward should report an ancestry error") - }, - twoSnapshotPreparation.test("branch.replaceBranch") { table => - val snapshots = snapshotIds(table.spark, table.name) - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH rb2") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb2'") == "5", - "new branch should point at the current head") - - table.spark.sql( - s"ALTER TABLE ${table.name} REPLACE BRANCH rb2 " + - s"AS OF VERSION ${snapshots.head}") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb2'") == "3", - "REPLACE BRANCH should retarget the branch to the older snapshot") - }) - } - - private def surfaceReaderProcedureCases( - format: String): List[Plan.Case] = { - val basePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - val twoSnapshotPreparation = TablePreparation( + private def surfaceTwoSnapshotPreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( format, TableTest(Core) .sql("create")(table => @@ -375,36 +91,53 @@ trait SurfaceScenarios extends ScenarioKit { .sql("insertMore")(table => s"INSERT INTO $table VALUES " + "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + - "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) - val emptyPreparation = TablePreparation( + "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")(), + description = s"Five rows across two snapshots (a 3-row seed then a 2-row insert) in an " + + s"unpartitioned $format table.") + + private def surfaceEmptyPreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( format, TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")()) - val morPreparation = TablePreparation( + s"TBLPROPERTIES ('write.format.default'='$format')")(), + description = s"An unseeded, empty unpartitioned $format table.") + + private def surfaceHashPreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( format, TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"PARTITIONED BY (${Core.datePartition.columnName}) " + "TBLPROPERTIES (" + s"'write.format.default'='$format', " + - "'write.delete.mode'='merge-on-read')")() - .sql("seed")(table => - s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM " + - s"(${RowGenerator.valuesClause(Core, 3)}) AS seed")()) - val wapPreparation = TablePreparation( + "'write.distribution-mode'='hash')")() + .insert(3)(), + description = s"Three seed rows in a $format table partitioned by datepartition with " + + "write.distribution-mode=hash.") + + private def surfaceTargetFileSizePreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation( format, TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableWap")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")()) + "TBLPROPERTIES (" + + s"'write.format.default'='$format', " + + "'write.target-file-size-bytes'='1048576')")() + .insert(3)(), + description = s"Three seed rows in an unpartitioned $format table with " + + "write.target-file-size-bytes=1048576.") + // The structured-streaming reader and writer, and the changelog view. + def surfaceReaderCases(format: String): List[Plan.Case] = List( - basePreparation.test("surface.stream.read") { table => + surfaceBasePreparation(format).test( + "surface.stream.read", + "A Spark structured streaming read of the table, run in AvailableNow batch mode, " + + "delivers all 3 seed rows to a memory sink within 120 seconds.") { table => val checkpoint = java.nio.file.Files.createTempDirectory("ck-read").toString val sink = s"memsink_${System.nanoTime}" @@ -424,7 +157,10 @@ trait SurfaceScenarios extends ScenarioKit { countOf(table.spark, s"SELECT count(*) FROM $sink") == "3", "streaming read should deliver the three seed rows") }, - basePreparation.test("surface.stream.write") { table => + surfaceBasePreparation(format).test( + "surface.stream.write", + "A Spark structured streaming append of two rows through the iceberg write-stream " + + "format lands both rows, growing the table from 3 to 5 rows.") { table => import table.spark.implicits._ implicit val sqlContext: org.apache.spark.sql.SQLContext = table.spark.sqlContext @@ -454,7 +190,10 @@ trait SurfaceScenarios extends ScenarioKit { s"SELECT count(*) FROM ${table.name}") == "5", "streaming write should append two rows") }, - twoSnapshotPreparation.test("surface.cdc.changelogView") { table => + surfaceTwoSnapshotPreparation(format).test( + "surface.cdc.changelogView", + "create_changelog_view over an append-only history reports 5 changes, all of change " + + "type INSERT.") { table => val view = table.spark .sql( "CALL openhouse.system.create_changelog_view(" + @@ -477,8 +216,15 @@ trait SurfaceScenarios extends ScenarioKit { assert( changeTypes == Set("INSERT"), s"append-only changelog should contain only INSERT: $changeTypes") - }, - emptyPreparation.test("surface.proc.rewriteManifests") { table => + }) + + // The rewrite procedure that compacts the manifest set. + def surfaceRewriteProcedureCases(format: String): List[Plan.Case] = + List( + surfaceEmptyPreparation(format).test( + "surface.proc.rewriteManifests", + "After 5 single-row inserts fragment the manifest list, rewrite_manifests compacts it " + + "to fewer manifests while preserving all 5 rows.") { table => (1 to 5).foreach(index => table.spark.sql( s"INSERT INTO ${table.name} VALUES " + @@ -508,52 +254,14 @@ trait SurfaceScenarios extends ScenarioKit { manifestCountBefore >= 2 && manifestCountAfter < manifestCountBefore, "rewrite_manifests should compact the manifest set") - }, - morPreparation.test( - "surface.proc.rewritePositionDeletes") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.all_delete_files") == "1", - "MoR delete should create one position-delete file") - - table.spark.sql( - "CALL openhouse.system.rewrite_position_delete_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('rewrite-all', 'true'))") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "2", - "rewrite_position_delete_files should preserve live rows") - }, - wapPreparation.test("surface.proc.publishChanges") { table => - table.spark.conf.set("spark.wap.id", "pw1") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - } finally { - table.spark.conf.unset("spark.wap.id") - } - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "staged write should not be visible before publish") + }) - table.spark.sql( - "CALL openhouse.system.publish_changes(" + - s"table => '${catalogRelative(table.name)}', wap_id => 'pw1')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "4", - "publish_changes should publish the staged row") - }, - twoSnapshotPreparation.test("surface.proc.ancestorsOf") { table => + // The procedures that read snapshot ancestry and remove orphan files. + def surfaceSnapshotProcedureCases(format: String): List[Plan.Case] = + List( + surfaceTwoSnapshotPreparation(format).test( + "surface.proc.ancestorsOf", + "ancestors_of lists both snapshots of the table's two-snapshot history.") { table => val ancestorCount = table.spark .sql( "CALL openhouse.system.ancestors_of(" + @@ -565,7 +273,10 @@ trait SurfaceScenarios extends ScenarioKit { ancestorCount == 2, s"ancestors_of should list two snapshots, got $ancestorCount") }, - basePreparation.test("surface.proc.removeOrphanReal") { table => + surfaceBasePreparation(format).test( + "surface.proc.removeOrphanReal", + "remove_orphan_files deletes a planted, backdated stray file next to a real data file " + + "while the table's 3 live rows remain intact.") { table => val dataFile = table.spark .sql(s"SELECT file_path FROM ${table.name}.files LIMIT 1") .collect()(0) @@ -594,8 +305,16 @@ trait SurfaceScenarios extends ScenarioKit { table.spark, s"SELECT count(*) FROM ${table.name}") == "3", "remove_orphan_files should preserve live data") - }, - basePreparation.test("surface.meta.hiddenColumns") { table => + }) + + // The hidden metadata columns and the Iceberg metadata tables. + def surfaceMetadataCases(format: String): List[Plan.Case] = + List( + surfaceBasePreparation(format).test( + "surface.meta.hiddenColumns", + "Selecting the hidden metadata columns _file, _pos, _spec_id and _partition returns " + + "one row per seed row, each with a populated file path and a non-negative position.") { + table => val rows = table.spark .sql( s"SELECT _file, _pos, _spec_id, _partition FROM ${table.name}") @@ -613,7 +332,11 @@ trait SurfaceScenarios extends ScenarioKit { rows.forall(_.getLong(1) >= 0), "_pos should be non-negative for every row") }, - twoSnapshotPreparation.test("surface.meta.tableSweep") { table => + surfaceTwoSnapshotPreparation(format).test( + "surface.meta.tableSweep", + "Every Iceberg metadata table (entries, files, manifests, snapshots, history, refs, " + + "partitions, and their all_* variants) is queryable without error, and the snapshots " + + "metadata table reports the table's 2 snapshots.") { table => val metadataTables = Seq( "entries", "files", @@ -629,71 +352,26 @@ trait SurfaceScenarios extends ScenarioKit { "all_entries", "all_files") metadataTables.foreach { metadataTable => - val rowCount = table.spark + table.spark .sql( s"SELECT count(*) FROM ${table.name}.`$metadataTable`") - .collect()(0) - .getLong(0) - assert( - rowCount >= 0, - s"metadata table $metadataTable should be queryable") + .collect() } assert( countOf( table.spark, s"SELECT count(*) FROM ${table.name}.snapshots") == "2", "snapshot metadata should contain two snapshots") - }, - morPreparation.test("surface.meta.positionDeletes") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.position_deletes") == "1", - "position_deletes should expose the MoR position delete") }) - } - - private def surfaceRemainingCases(format: String): List[Plan.Case] = { - val basePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - val replacePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableReplace")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")()) - val hashPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.datePartition.columnName}) " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', " + - "'write.distribution-mode'='hash')")() - .insert(3)()) - val targetSizePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', " + - "'write.target-file-size-bytes'='1048576')")() - .insert(3)()) + // Two writers racing on one table. Every outcome is either a commit or a typed commit conflict. + def surfaceConcurrencyCases(format: String): List[Plan.Case] = List( - basePreparation.test("surface.conc.appendAppend") { table => + surfaceBasePreparation(format).test( + "surface.conc.appendAppend", + "Two threads concurrently insert 3 rows each; every insert either commits or fails " + + "with a typed commit-conflict exception, and the final row count matches 3 plus the " + + "number of inserts that actually committed.") { table => val failureCount = new java.util.concurrent.atomic.AtomicInteger(0) def writer(base: Int): () => Unit = () => @@ -730,7 +408,11 @@ trait SurfaceScenarios extends ScenarioKit { s"DIAG conc.appendAppend: ${failureCount.get}/6 inserts " + "hit a typed commit conflict") }, - basePreparation.test("surface.conc.updateUpdate") { table => + surfaceBasePreparation(format).test( + "surface.conc.updateUpdate", + "Two threads concurrently UPDATE the same row to different values; the row count stays " + + "at 3, and the final value is one of the two competing updates or the original seed " + + "value, with any failure being a typed commit conflict.") { table => val column = Core.string0.columnName def updater(value: String): () => Unit = () => try { @@ -766,48 +448,15 @@ trait SurfaceScenarios extends ScenarioKit { table.spark, s"SELECT count(*) FROM ${table.name}") == "3", "concurrent updates should not change row count") - }, - replacePreparation.test("surface.conc.rtasVsAppend") { table => - def replaceTable(): Unit = - try { - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - } catch { - case exception: Throwable => - assert( - isTypedCommitConflict(exception), - s"RTAS race failed with ${exception.getClass.getName}") - } - def appendRow(): Unit = - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(30 AS BIGINT), 30, 'row-30', 30.5, " + - "true, '2024-01-09-01')") - } catch { - case exception: Throwable => - assert( - isTypedCommitConflict(exception), - s"append race failed with ${exception.getClass.getName}") - } - val threadErrors = - runConcurrently(Seq(() => replaceTable(), () => appendRow())) + }) - assert( - threadErrors.isEmpty, - s"racing thread failed with a non-conflict error: $threadErrors") - table.spark.sql(s"REFRESH TABLE ${table.name}") - val rowCount = countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}").toLong - assert( - rowCount == 2 || rowCount == 3, - s"RTAS and append race settled at $rowCount rows") - println(s"DIAG conc.rtasVsAppend: settled at $rowCount rows") - }, - basePreparation.test("surface.schema.relaxNotNull") { table => + // Schema changes that Iceberg allows and the ones the catalog rejects. + def surfaceSchemaCases(format: String): List[Plan.Case] = + List( + surfaceBasePreparation(format).test( + "surface.schema.relaxNotNull", + "On a side table, dropping NOT NULL from a column allows a subsequent insert of a null " + + "value for that column.") { table => val sideTable = s"${table.name}_nn" table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") try { @@ -828,7 +477,10 @@ trait SurfaceScenarios extends ScenarioKit { table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") } }, - basePreparation.test("surface.schema.decimalWiden") { table => + surfaceBasePreparation(format).test( + "surface.schema.decimalWiden", + "On a side table, widening a decimal column's precision preserves the original row and " + + "accepts a new row whose value only fits the wider precision.") { table => val sideTable = s"${table.name}_dec" table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") try { @@ -853,7 +505,10 @@ trait SurfaceScenarios extends ScenarioKit { table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") } }, - basePreparation.test("surface.schema.nestedAddField") { table => + surfaceBasePreparation(format).test( + "surface.schema.nestedAddField", + "On a side table, ADD COLUMN of a new nested struct field null-fills it for the " + + "existing row and accepts a new row that sets the field.") { table => val sideTable = s"${table.name}_nst" table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") try { @@ -886,7 +541,10 @@ trait SurfaceScenarios extends ScenarioKit { table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") } }, - basePreparation.test("surface.schema.nestedDropField") { table => + surfaceBasePreparation(format).test( + "surface.schema.nestedDropField", + "On a side table, ALTER TABLE DROP COLUMN of a nested struct field is rejected with an " + + "exception, and the field remains readable afterward.") { table => val sideTable = s"${table.name}_nsd" table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") try { @@ -896,14 +554,10 @@ trait SurfaceScenarios extends ScenarioKit { table.spark.sql( s"INSERT INTO $sideTable VALUES " + "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") - val exception = Check.intercept[Exception]( + Check.intercept[Exception]( table.spark.sql( s"ALTER TABLE $sideTable DROP COLUMN s.x")) - println( - "DIAG nestedDropField: " + - s"${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(180)) assert( table.spark .sql(s"SELECT s.x FROM $sideTable") @@ -914,7 +568,10 @@ trait SurfaceScenarios extends ScenarioKit { table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") } }, - basePreparation.test("surface.schema.reorderExisting") { table => + surfaceBasePreparation(format).test( + "surface.schema.reorderExisting", + "ALTER TABLE ALTER COLUMN ... FIRST moves that column to the front of the schema while " + + "preserving all 3 rows.") { table => table.spark.sql( s"ALTER TABLE ${table.name} " + s"ALTER COLUMN ${Core.string0.columnName} FIRST") @@ -931,8 +588,15 @@ trait SurfaceScenarios extends ScenarioKit { table.spark, s"SELECT count(*) FROM ${table.name}") == "3", "column reorder should preserve the rows") - }, - hashPreparation.test("surface.write.distributionHash") { table => + }) + + // The write-planning properties: distribution mode and target file size. + def surfaceWriteCases(format: String): List[Plan.Case] = + List( + surfaceHashPreparation(format).test( + "surface.write.distributionHash", + "The write.distribution-mode=hash property requested at creation is retained and the " + + "table holds its 3 seed rows.") { table => val properties = tableProps(table.spark, table.name) val rowCount = table.spark .sql(s"SELECT count(*) FROM ${table.name}") @@ -946,7 +610,10 @@ trait SurfaceScenarios extends ScenarioKit { rowCount == 3, s"hash-distributed seed should contain 3 rows, got $rowCount") }, - targetSizePreparation.test("surface.write.targetFileSize") { table => + surfaceTargetFileSizePreparation(format).test( + "surface.write.targetFileSize", + "The write.target-file-size-bytes=1048576 property requested at creation is retained " + + "and the table holds its 3 seed rows.") { table => val properties = tableProps(table.spark, table.name) val rowCount = table.spark .sql(s"SELECT count(*) FROM ${table.name}") @@ -961,108 +628,71 @@ trait SurfaceScenarios extends ScenarioKit { assert( rowCount == 3, s"custom target-size seed should contain 3 rows, got $rowCount") - }, - basePreparation.test("surface.write.dfToBranch") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH wb") - val row = table.spark.sql( - s"SELECT CAST(50 AS BIGINT) AS ${Core.long0.columnName}, " + - s"50 AS ${Core.int0.columnName}, " + - s"'row-50' AS ${Core.string0.columnName}, " + - s"50.5 AS ${Core.double0.columnName}, " + - s"true AS ${Core.boolean0.columnName}, " + - s"'2024-01-09-01' AS ${Core.datePartition.columnName}") - row.writeTo(s"${table.name}.branch_wb").append() + }) - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'wb'") == "4", - "DataFrame writer should append to the branch") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "DataFrame branch write should leave main unchanged") - }, - basePreparation.test("surface.pin.importProcs") { table => + // Pins on the surfaces the catalog rejects: the import procedures, views and ANALYZE TABLE. + def surfacePinCases(format: String): List[Plan.Case] = + List( + surfaceBasePreparation(format).test( + "surface.pin.importProcs", + "register_table onto a new name makes the source table's snapshot readable there " + + "(3 rows) without affecting the source, and dropping the registered table leaves " + + "the source untouched; the system.snapshot and system.add_files procedures are " + + "each confirmed to reject their unsupported inputs with an exception.") { table => + val registeredTable = s"${table.name}_registered" val metadataFile = table.spark .sql( s"SELECT file FROM ${table.name}.metadata_log_entries " + "ORDER BY timestamp DESC LIMIT 1") .collect()(0) .getString(0) - val registerOutcome = - try { - table.spark.sql( - "CALL openhouse.system.register_table(" + - "table => 'dbMatrix.zz_reg', " + - s"metadata_file => '$metadataFile')") - val rowCount = countOf( + + try { + table.spark.sql( + "CALL openhouse.system.register_table(" + + s"table => '${catalogRelative(registeredTable)}', " + + s"metadata_file => '$metadataFile')") + assert( + countOf( table.spark, - "SELECT count(*) FROM openhouse.dbMatrix.zz_reg") + s"SELECT count(*) FROM $registeredTable") == "3", + "register_table should make all source rows readable") + } finally { + try { table.spark.sql( - "DROP TABLE IF EXISTS openhouse.dbMatrix.zz_reg") - s"REGISTERED (readable, $rowCount rows)" + s"DROP TABLE IF EXISTS $registeredTable") } catch { - case exception: Throwable => - s"REJECTED ${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(160) + case NonFatal(_) => () } - println(s"DIAG pin.register_table(real): $registerOutcome") + } + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "dropping the registered table should not remove source rows") - val snapshotException = Check.intercept[Exception]( + Check.intercept[Exception]( table.spark.sql( "CALL openhouse.system.snapshot(" + s"source_table => '${catalogRelative(table.name)}', " + "table => 'dbMatrix.zz_snap')")) - println( - "DIAG pin.snapshot: " + - s"${snapshotException.getClass.getName} :: " + - Option(snapshotException.getMessage).getOrElse("").take(160)) - val addFilesException = Check.intercept[Exception]( + Check.intercept[Exception]( table.spark.sql( "CALL openhouse.system.add_files(" + s"table => '${catalogRelative(table.name)}', " + "source_table => '`parquet`.`/tmp/zz_nope_dir`')")) - println( - "DIAG pin.add_files: " + - s"${addFilesException.getClass.getName} :: " + - Option(addFilesException.getMessage).getOrElse("").take(160)) }, - basePreparation.test("surface.pin.viewsAnalyze") { table => - val viewException = Check.intercept[Exception]( + surfaceBasePreparation(format).test( + "surface.pin.viewsAnalyze", + "CREATE VIEW and ANALYZE TABLE COMPUTE STATISTICS are each rejected with an " + + "exception.") { table => + Check.intercept[Exception]( table.spark.sql( "CREATE VIEW openhouse.dbMatrix.zz_v1 AS SELECT 1 AS one")) - println( - "DIAG pin.createView: " + - s"${viewException.getClass.getName} :: " + - Option(viewException.getMessage).getOrElse("").take(160)) - val analyzeException = Check.intercept[Exception]( + Check.intercept[Exception]( table.spark.sql( s"ANALYZE TABLE ${table.name} COMPUTE STATISTICS")) - println( - "DIAG pin.analyze: " + - s"${analyzeException.getClass.getName} :: " + - Option(analyzeException.getMessage).getOrElse("").take(160)) }) - } - - val surfaceCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - surfaceBranchCases(format) ++ - surfaceReaderProcedureCases(format) ++ - surfaceRemainingCases(format) - } - - // ═══ Hazard demonstrations H1-H8 (MODALITY-RECON.md; gates cleared per FEATURE-ANALYSIS-PLAN) ══ - // Each was PREDICTED by the state-flow model, verified in code/bytecode, and is demonstrated - // live here. Characterizations flip loudly if the product fixes the hazard. - - // H1 — streaming checkpoint × expiration (G11's streaming twin). Three acts: - // (1) stream + checkpoint; (2) CONTROL: plain restart picks up new rows (restart mechanics fine); - // (3) expire past the checkpointed offset → restart is BRICKED with the typed error. - } diff --git a/integrations/spark/delta-harness/src/test/scala/harness/BranchDmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/BranchDmlCaseCatalogTest.scala new file mode 100644 index 000000000..aae1a2eaf --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/BranchDmlCaseCatalogTest.scala @@ -0,0 +1,83 @@ +package harness + +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} +import org.junit.jupiter.api.Test + +/** + * Pins the shape of the branch DML buckets: each one is a branch-routed preparation list crossed + * with a DML test-case list the standard layer names. Reading these lists does not execute a case + * or start Spark. + */ +final class BranchDmlCaseCatalogTest { + + @Test + def eachBucketIsThePreparationListCrossedWithItsTestCaseList(): Unit = { + assertEquals( + caseIds(Scenarios.preparedBranchCoreTables, Scenarios.allDmlTestCases) ++ + caseIds(Scenarios.preparedNullStringBranchCoreTables, Scenarios.nullStringRowTestCases), + Scenarios.branchDmlCases.map(_.id), + "branchDmlCases is not its named preparations crossed with its named test cases") + assertEquals( + caseIds(Scenarios.preparedPartitionedBranchCoreTables, Scenarios.partitionedTableTestCases), + Scenarios.branchPartitionedDmlCases.map(_.id), + "branchPartitionedDmlCases is not its named preparations crossed with its named test cases") + assertEquals( + caseIds(Scenarios.preparedBranchMorCoreTables, Scenarios.rowMutationTestCases) ++ + caseIds(Scenarios.preparedNullStringBranchMorCoreTables, Scenarios.nullStringRowTestCases), + Scenarios.branchMorDmlCases.map(_.id), + "branchMorDmlCases is not its named preparations crossed with its named test cases") + } + + @Test + def everyBranchPreparationDescribesTheRoutingItSetsUp(): Unit = { + val describedPreparations = + Scenarios.preparedBranchCoreTables ++ + Scenarios.preparedPartitionedBranchCoreTables ++ + Scenarios.preparedBranchMorCoreTables + + describedPreparations.foreach { preparation => + assertTrue( + preparation.description.contains("spark.wap.branch"), + s"${preparation.label} does not describe the branch routing it sets up") + } + } + + @Test + def theLayoutFormatCasesRunOnTheBranchPreparations(): Unit = + assertEquals( + caseIds(Scenarios.branchLayoutFormatPreparations, "format.materialization"), + Scenarios.branchLayoutFormatCases.map(_.id)) + + @Test + def everyBranchCaseCarriesItsOwnDescriptionAndItsPreparationDescription(): Unit = { + val describedBuckets = List( + Scenarios.branchDmlCases, + Scenarios.branchPartitionedDmlCases, + Scenarios.branchMorDmlCases, + Scenarios.branchLayoutFormatCases).flatten + + describedBuckets.foreach { testCase => + assertTrue( + testCase.description.trim.nonEmpty, + s"${testCase.id} has no description of the operation it runs") + assertTrue( + testCase.preparationDescription.trim.nonEmpty, + s"${testCase.id} has no description of the state it starts from") + } + } + + private def caseIds( + preparations: List[TablePreparation[CoreTable.type]], + testCases: List[DmlTestCase[CoreTable.type]] + ): List[String] = + preparations.flatMap(preparation => + testCases.map(testCase => + s"${preparation.casePrefix}${testCase.id} @ ${preparation.label}")) + + private def caseIds( + preparations: List[TablePreparation[CoreTable.type]], + testCaseId: String + ): List[String] = + preparations.map(preparation => + s"${preparation.casePrefix}$testCaseId @ ${preparation.label}") +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala index 3fdb41dc3..bab8e1efc 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala @@ -7,13 +7,14 @@ import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} import org.junit.jupiter.api.Test final class CaseCatalogTest { - private val expectedCaseCount = 2574 + private val expectedCaseCount = 2572 private val expectedCatalogSha256 = - "9e5ec513f2bbc775469154c8d1cf45e14654af2fca0e0f29b4bba6acae286a0a" + "ffa5fde92303f703e2f9f7febddfe9e912323c3ade8f95339fd073f89a8028c3" @Test def orderedCaseCatalogMatchesBaseline(): Unit = { - val caseIds = Plan.caseIds + val cases = Plan.cases + val caseIds = cases.map(_.id) val actualCatalogSha256 = sha256(caseIds.mkString("\n")) val duplicateCaseIds = caseIds.groupBy(identity).collect { case (caseId, occurrences) if occurrences.size > 1 => caseId @@ -22,6 +23,9 @@ final class CaseCatalogTest { assertTrue( duplicateCaseIds.isEmpty, s"case IDs must be unique; duplicates=${duplicateCaseIds.mkString(", ")}") + assertTrue( + cases.forall(_.description.trim.nonEmpty), + "every catalog case must describe the behavior it verifies") assertEquals( expectedCaseCount, caseIds.size, diff --git a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala new file mode 100644 index 000000000..c32dec2a4 --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala @@ -0,0 +1,238 @@ +package harness + +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} +import org.junit.jupiter.api.Test + +/** + * Pins the shape the standard DML tests are written in: one list of test cases, one list of + * preparations, and a bucket that is the cross of the two. Each feature layer pins its own buckets + * in its own test. Reading these lists does not execute a case or start Spark. + */ +final class DmlCaseCatalogTest { + private val expectedReadTestCaseIds = List("read.projection", "read.filter") + + private val expectedDeleteTestCaseIds = List( + "delete.byPredicate", + "delete.byInList", + "delete.byInSubquery", + "delete.byNotInSubquery", + "delete.byExistsSubquery", + "delete.byNotExistsSubquery", + "delete.byScalarSubquery", + "delete.all", + "delete.none", + "delete.byPartitionPredicate", + "delete.withAlias", + "delete.whereFalse.noSnapshot", + "delete.truncate", + "delete.atSnapshot.rejected") + + private val expectedUpdateTestCaseIds = List( + "update.byPredicate", + "update.withoutCondition", + "update.noMatch", + "update.byInSubquery", + "update.byNotInSubquery", + "update.byExistsSubquery", + "update.byNotExistsSubquery", + "update.byScalarSubquery", + "update.withAlias", + "update.multipleColumns", + "update.byExpression", + "update.movePartition", + "update.nullAssignment") + + private val expectedMergeTestCaseIds = List( + "merge.insertNotMatched", + "merge.updateMatched", + "merge.deleteMatched", + "merge.upsert", + "merge.deleteNotMatchedBySource", + "merge.conditionalUpdate", + "merge.multipleMatchedClauses", + "merge.conditionalInsert", + "merge.allClauses", + "merge.updateStar", + "merge.insertExplicitColumns", + "merge.sourceCTE", + "merge.sourceSetOp", + "merge.intoEmptyTarget", + "merge.nullJoinKey", + "merge.resolveByName") + + private val expectedDmlTestCaseIds = List( + "read.projection", + "read.filter", + "delete.byPredicate", + "delete.byInList", + "delete.byInSubquery", + "delete.byNotInSubquery", + "delete.byExistsSubquery", + "delete.byNotExistsSubquery", + "delete.byScalarSubquery", + "delete.all", + "delete.none", + "delete.byPartitionPredicate", + "delete.withAlias", + "delete.whereFalse.noSnapshot", + "delete.truncate", + "delete.atSnapshot.rejected", + "update.byPredicate", + "update.withoutCondition", + "update.noMatch", + "update.byInSubquery", + "update.byNotInSubquery", + "update.byExistsSubquery", + "update.byNotExistsSubquery", + "update.byScalarSubquery", + "update.withAlias", + "update.multipleColumns", + "update.byExpression", + "update.movePartition", + "update.nullAssignment", + "merge.insertNotMatched", + "merge.updateMatched", + "merge.deleteMatched", + "merge.upsert", + "merge.deleteNotMatchedBySource", + "merge.conditionalUpdate", + "merge.multipleMatchedClauses", + "merge.conditionalInsert", + "merge.allClauses", + "merge.updateStar", + "merge.insertExplicitColumns", + "merge.sourceCTE", + "merge.sourceSetOp", + "merge.intoEmptyTarget", + "merge.nullJoinKey", + "merge.resolveByName", + "insert.into", + "insert.explicitColumns", + "insert.intoSelect", + "append.dataFrame", + "insert.overwrite", + "overwrite.dataFrame") + + @Test + def everyDmlTestCaseIsListedOnceInOrder(): Unit = { + val caseIds = Scenarios.allDmlTestCases.map(_.id) + + assertEquals(expectedDmlTestCaseIds, caseIds) + assertEquals(caseIds.distinct.size, caseIds.size, s"duplicate DML case id in $caseIds") + } + + @Test + def eachCompatibilityListNamesTheOperationsItsStartingStateSupports(): Unit = { + assertEquals( + expectedDeleteTestCaseIds ++ expectedUpdateTestCaseIds ++ expectedMergeTestCaseIds, + Scenarios.rowMutationTestCases.map(_.id)) + assertEquals( + expectedReadTestCaseIds ++ expectedDeleteTestCaseIds ++ expectedUpdateTestCaseIds, + Scenarios.testCasesCompatibleWithAnAddedColumn.map(_.id)) + assertEquals(expectedReadTestCaseIds, Scenarios.readTestCases.map(_.id)) + assertEquals(List("delete.byNullCondition"), Scenarios.nullStringRowTestCases.map(_.id)) + } + + @Test + def orderedPreparationMarksItsKnownFailingMatrixCellExplicitly(): Unit = { + assertEquals( + List("delete.byPartitionPredicate"), + Scenarios.orderedDmlTestCases.collect { + case testCase if testCase.knownBugReason.nonEmpty => testCase.id + }) + } + + @Test + def theNullStringPreparationDescribesTheRowItAppends(): Unit = { + Scenarios.preparedNullStringCoreTables.foreach { preparation => + assertTrue( + preparation.description.contains("null"), + s"${preparation.label} does not describe the null-string row it appends") + } + } + + @Test + def formatMaterializationIsNotADmlOperation(): Unit = { + assertTrue( + !Scenarios.allDmlTestCases.map(_.id).contains("format.materialization"), + "format.materialization describes the preparation, not an operation run against it") + assertEquals( + caseIds(Scenarios.layoutFormatPreparations, "format.materialization"), + Scenarios.layoutFormatCases.map(_.id)) + } + + @Test + def eachBucketIsThePreparationListCrossedWithItsTestCaseList(): Unit = { + val noNullStringPreparations = List.empty[TablePreparation[CoreTable.type]] + val buckets = List( + ("coreDmlCases", Scenarios.coreDmlCases, Scenarios.preparedCoreTables, Scenarios.allDmlTestCases, Scenarios.preparedNullStringCoreTables), + ("orderedDmlCases", Scenarios.orderedDmlCases, Scenarios.preparedOrderedCoreTables, Scenarios.allDmlTestCases, Scenarios.preparedNullStringOrderedCoreTables), + ("evolvedDmlCases", Scenarios.evolvedDmlCases, Scenarios.preparedEvolvedCoreTables, Scenarios.testCasesCompatibleWithAnAddedColumn, noNullStringPreparations), + ("partitionedDmlCases", Scenarios.partitionedDmlCases, Scenarios.preparedPartitionedCoreTables, Scenarios.partitionedTableTestCases, noNullStringPreparations)) + + buckets.foreach { case (bucketName, bucket, preparations, testCases, nullStringPreparations) => + val expectedIds = + caseIds(preparations, testCases) ++ + caseIds(nullStringPreparations, Scenarios.nullStringRowTestCases) + + assertEquals( + expectedIds, + bucket.map(_.id), + s"$bucketName is not its named preparations crossed with its named test cases") + } + } + + @Test + def everyDmlCaseCarriesItsOwnDescriptionAndItsPreparationDescription(): Unit = { + val describedBuckets = List( + Scenarios.coreDmlCases, + Scenarios.orderedDmlCases, + Scenarios.evolvedDmlCases, + Scenarios.partitionedDmlCases, + Scenarios.layoutFormatCases).flatten + + describedBuckets.foreach { testCase => + assertTrue( + testCase.description.trim.nonEmpty, + s"${testCase.id} has no description of the operation it runs") + assertTrue( + testCase.preparationDescription.trim.nonEmpty, + s"${testCase.id} has no description of the state it starts from") + assertTrue( + testCase.description != testCase.id, + s"${testCase.id} repeats its id; the description must explain the operation") + } + } + + @Test + def everyLayoutDescribesTheTableItCreates(): Unit = { + val describedLayouts = + Scenarios.layouts ++ + Scenarios.partitionedLayouts ++ + Scenarios.parquetAndOrcLayouts + + describedLayouts.foreach { layout => + assertTrue( + layout.description.trim.nonEmpty, + s"layout ${layout.label} has no description") + assertTrue( + layout.description != layout.label, + s"layout ${layout.label} repeats its label; the description must explain the table") + } + } + + private def caseIds( + preparations: List[TablePreparation[CoreTable.type]], + testCases: List[DmlTestCase[CoreTable.type]] + ): List[String] = + preparations.flatMap(preparation => + testCases.map(testCase => + s"${preparation.casePrefix}${testCase.id} @ ${preparation.label}")) + + private def caseIds( + preparations: List[TablePreparation[CoreTable.type]], + testCaseId: String + ): List[String] = + preparations.map(preparation => + s"${preparation.casePrefix}$testCaseId @ ${preparation.label}") +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/MorDmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/MorDmlCaseCatalogTest.scala new file mode 100644 index 000000000..9768ddb2b --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/MorDmlCaseCatalogTest.scala @@ -0,0 +1,96 @@ +package harness + +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} +import org.junit.jupiter.api.Test + +/** + * Pins the shape of the merge-on-read DML buckets: each one is a merge-on-read preparation list + * crossed with a DML test-case list the standard layer names, plus the pair of cases that assert + * the physical difference between the two write modes. Reading these lists does not execute a case + * or start Spark. + */ +final class MorDmlCaseCatalogTest { + + @Test + def eachBucketIsThePreparationListCrossedWithItsTestCaseList(): Unit = { + assertEquals( + caseIds(Scenarios.preparedMorCoreTables, Scenarios.rowMutationTestCases) ++ + caseIds(Scenarios.preparedNullStringMorCoreTables, Scenarios.nullStringRowTestCases), + Scenarios.morDmlCases.map(_.id), + "morDmlCases is not its named preparations crossed with its named test cases") + assertEquals( + caseIds(Scenarios.preparedRtasMorCoreTables, Scenarios.rowMutationTestCases) ++ + caseIds(Scenarios.preparedNullStringRtasMorCoreTables, Scenarios.nullStringRowTestCases), + Scenarios.rtasMorDmlCases.map(_.id), + "rtasMorDmlCases is not its named preparations crossed with its named test cases") + assertEquals( + caseIds(Scenarios.preparedMorReadCoreTables, Scenarios.readTestCases), + Scenarios.morReadDmlCases.map(_.id), + "morReadDmlCases is not its named preparations crossed with its named test cases") + } + + @Test + def theDeleteFileModeBucketPairsOneMergeOnReadCaseWithOneCopyOnWriteCase(): Unit = + assertEquals( + Scenarios.morVerifyLayouts.map(layout => s"mor.writesDeleteFiles @ ${layout.label}") ++ + Scenarios.cowVerifyLayouts.map(layout => s"cow.writesNoDeleteFiles @ ${layout.label}"), + Scenarios.deleteFileModeCases.map(_.id)) + + @Test + def theLayoutFormatCasesRunOnTheMergeOnReadReadPreparations(): Unit = + assertEquals( + caseIds(Scenarios.morReadLayoutFormatPreparations, "format.materialization"), + Scenarios.morReadLayoutFormatCases.map(_.id)) + + @Test + def everyMergeOnReadLayoutDescribesTheTableItCreates(): Unit = { + val describedLayouts = + Scenarios.morLayouts ++ + Scenarios.unpartitionedMorLayouts ++ + Scenarios.morVerifyLayouts ++ + Scenarios.cowVerifyLayouts + + describedLayouts.foreach { layout => + assertTrue( + layout.description.trim.nonEmpty, + s"layout ${layout.label} has no description") + assertTrue( + layout.description != layout.label, + s"layout ${layout.label} repeats its label; the description must explain the table") + } + } + + @Test + def everyMergeOnReadCaseCarriesItsOwnDescriptionAndItsPreparationDescription(): Unit = { + val describedBuckets = List( + Scenarios.morDmlCases, + Scenarios.rtasMorDmlCases, + Scenarios.morReadDmlCases, + Scenarios.deleteFileModeCases, + Scenarios.morReadLayoutFormatCases).flatten + + describedBuckets.foreach { testCase => + assertTrue( + testCase.description.trim.nonEmpty, + s"${testCase.id} has no description of the operation it runs") + assertTrue( + testCase.preparationDescription.trim.nonEmpty, + s"${testCase.id} has no description of the state it starts from") + } + } + + private def caseIds( + preparations: List[TablePreparation[CoreTable.type]], + testCases: List[DmlTestCase[CoreTable.type]] + ): List[String] = + preparations.flatMap(preparation => + testCases.map(testCase => + s"${preparation.casePrefix}${testCase.id} @ ${preparation.label}")) + + private def caseIds( + preparations: List[TablePreparation[CoreTable.type]], + testCaseId: String + ): List[String] = + preparations.map(preparation => + s"${preparation.casePrefix}$testCaseId @ ${preparation.label}") +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/RtasDmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/RtasDmlCaseCatalogTest.scala new file mode 100644 index 000000000..f8b331656 --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/RtasDmlCaseCatalogTest.scala @@ -0,0 +1,72 @@ +package harness + +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} +import org.junit.jupiter.api.Test + +/** + * Pins the shape of the RTAS DML buckets: each one is a replace-lineage preparation list crossed + * with a DML test-case list the standard layer names. Reading these lists does not execute a case + * or start Spark. + */ +final class RtasDmlCaseCatalogTest { + + @Test + def eachBucketIsThePreparationListCrossedWithItsTestCaseList(): Unit = { + assertEquals( + caseIds(Scenarios.preparedRtasCoreTables, Scenarios.allDmlTestCases) ++ + caseIds(Scenarios.preparedNullStringRtasCoreTables, Scenarios.nullStringRowTestCases), + Scenarios.rtasDmlCases.map(_.id), + "rtasDmlCases is not its named preparations crossed with its named test cases") + assertEquals( + caseIds(Scenarios.preparedRtasPartitionedCoreTables, Scenarios.partitionedTableTestCases), + Scenarios.rtasPartitionedDmlCases.map(_.id), + "rtasPartitionedDmlCases is not its named preparations crossed with its named test cases") + } + + @Test + def theReplaceLineagePreparationsDescribeTheReplaceTheyPerform(): Unit = { + Scenarios.preparedRtasCoreTables.foreach { preparation => + assertTrue( + preparation.description.contains("CREATE OR REPLACE TABLE AS SELECT"), + s"${preparation.label} does not describe the replace it performs") + } + } + + @Test + def theLayoutFormatCasesRunOnTheReplaceLineagePreparations(): Unit = + assertEquals( + caseIds(Scenarios.rtasLayoutFormatPreparations, "format.materialization"), + Scenarios.rtasLayoutFormatCases.map(_.id)) + + @Test + def everyRtasCaseCarriesItsOwnDescriptionAndItsPreparationDescription(): Unit = { + val describedBuckets = List( + Scenarios.rtasDmlCases, + Scenarios.rtasPartitionedDmlCases, + Scenarios.rtasLayoutFormatCases).flatten + + describedBuckets.foreach { testCase => + assertTrue( + testCase.description.trim.nonEmpty, + s"${testCase.id} has no description of the operation it runs") + assertTrue( + testCase.preparationDescription.trim.nonEmpty, + s"${testCase.id} has no description of the state it starts from") + } + } + + private def caseIds( + preparations: List[TablePreparation[CoreTable.type]], + testCases: List[DmlTestCase[CoreTable.type]] + ): List[String] = + preparations.flatMap(preparation => + testCases.map(testCase => + s"${preparation.casePrefix}${testCase.id} @ ${preparation.label}")) + + private def caseIds( + preparations: List[TablePreparation[CoreTable.type]], + testCaseId: String + ): List[String] = + preparations.map(preparation => + s"${preparation.casePrefix}$testCaseId @ ${preparation.label}") +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala index c024ac223..77bc12b9f 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala @@ -9,12 +9,41 @@ final class TablePreparationTest { val preparation = TablePreparation( "partitioned/orc", TableTest(CoreTable), - "prep.evolved:") + "prep.evolved:", + description = "Three rows in an evolved ORC table.") - val testCase = preparation.test("delete.byPredicate")(_ => ()) + val testCase = preparation.test( + "delete.byPredicate", + "DELETE removes the rows selected by its predicate.")(_ => ()) assertEquals( "prep.evolved:delete.byPredicate @ partitioned/orc", testCase.id) + assertEquals( + "Three rows in an evolved ORC table.", + testCase.preparationDescription) + assertEquals( + "DELETE removes the rows selected by its predicate.", + testCase.description) + } + + @Test + def runsDescribedDmlCaseOnPreparation(): Unit = { + val preparation = TablePreparation( + "unpartitioned/parquet", + TableTest(CoreTable), + description = "Three rows in an unpartitioned Parquet table.") + val dmlTestCase = DmlTestCase( + "insert.append", + "INSERT appends one row and commits one snapshot.", + (_: PreparedTable[CoreTable.type]) => ()) + + val testCase = dmlTestCase.runOn(preparation) + + assertEquals( + "insert.append @ unpartitioned/parquet", + testCase.id) + assertEquals(dmlTestCase.description, testCase.description) + assertEquals(preparation.description, testCase.preparationDescription) } } From 3dc3763ed33ae2dfa2dc6e3a78499059ab34ee34 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Wed, 26 Aug 2026 13:02:24 -0700 Subject: [PATCH 07/24] refactor(delta-harness): isolate standard cases Keep the standard branch focused on copy-on-write behavior, shared table preparations, bespoke DDL coverage, and the local execution framework. Remove RTAS, merge-on-read, branch, and WAP scenario ownership from this layer. Pin the resulting ordered standard catalog at 1,181 cases so each child branch can add one reviewable feature delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../openhouse/BranchDmlScenarios.scala | 54 -- .../openhouse/BranchHazardScenarios.scala | 148 ---- .../BranchInteractionScenarios.scala | 453 ---------- .../openhouse/BranchMorScenarios.scala | 163 ---- .../harness/openhouse/BranchScenarioKit.scala | 80 -- .../openhouse/BranchSurfaceScenarios.scala | 422 ---------- .../openhouse/BranchWapScenarios.scala | 790 ------------------ .../harness/openhouse/MorDmlScenarios.scala | 107 --- .../harness/openhouse/MorForkScenarios.scala | 72 -- .../openhouse/MorInteractionScenarios.scala | 59 -- .../harness/openhouse/MorMaintScenarios.scala | 480 ----------- .../openhouse/MorReaderWriterScenarios.scala | 194 ----- .../harness/openhouse/MorScenarioKit.scala | 137 --- .../openhouse/MorSurfaceScenarios.scala | 76 -- .../harness/openhouse/OpenHouseMatrix.scala | 24 +- .../main/scala/harness/openhouse/Plan.scala | 47 +- .../harness/openhouse/RtasDdlScenarios.scala | 76 -- .../harness/openhouse/RtasDmlScenarios.scala | 16 - .../openhouse/RtasHazardScenarios.scala | 57 -- .../openhouse/RtasInteractionScenarios.scala | 390 --------- .../harness/openhouse/RtasScenarioKit.scala | 56 -- .../openhouse/RtasSurfaceScenarios.scala | 120 --- .../harness/BranchDmlCaseCatalogTest.scala | 83 -- .../test/scala/harness/CaseCatalogTest.scala | 4 +- .../scala/harness/MorDmlCaseCatalogTest.scala | 96 --- .../harness/RtasDmlCaseCatalogTest.scala | 72 -- 26 files changed, 4 insertions(+), 4272 deletions(-) delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchDmlScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchHazardScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchInteractionScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchMorScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchScenarioKit.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchSurfaceScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorDmlScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorForkScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorInteractionScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorReaderWriterScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorScenarioKit.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorSurfaceScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDdlScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDmlScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasHazardScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasInteractionScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasScenarioKit.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasSurfaceScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/BranchDmlCaseCatalogTest.scala delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/MorDmlCaseCatalogTest.scala delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/RtasDmlCaseCatalogTest.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchDmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchDmlScenarios.scala deleted file mode 100644 index a8bfc9796..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchDmlScenarios.scala +++ /dev/null @@ -1,54 +0,0 @@ -package harness - -// The branch DML buckets. Each bucket is a branch-routed preparation list crossed with one of the -// shared DML test-case lists that DmlScenarios names. A case captures its before state from the -// branch it is routed at, so the same body holds on a branch and on main, and the preparation's own -// isolation check confirms main kept its three seed rows. -trait BranchDmlScenarios extends BranchScenarioKit { this: DmlScenarios => - - lazy val branchDmlCases: List[Plan.Case] = - preparedBranchCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) ++ - preparedNullStringBranchCoreTables.flatMap(preparation => - nullStringRowTestCases.map(_.runOn(preparation))) - - lazy val branchPartitionedDmlCases: List[Plan.Case] = - preparedPartitionedBranchCoreTables.flatMap(preparation => - partitionedTableTestCases.map(_.runOn(preparation))) - - lazy val branchMorDmlCases: List[Plan.Case] = - preparedBranchMorCoreTables.flatMap(preparation => - rowMutationTestCases.map(_.runOn(preparation))) ++ - preparedNullStringBranchMorCoreTables.flatMap(preparation => - nullStringRowTestCases.map(_.runOn(preparation))) - - // The branch a consumer creates after the DDL. It runs against each of the standard DDL-consumer - // preparations, so Plan places it inside that walk. - def branchDdlConsumerCases( - preparation: TablePreparation[CoreTable.type]): List[Plan.Case] = - List( - preparation.test( - "branch", - "A write to a branch created after the DDL takes the branch to four rows and leaves " + - "main on its three rows.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH cb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_cb " + - s"SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") - .collect()(0) - .getLong(0) == 4, - "branch write failed after DDL") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 3, - "branch write changed the main table") - }) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchHazardScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchHazardScenarios.scala deleted file mode 100644 index 8a11bd8d8..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchHazardScenarios.scala +++ /dev/null @@ -1,148 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The branch hazard families. Each case creates a named branch and then runs an operation that could -// disturb it: a retention policy, a table rename, or turning write.wap.enabled off and on again. The -// cases run on parquet and orc. -trait BranchHazardScenarios extends BranchScenarioKit { this: HazardReaderWriterScenarios => - import Rows._ - - def hazardBranchCases(format: String): List[Plan.Case] = { - val partitionedPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.datePartition.columnName}) " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = s"Three seed rows in a $format table partitioned by datepartition.") - val twoSnapshotPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => cowCreate(table, format))() - .insert(3)() - .sql("insertMore")(table => - s"INSERT INTO $table VALUES " + - "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + - "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")(), - description = s"Five seed rows across two snapshots in a copy-on-write $format table.") - val wapPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => cowCreate(table, format))() - .insert(3)() - .sql("enableWap")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")(), - description = s"Three seed rows in a $format table with write.wap.enabled set to true.") - - List( - partitionedPreparation.test( - "hazard.retentionBranch.defended", - "After a branch is created, main is trimmed by DELETE, and snapshot expiration plus " + - "orphan-file removal run, the branch still reads its 3 rows and main reflects the " + - "trimmed row count.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH rbb") - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} <= 2") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - table.spark.sql( - "CALL openhouse.system.remove_orphan_files(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2020-01-01 00:00:00')") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rbb'") == "3", - "branch should remain readable after retention cleanup") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "1", - "main should reflect the retention-shaped delete") - }, - twoSnapshotPreparation.test( - "hazard.rename.consumers", - "After ALTER TABLE RENAME, both a pre-existing branch and pre-existing time travel to an " + - "old snapshot remain readable under the new name, and the renamed table still accepts " + - "writes.") { table => - val snapshots = snapshotIds(table.spark, table.name) - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH rnb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_rnb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val renamedTable = s"${table.name}_rn" - table.spark.sql( - s"ALTER TABLE ${table.name} RENAME TO $renamedTable") - try { - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $renamedTable " + - "VERSION AS OF 'rnb'") == "6", - "branch should survive table rename") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $renamedTable " + - s"VERSION AS OF ${snapshots.head}") == "3", - "time travel should survive table rename") - - table.spark.sql( - s"INSERT INTO $renamedTable VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $renamedTable") == "6", - "renamed table should remain writable") - } finally { - table.spark.sql( - s"ALTER TABLE $renamedTable RENAME TO ${table.name}") - } - }, - wapPreparation.test( - "hazard.wapToggle.branchesSurvive", - "A named branch keeps accumulating its own rows across write.wap.enabled being turned " + - "off, while main stays at its original 3 rows.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH wtb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_wtb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='false')") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_wtb VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'wtb'") == "5", - "named branch should survive disabling WAP") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "branch writes should leave main unchanged") - }) - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchInteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchInteractionScenarios.scala deleted file mode 100644 index 44174a98c..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchInteractionScenarios.scala +++ /dev/null @@ -1,453 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The branch interaction family. Each case composes a branch or a write-audit-publish staged commit -// with another table state or another operation, so the cases show how branch routing behaves -// alongside DDL, snapshot references and maintenance. The cases run on parquet and orc. -trait BranchInteractionScenarios extends BranchScenarioKit { - import Rows._ - - def interactionBranchCases(format: String): List[Plan.Case] = { - val basePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = s"Three seed rows in a $format table.") - val twoSnapshotPreparation = TablePreparation( - format, - coreTwoSnapshots(format), - description = s"Five seed rows across two snapshots in a $format table.") - val wapPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableWap")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")(), - description = s"Three seed rows in a $format table with write.wap.enabled set to true.") - - List( - twoSnapshotPreparation.test( - "interact.branch.ttBeforeBranchPoint", - "After branching and writing to the branch, a snapshot ID or timestamp from before the " + - "branch point still resolves to the pre-branch 3 rows, both on main and while " + - "spark.wap.branch selects the branch.") { table => - val snapshots = snapshotIds(table.spark, table.name) - val firstCommitTimestamp = table.spark - .sql( - s"SELECT CAST(committed_at AS STRING) FROM ${table.name}.snapshots " + - "ORDER BY committed_at LIMIT 1") - .collect()(0) - .getString(0) - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH tb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_tb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'tb'") - .collect()(0) - .getLong(0) == 6, - "branch head should contain 6 rows") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF ${snapshots.head}") - .collect()(0) - .getLong(0) == 3, - "snapshot ID should resolve before the branch point") - - table.spark.conf.set("spark.wap.branch", "tb") - try { - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"TIMESTAMP AS OF '$firstCommitTimestamp'") - .collect()(0) - .getLong(0) == 3, - "explicit timestamp should override spark.wap.branch") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF ${snapshots.head}") - .collect()(0) - .getLong(0) == 3, - "explicit snapshot ID should override spark.wap.branch") - } finally { - table.spark.conf.unset("spark.wap.branch") - } - }, - basePreparation.test( - "interact.branch.mainDdlImmediate", - "ALTER TABLE ADD COLUMN changes the schema seen from a branch immediately, an old-arity " + - "insert into the branch fails afterward, and a new-arity insert matching the added " + - "column succeeds.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_mb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - val branchColumns = table.spark - .sql( - s"SELECT * FROM ${table.name} VERSION AS OF 'mb' LIMIT 1") - .columns - .toSeq - - assert( - branchColumns.contains("extra_col"), - s"main DDL should change the table-global schema: $branchColumns") - - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"INSERT INTO ${table.name}.branch_mb VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')")) - assert( - exception.getMessage.toLowerCase.contains("not enough data columns"), - "old-arity branch writer should fail after main DDL") - - table.spark.sql( - s"INSERT INTO ${table.name}.branch_mb VALUES " + - "(CAST(8 AS BIGINT), 8, 'row-8', 8.5, true, " + - "'2024-01-08-07', 44)") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") - .collect()(0) - .getLong(0) == 5, - "new-arity branch write should succeed after main DDL") - }, - twoSnapshotPreparation.test( - "interact.branch.expireProtectsRefs", - "Snapshot expiration after writes on both main and a branch keeps both ref heads, drops " + - "the intermediate snapshots, and leaves both main and the branch fully readable.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH eb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_eb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}.snapshots") - .collect()(0) - .getLong(0) == 4, - "expected four snapshots before expiration") - - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - val refs = table.spark - .sql(s"SELECT name FROM ${table.name}.refs") - .collect() - .map(_.getString(0)) - .toSet - val snapshotCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.snapshots") - .collect()(0) - .getLong(0) - val branchRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'eb'") - .collect()(0) - .getLong(0) - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert(refs == Set("main", "eb"), s"refs changed: $refs") - assert( - snapshotCount == 2, - s"expiration should retain two ref heads, got $snapshotCount") - assert( - branchRowCount == 6, - s"branch should remain readable with 6 rows, got $branchRowCount") - assert( - mainRowCount == 6, - s"main should remain readable with 6 rows, got $mainRowCount") - }, - twoSnapshotPreparation.test( - "interact.branch.rollbackWhileWapConf", - "Calling rollback_to_snapshot while spark.wap.branch selects a branch still rolls back " + - "main, leaving the branch's own rows unaffected.") { table => - val firstSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH rb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_rb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.conf.set("spark.wap.branch", "rb") - try { - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $firstSnapshotId)") - } finally { - table.spark.conf.unset("spark.wap.branch") - } - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val branchRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb'") - .collect()(0) - .getLong(0) - - assert( - mainRowCount == 3, - s"rollback should target main and restore 3 rows, got $mainRowCount") - assert( - branchRowCount == 6, - s"rollback should leave branch at 6 rows, got $branchRowCount") - }, - twoSnapshotPreparation.test( - "interact.restore.expireAfterRollback", - "After rolling back to the first snapshot, expiring snapshots removes the rolled-past " + - "snapshot and keeps the current 3 rows readable, but time travel to that expired " + - "snapshot now fails.") { table => - val snapshots = snapshotIds(table.spark, table.name) - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', ${snapshots.head})") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - val snapshotCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.snapshots") - .collect()(0) - .getLong(0) - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - snapshotCount == 1, - s"rolled-past snapshot should expire, got $snapshotCount snapshots") - assert( - rowCount == 3, - s"rollback should preserve 3 current rows, got $rowCount") - - val exception = Check.intercept[Exception]( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF ${snapshots(1)}") - .collect()) - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage) - .exists(_.toLowerCase.contains("snapshot"))), - "time travel to the expired rolled-past snapshot should fail") - }, - basePreparation.test( - "interact.branch.expireMerge.spuriousReject", - "Expiring snapshots after two writes to a branch removes the intermediate branch " + - "snapshot but keeps the branch fully readable; fast_forward onto that punctured " + - "ancestry is rejected, and main stays consistent whether or not a cherry-pick recovery " + - "succeeds.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_mb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_mb VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots") == "3", - "expected parent and two branch snapshots") - - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots") == "2", - "expiration should remove the intermediate branch snapshot") - val refs = table.spark - .sql(s"SELECT name FROM ${table.name}.refs") - .collect() - .map(_.getString(0)) - .toSet - assert(refs == Set("main", "mb"), s"refs changed: $refs") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") == "5", - "branch should remain readable after expiration") - - val exception = Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.fast_forward(" + - s"'${catalogRelative(table.name)}', 'main', 'mb')")) - assert( - Option(exception.getMessage).exists(_.contains("not an ancestor")), - "fast_forward should reject the punctured branch ancestry") - - val branchHeadSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.refs WHERE name = 'mb'") - .collect()(0) - .getLong(0) - val cherryPickOutcome = - try { - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', " + - s"${branchHeadSnapshotId}L)") - s"SUCCEEDED: main now ${countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}")} rows" - } catch { - case exception: Throwable => - s"REJECTED ${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(160) - } - println( - s"DIAG expireMerge.cherrypickFallback: $cherryPickOutcome") - val mainRowCount = countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}").toLong - - assert( - mainRowCount == 3 || mainRowCount == 4, - s"main should remain consistent, got $mainRowCount rows") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mb'") == "5", - "branch data should remain available for copy-out recovery") - }, - wapPreparation.test( - "interact.branch.expireMerge.stagedWapLoss", - "Snapshot expiration removes an unreferenced staged WAP snapshot, and publishing that " + - "wap_id afterward fails while main remains at its original 3 rows.") { table => - table.spark.conf.set("spark.wap.id", "w2") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - } finally { - table.spark.conf.unset("spark.wap.id") - } - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w2'") == "1", - "WAP write should create one staged snapshot") - - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w2'") == "0", - "expiration should remove the unreferenced staged snapshot") - - val exception = Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.publish_changes(" + - s"table => '${catalogRelative(table.name)}', wap_id => 'w2')")) - println( - "DIAG stagedWapLoss.publish: " + - s"${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(180)) - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "main should remain unchanged after staged snapshot loss") - }) - } - - // A table created with write.wap.enabled and replace.enabled both set. The case reads those flags - // back, then creates a branch and confirms the replace path is refused while the branch exists. - def interactionBranchFlagCases(format: String): List[Plan.Case] = { - val flagPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', " + - "'write.wap.enabled'='true', 'replace.enabled'='true')")() - .insert(3)(), - description = s"Three seed rows in a $format table with write.wap.enabled and " + - "replace.enabled both set to true at create time.") - - List( - flagPreparation.test( - "interact.flags.wapReplaceAtCreate", - "WAP and replace flags set at CREATE time are active, and a subsequent RTAS is rejected " + - "while a branch exists and WAP is enabled.") { table => - val properties = tableProps(table.spark, table.name) - assert( - properties.get("write.wap.enabled").contains("true") && - properties.get("replace.enabled").contains("true"), - "WAP and replace flags should be active when set at CREATE") - - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH cb") - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name}")) - assert( - exception.getMessage.contains("while WAP"), - "RTAS should reject a table with WAP enabled at CREATE") - }) - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchMorScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchMorScenarios.scala deleted file mode 100644 index 92300062b..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchMorScenarios.scala +++ /dev/null @@ -1,163 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// Branch merges on a merge-on-read table. A DELETE or UPDATE on a branch of a merge-on-read table -// writes position-delete files on the branch, and this family pins what merging that branch back to -// main does with them. It needs both the merge-on-read write modes and branch refs, so it belongs to -// the branch layer that sits above merge-on-read. -trait BranchMorScenarios extends BranchScenarioKit { - import Rows._ - - // Merge-on-read tables with branch merges: a DELETE or UPDATE on a branch of a MoR table writes - // position-delete files on the branch, and merging the branch back to main must carry those - // deletes correctly. The base table is a single-file MoR seed (a coalesced write of 1 file) so a - // strict-subset DELETE produces a real position delete. Merge operates on refs and snapshots, so - // one MoR layout covers the format-independent behavior. Each case checks that deletes are - // carried across the merge, that deleted rows stay absent from main, and how cherry-pick handles - // row-delete snapshots. - lazy val morBranchMergeCases: List[Plan.Case] = - morVerifyLayouts - .filter(layout => - layout.label == "mor-verify/parquet" || - layout.label == "mor-verify/orc") - .map(layout => - TablePreparation( - layout.label, - createAndSeedSingleFile(layout, 3), - description = s"Three seed rows written as one data file in ${layout.description}.")) - .flatMap { preparation => - List( - preparation.test( - "mbranch.fastForwardDelete", - "fast_forward carries a branch's position-delete DELETE onto main: main gains the " + - "branch's 2-row state and the deleted row does not reappear.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mfb") - table.spark.sql( - s"DELETE FROM ${table.name}.branch_mfb " + - s"WHERE ${Core.long0.columnName} = 1") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "main advanced before fast-forward") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mfb'") == "2", - "branch delete was not applied") - - table.spark.sql( - "CALL openhouse.system.fast_forward(" + - s"'${catalogRelative(table.name)}', 'main', 'mfb')") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "2", - "fast-forward did not carry the branch position delete") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") == "0", - "deleted row reappeared after fast-forward") - }, - preparation.test( - "mbranch.fastForwardUpdate", - "fast_forward carries a branch's UPDATE onto main: the row count stays at 3 and main " + - "reads the branch's updated value.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mub") - table.spark.sql( - s"UPDATE ${table.name}.branch_mub " + - s"SET ${Core.string0.columnName} = 'br-upd' " + - s"WHERE ${Core.long0.columnName} = 2") - table.spark.sql( - "CALL openhouse.system.fast_forward(" + - s"'${catalogRelative(table.name)}', 'main', 'mub')") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "fast-forward of an update changed the main row count") - assert( - table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 2") - .collect()(0) - .getString(0) == "br-upd", - "fast-forward did not carry the branch update") - }, - preparation.test( - "mbranch.cherrypickDelete", - "Cherry-picking a branch's position-delete DELETE snapshot onto main applies that " + - "delete to main, leaving 2 rows.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mcb") - table.spark.sql( - s"DELETE FROM ${table.name}.branch_mcb " + - s"WHERE ${Core.long0.columnName} = 1") - val deleteSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "ORDER BY committed_at DESC LIMIT 1") - .collect()(0) - .getLong(0) - - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', ${deleteSnapshotId}L)") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "2", - "cherry-pick should apply the branch delete to main") - }, - preparation.test( - "mbranch.replaceBranchDelete", - "REPLACE BRANCH AS OF a pre-delete snapshot undoes a branch's earlier position-delete " + - "DELETE, restoring the branch to 3 rows.") { table => - val seedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "ORDER BY committed_at DESC LIMIT 1") - .collect()(0) - .getLong(0) - - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH mrb") - table.spark.sql( - s"DELETE FROM ${table.name}.branch_mrb " + - s"WHERE ${Core.long0.columnName} = 1") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mrb'") == "2", - "branch delete was not applied") - - table.spark.sql( - s"ALTER TABLE ${table.name} REPLACE BRANCH mrb " + - s"AS OF VERSION $seedSnapshotId") - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mrb'") == "3", - "replacing the branch target did not undo its position delete") - }) - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchScenarioKit.scala deleted file mode 100644 index b34c1e647..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchScenarioKit.scala +++ /dev/null @@ -1,80 +0,0 @@ -package harness - -// The branch and write-audit-publish preparation kit. A branch preparation seeds main, creates -// branch b, and routes the session at that branch through spark.wap.branch, so every read and write -// the case performs lands on the branch while main keeps its seed rows. This layer sits above -// merge-on-read, so it also owns the branch-on-merge-on-read preparations. The members are lazy so -// they initialize on first read, after every trait mixed into `object Scenarios` has been -// constructed. -trait BranchScenarioKit extends MorScenarioKit { - - // Seed on main, create a branch, then set spark.wap.branch so every later read and write in the - // case lands on the branch. A case captures its own before state from the branch and asserts - // against it, so the same case body holds on a branch and on main. Each case runs in its own - // spark.newSession(), which keeps the setting scoped to that case. - def createAndSeedOnBranch(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = - createAndSeed(layout, numberOfRows) - .sql("prep.enableWap")(t => s"ALTER TABLE $t SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .step("prep.routeToBranch") { (spark, table) => - spark.sql(s"ALTER TABLE $table CREATE BRANCH b") - spark.conf.set("spark.wap.branch", "b") - }() - - private def assertBranchMainIsolation(table: PreparedTable[CoreTable.type]): Unit = { - table.spark.conf.unset("spark.wap.branch") - val mainCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - assert( - mainCount == 3, - s"branch operation leaked to main: expected 3 rows, got $mainCount") - } - - private def branchPreparationDescription(layout: Layout): String = - s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, with write.wap.enabled set, " + - "branch b created, and spark.wap.branch set to b, so every read and write in the case lands " + - "on branch b while main keeps its three seed rows." - - lazy val preparedBranchCoreTables: List[TablePreparation[CoreTable.type]] = - layouts.map { layout => - TablePreparation( - layout.label, - createAndSeedOnBranch(layout, 3), - "branchWap:", - assertBranchMainIsolation, - branchPreparationDescription(layout)) - } - - lazy val preparedPartitionedBranchCoreTables: List[TablePreparation[CoreTable.type]] = - partitionedLayouts.map { layout => - TablePreparation( - layout.label, - createAndSeedOnBranch(layout, 3), - "branchWap:", - assertBranchMainIsolation, - branchPreparationDescription(layout)) - } - - lazy val preparedBranchMorCoreTables: List[TablePreparation[CoreTable.type]] = - unpartitionedMorLayouts.map { layout => - TablePreparation( - layout.label, - createAndSeedOnBranch(layout, 3), - "branchWap:", - assertBranchMainIsolation, - branchPreparationDescription(layout)) - } - - lazy val preparedNullStringBranchCoreTables: List[TablePreparation[CoreTable.type]] = - preparedBranchCoreTables.map(withNullStringRow) - - lazy val preparedNullStringBranchMorCoreTables: List[TablePreparation[CoreTable.type]] = - preparedBranchMorCoreTables.map(withNullStringRow) - - lazy val branchLayoutFormatPreparations: List[TablePreparation[CoreTable.type]] = - preparedBranchCoreTables - - def branchLayoutFormatCases: List[Plan.Case] = - layoutFormatCasesFor(branchLayoutFormatPreparations) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchSurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchSurfaceScenarios.scala deleted file mode 100644 index 1e09109e7..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchSurfaceScenarios.scala +++ /dev/null @@ -1,422 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The branch and write-audit-publish surface families. Each case pins one edge of what branch -// routing exposes: what a branch keeps to itself, what it writes through to main, how a staged -// commit is published, and how maintenance behaves while a branch exists. The cases run on parquet -// and orc. -trait BranchSurfaceScenarios extends BranchScenarioKit { - import Rows._ - - // Each surface family builds the starting states it needs, so a family reads on its own. - private def surfaceBasePreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = s"Three seed rows with keys 1, 2 and 3 in an unpartitioned $format table.") - - private def surfaceTwoSnapshotPreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("insertMore")(table => - s"INSERT INTO $table VALUES " + - "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + - "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")(), - description = s"Five rows across two snapshots (a 3-row seed then a 2-row insert) in an " + - s"unpartitioned $format table.") - - private def surfaceWapPreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableWap")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")(), - description = s"Three seed rows in an unpartitioned $format table with " + - "write.wap.enabled=true.") - - // Compaction run against a table that carries a branch. - def surfaceBranchMaintenanceCases(format: String): List[Plan.Case] = - List( - surfaceTwoSnapshotPreparation(format).test( - "surface.maint.compactWithBranch", - "Compacting main while a branch exists preserves both main's and the branch's 6 rows; " + - "a follow-up compaction attempt routed at the branch via spark.wap.branch still leaves " + - "main and the branch at 6 rows, whichever way that routed attempt resolves.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH cb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_cb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - val compactionResult = table.spark - .sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('min-input-files', '2'))") - .collect()(0) - - println( - "DIAG compactWithBranch: " + - s"mainCompaction rewritten=${compactionResult.get(0)} " + - s"added=${compactionResult.get(1)}") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "6", - "main compaction should preserve 6 rows") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") == "6", - "main compaction should preserve the branch") - - table.spark.conf.set("spark.wap.branch", "cb") - val branchRoutedOutcome = - try { - val result = table.spark - .sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}')") - .collect()(0) - s"RAN (rewritten=${result.get(0)}, added=${result.get(1)})" - } catch { - case exception: Throwable => - s"THREW ${exception.getClass.getSimpleName} :: " + - Option(exception.getMessage).getOrElse("").take(140) - } finally { - table.spark.conf.unset("spark.wap.branch") - } - println(s"DIAG compactUnderWapConf: $branchRoutedOutcome") - - table.spark.sql(s"REFRESH TABLE ${table.name}") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "6", - "branch-routed compaction attempt should preserve main") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'cb'") == "6", - "branch-routed compaction attempt should preserve the branch") - }) - - // What a branch keeps to itself and what it leaks to main, plus the branch merge and retarget procedures. - def surfaceBranchCases(format: String): List[Plan.Case] = - List( - surfaceBasePreparation(format).test( - "branch.leak.setProps", - "SET TBLPROPERTIES issued while spark.wap.branch is set changes table-global metadata: " + - "the user property is visible on the table's own properties.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH lb2") - table.spark.conf.set("spark.wap.branch", "lb2") - try { - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('user.leaked'='yes')") - } finally { - table.spark.conf.unset("spark.wap.branch") - } - - assert( - tableProps(table.spark, table.name) - .get("user.leaked") - .contains("yes"), - "branch-routed property update should change table-global metadata") - }, - surfaceBasePreparation(format).test( - "branch.leak.writeOrderedBy", - "WRITE ORDERED BY issued while spark.wap.branch is set changes table-global metadata: " + - "write.distribution-mode becomes range on the table itself.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH lb3") - table.spark.conf.set("spark.wap.branch", "lb3") - try { - table.spark.sql( - s"ALTER TABLE ${table.name} " + - s"WRITE ORDERED BY ${Core.long0.columnName}") - } finally { - table.spark.conf.unset("spark.wap.branch") - } - - assert( - tableProps(table.spark, table.name) - .get("write.distribution-mode") - .contains("range"), - "branch-routed ordering should change table-global metadata") - }, - surfaceWapPreparation(format).test( - "branch.wapToggle.noGuard", - "A spark.wap.id-tagged insert produces exactly one staged snapshot carrying that " + - "wap.id, and that snapshot is unaffected by later disabling write.wap.enabled on the " + - "table.") { table => - table.spark.conf.set("spark.wap.id", "w9") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val stagedSnapshotCount = countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w9'") - assert( - stagedSnapshotCount == "1", - s"expected one staged snapshot, got $stagedSnapshotCount") - - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='false')") - val stagedAfterToggle = countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w9'") - - assert( - stagedAfterToggle == "1", - "disabling write.wap.enabled should not remove an already-staged snapshot, " + - s"got $stagedAfterToggle") - }, - surfaceWapPreparation(format).test( - "wap.neg.doubleCherrypick", - "Cherry-picking a WAP-staged snapshot publishes its row (row count goes from 3 to 4); " + - "cherry-picking that same snapshot a second time is rejected as a duplicate.") { table => - table.spark.conf.set("spark.wap.id", "w1") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val stagedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w1'") - .collect()(0) - .getLong(0) - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', ${stagedSnapshotId}L)") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "4", - "first cherry-pick should publish the staged row") - - val exception = Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', ${stagedSnapshotId}L)")) - println( - "DIAG doubleCherrypick: " + - s"${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(180)) - assert( - Option(exception.getMessage).exists(message => - message.toLowerCase.contains("duplicate") || - message.toLowerCase.contains("already")), - "second cherry-pick should reject the duplicate WAP commit") - }, - surfaceBasePreparation(format).test( - "wap.neg.expireRefTarget", - "Expiring the snapshot a branch currently points to is rejected with an exception, and " + - "the branch ref still points at its original snapshot afterward.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH eb2") - val branchHeadSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.refs " + - "WHERE name = 'eb2'") - .collect()(0) - .getLong(0) - Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - s"snapshot_ids => ARRAY(${branchHeadSnapshotId}L))")) - - val branchHeadSnapshotIdAfter = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.refs " + - "WHERE name = 'eb2'") - .collect()(0) - .getLong(0) - assert( - branchHeadSnapshotIdAfter == branchHeadSnapshotId, - "rejected expiration should leave the branch ref pointing at its original snapshot") - }, - surfaceBasePreparation(format).test( - "branch.fastForward.merge", - "fast_forward moves main to a branch's head after two branch-only inserts, growing " + - "main from 3 rows to 5.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH fb") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_fb VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_fb VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "branch writes should not advance main") - - table.spark.sql( - "CALL openhouse.system.fast_forward(" + - s"'${catalogRelative(table.name)}', 'main', 'fb')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "5", - "fast_forward should move main to the branch head") - }, - surfaceBasePreparation(format).test( - "branch.fastForward.divergent", - "fast_forward is rejected with an ancestry error when main and the branch have both " + - "advanced independently since they diverged.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH db") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_db VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - val exception = Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.fast_forward(" + - s"'${catalogRelative(table.name)}', 'main', 'db')")) - - println( - "DIAG ffDivergent: " + - s"${exception.getClass.getName} :: " + - Option(exception.getMessage).getOrElse("").take(180)) - assert( - Option(exception.getMessage).exists(message => - message.toLowerCase.contains("ancestor") || - message.toLowerCase.contains("fast-forward")), - "divergent fast_forward should report an ancestry error") - }, - surfaceTwoSnapshotPreparation(format).test( - "branch.replaceBranch", - "A new branch starts pointing at the current 5-row head; REPLACE BRANCH AS OF the " + - "earlier snapshot retargets it back to the 3-row seed state.") { table => - val snapshots = snapshotIds(table.spark, table.name) - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH rb2") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb2'") == "5", - "new branch should point at the current head") - - table.spark.sql( - s"ALTER TABLE ${table.name} REPLACE BRANCH rb2 " + - s"AS OF VERSION ${snapshots.head}") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'rb2'") == "3", - "REPLACE BRANCH should retarget the branch to the older snapshot") - }) - - // Publishing a write-audit-publish staged commit onto main. - def surfaceBranchPublishCases(format: String): List[Plan.Case] = - List( - surfaceWapPreparation(format).test( - "surface.proc.publishChanges", - "A WAP-staged insert stays invisible on main (still 3 rows) until publish_changes " + - "publishes it, growing main to 4 rows.") { table => - table.spark.conf.set("spark.wap.id", "pw1") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(9 AS BIGINT), 9, 'row-9', 9.5, true, '2024-01-09-01')") - } finally { - table.spark.conf.unset("spark.wap.id") - } - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "staged write should not be visible before publish") - - table.spark.sql( - "CALL openhouse.system.publish_changes(" + - s"table => '${catalogRelative(table.name)}', wap_id => 'pw1')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "4", - "publish_changes should publish the staged row") - }) - - // The DataFrame writer targeting a branch. - def surfaceBranchWriteCases(format: String): List[Plan.Case] = - List( - surfaceBasePreparation(format).test( - "surface.write.dfToBranch", - "A DataFrame writeTo(...).append() targeting a branch adds the row to that branch " + - "(4 rows) while leaving main unchanged at 3 rows.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH wb") - val row = table.spark.sql( - s"SELECT CAST(50 AS BIGINT) AS ${Core.long0.columnName}, " + - s"50 AS ${Core.int0.columnName}, " + - s"'row-50' AS ${Core.string0.columnName}, " + - s"50.5 AS ${Core.double0.columnName}, " + - s"true AS ${Core.boolean0.columnName}, " + - s"'2024-01-09-01' AS ${Core.datePartition.columnName}") - row.writeTo(s"${table.name}.branch_wb").append() - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'wb'") == "4", - "DataFrame writer should append to the branch") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "DataFrame branch write should leave main unchanged") - }) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala deleted file mode 100644 index 883557407..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/BranchWapScenarios.scala +++ /dev/null @@ -1,790 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -trait BranchWapScenarios extends BranchScenarioKit { - import Rows._ - - val wapStagedCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - val preparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableWap")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")(), - description = s"Three seed rows in a $format table with write.wap.enabled set to true.") - - List( - preparation.test( - "wapStaged.insert", - "A staged INSERT under spark.wap.id does not change main until its snapshot is " + - "cherry-picked, after which main includes the inserted row.") { table => - table.spark.conf.set("spark.wap.id", "wS") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES ${coreRow(99, "staged")}") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val stagedSnapshotCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'wS'") - .collect()(0) - .getLong(0) - - println( - "DIAG wapStaged.insert: " + - s"mainPreCount=$mainRowCount stagedSnapshots=$stagedSnapshotCount") - assert(mainRowCount == 3, "staged insert changed main before publish") - - val stagedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'wS'") - .collect()(0) - .getLong(0) - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', $stagedSnapshotId)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "publishing the staged insert did not advance main") - }, - preparation.test( - "wapStaged.overwrite", - "A staged INSERT OVERWRITE under spark.wap.id does not change main until its snapshot " + - "is cherry-picked, after which main is replaced by the overwritten rows.") { table => - table.spark.conf.set("spark.wap.id", "wS") - try { - table.spark.sql( - s"INSERT OVERWRITE ${table.name} VALUES ${coreRow(7, "ow")}") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val stagedSnapshotCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'wS'") - .collect()(0) - .getLong(0) - - println( - "DIAG wapStaged.overwrite: " + - s"mainPreCount=$mainRowCount stagedSnapshots=$stagedSnapshotCount") - assert(mainRowCount == 3, "staged overwrite changed main before publish") - - val stagedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'wS'") - .collect()(0) - .getLong(0) - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', $stagedSnapshotId)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 1, - "publishing the staged overwrite did not replace main") - }, - preparation.test( - "wapStaged.delete.bypassesWap", - "A DELETE issued under spark.wap.id commits directly to main with no staged snapshot, " + - "unlike INSERT, OVERWRITE, and MERGE.") { table => - table.spark.conf.set("spark.wap.id", "wD") - try { - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val stagedSnapshotCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'wD'") - .collect()(0) - .getLong(0) - - println( - "DIAG wapStaged.delete.bypassesWap: " + - s"mainAfterStagedDelete=$mainRowCount " + - s"stagedSnapshots=$stagedSnapshotCount") - assert( - mainRowCount == 2 && stagedSnapshotCount == 0, - "staged DELETE should commit directly to main without a WAP snapshot") - }, - preparation.test( - "wapStaged.merge", - "A staged MERGE INSERT under spark.wap.id does not change main until its snapshot is " + - "cherry-picked, after which main includes the merged row.") { table => - table.spark.conf.set("spark.wap.id", "wS") - try { - table.spark.sql( - s"MERGE INTO ${table.name} " + - "USING (SELECT CAST(99 AS BIGINT) AS key) source " + - s"ON ${table.name}.${Core.long0.columnName} = source.key " + - "WHEN NOT MATCHED THEN INSERT " + - s"(${Core.columnNames.mkString(", ")}) " + - "VALUES (source.key, 9, 'm', 9.5, true, '2024-01-09-01')") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val stagedSnapshotCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'wS'") - .collect()(0) - .getLong(0) - - println( - "DIAG wapStaged.merge: " + - s"mainPreCount=$mainRowCount stagedSnapshots=$stagedSnapshotCount") - assert(mainRowCount == 3, "staged merge changed main before publish") - - val stagedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'wS'") - .collect()(0) - .getLong(0) - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', $stagedSnapshotId)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "publishing the staged merge did not advance main") - }, - preparation.test( - "wapStaged.update.valueVisibleOnlyAfterPublish", - "A staged UPDATE under spark.wap.id leaves the old value visible on main until its " + - "snapshot is cherry-picked, after which main reads the updated value.") { table => - table.spark.conf.set("spark.wap.id", "wU") - try { - table.spark.sql( - s"UPDATE ${table.name} " + - s"SET ${Core.string0.columnName} = 'staged-upd' " + - s"WHERE ${Core.long0.columnName} = 1") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val valueBeforePublish = table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - .collect()(0) - .getString(0) - - assert( - valueBeforePublish != "staged-upd", - s"staged update changed main before publish: $valueBeforePublish") - - val stagedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'wU'") - .collect()(0) - .getLong(0) - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', $stagedSnapshotId)") - val valueAfterPublish = table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - .collect()(0) - .getString(0) - - assert( - valueAfterPublish == "staged-upd", - s"published update returned $valueAfterPublish") - }, - preparation.test( - "wapStaged.twoIdsIndependent", - "Two inserts staged under different spark.wap.id values publish independently: " + - "cherry-picking one advances main without exposing the other's row until it too is " + - "cherry-picked.") { table => - def stageInsert(wapId: String, key: Int): Unit = { - table.spark.conf.set("spark.wap.id", wapId) - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - coreRow(key, s"s-$wapId")) - } finally { - table.spark.conf.unset("spark.wap.id") - } - } - def snapshotId(wapId: String): Long = - table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - s"WHERE summary['wap.id'] = '$wapId'") - .collect()(0) - .getLong(0) - - stageInsert("wa", 101) - stageInsert("wb", 102) - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 3, - "a staged ID changed main before publish") - - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', ${snapshotId("wa")})") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "publishing wa did not advance main") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 102") - .collect()(0) - .getLong(0) == 0, - "wb published before its cherry-pick") - - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', ${snapshotId("wb")})") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 5, - "publishing wb did not advance main") - }, - preparation.test( - "wapStaged.expireVsStaged", - "Expiring snapshots with retain_last=1 removes an unreferenced staged WAP snapshot, and " + - "cherry-picking it afterward fails because the snapshot is gone.") { table => - table.spark.conf.set("spark.wap.id", "wE") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES ${coreRow(200, "stg")}") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val stagedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'wE'") - .collect()(0) - .getLong(0) - - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - val survivedExpiration = table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.snapshots " + - s"WHERE snapshot_id = $stagedSnapshotId") - .collect()(0) - .getLong(0) - val publishOutcome = - try { - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', $stagedSnapshotId)") - "published" - } catch { - case NonFatal(exception) => - s"stranded:${Exceptions.root(exception).getClass.getSimpleName}" - } - - println( - "DIAG wapStaged.expireVsStaged: " + - s"stagedSurvivedExpire=$survivedExpiration " + - s"cherrypickAfterExpire=$publishOutcome") - assert( - survivedExpiration == 0 && publishOutcome.startsWith("stranded"), - "expiration should remove and strand the unreferenced staged snapshot") - }) - } - - val branchDdlCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - val preparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableWap")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.wap.enabled'='true')")() - .sql("createBranch")(table => - s"ALTER TABLE $table CREATE BRANCH bddl")(), - description = s"Three seed rows in a $format table with write.wap.enabled set to true and " + - "branch bddl created.") - - List( - preparation.test( - "branchDdl.addColumn.leaksToMain", - "ALTER TABLE ADD COLUMN issued while spark.wap.branch selects a branch is accepted and " + - "adds the column to the table's global schema, visible on main.") { table => - table.spark.conf.set("spark.wap.branch", "bddl") - val outcome = - try { - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN br_added int") - "accepted" - } catch { - case NonFatal(exception) => - s"rejected:${Exceptions.root(exception).getClass.getSimpleName}" - } finally { - table.spark.conf.unset("spark.wap.branch") - } - val columnNames = table.spark - .sql(s"DESCRIBE TABLE ${table.name}") - .collect() - .map(_.getString(0).trim) - .toSet - - println( - "DIAG branchDdl.addColumn.leaksToMain: " + - s"branch-routed DDL $outcome") - assert( - columnNames.contains("br_added"), - "ADD COLUMN on a branch should change the table-global schema") - }, - preparation.test( - "branchDdl.setTblProp.leaksToMain", - "ALTER TABLE SET TBLPROPERTIES issued while spark.wap.branch selects a branch is accepted " + - "and changes the table's global properties, visible on main.") { table => - table.spark.conf.set("spark.wap.branch", "bddl") - val outcome = - try { - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('user.branchkey'='v1')") - "accepted" - } catch { - case NonFatal(exception) => - s"rejected:${Exceptions.root(exception).getClass.getSimpleName}" - } finally { - table.spark.conf.unset("spark.wap.branch") - } - val properties = table.spark - .sql(s"SHOW TBLPROPERTIES ${table.name}") - .collect() - .map(row => row.getString(0) -> row.getString(1)) - .toMap - - println( - "DIAG branchDdl.setTblProp.leaksToMain: " + - s"branch-routed DDL $outcome") - assert( - properties.get("user.branchkey").contains("v1"), - "SET TBLPROPERTIES on a branch should change table-global properties") - }, - preparation.test( - "branchDdl.alterColumnComment.leaksToMain", - "ALTER TABLE ALTER COLUMN COMMENT issued while spark.wap.branch selects a branch is " + - "accepted and changes the table's global column comment, visible on main.") { table => - table.spark.conf.set("spark.wap.branch", "bddl") - val outcome = - try { - table.spark.sql( - s"ALTER TABLE ${table.name} " + - s"ALTER COLUMN ${Core.string0.columnName} COMMENT 'br-comment'") - "accepted" - } catch { - case NonFatal(exception) => - s"rejected:${Exceptions.root(exception).getClass.getSimpleName}" - } finally { - table.spark.conf.unset("spark.wap.branch") - } - val comment = table.spark - .sql(s"DESCRIBE TABLE ${table.name}") - .collect() - .find(_.getString(0).trim == Core.string0.columnName) - .map(_.getString(2)) - .getOrElse("") - - println( - "DIAG branchDdl.alterColumnComment.leaksToMain: " + - s"branch-routed DDL $outcome") - assert( - Option(comment).getOrElse("").contains("br-comment"), - "ALTER COLUMN COMMENT on a branch should change table-global metadata") - }, - preparation.test( - "branchDdl.dropColumn.rejected", - "ALTER TABLE DROP COLUMN issued while spark.wap.branch selects a branch is rejected, and " + - "the column remains present.") { table => - table.spark.conf.set("spark.wap.branch", "bddl") - val outcome = - try { - table.spark.sql( - s"ALTER TABLE ${table.name} " + - s"DROP COLUMN ${Core.string0.columnName}") - "accepted" - } catch { - case NonFatal(exception) => - s"rejected:${Exceptions.root(exception).getClass.getSimpleName}" - } finally { - table.spark.conf.unset("spark.wap.branch") - } - val columnNames = table.spark - .sql(s"DESCRIBE TABLE ${table.name}") - .collect() - .map(_.getString(0).trim) - .toSet - - println( - "DIAG branchDdl.dropColumn.rejected: " + - s"branch-routed DDL $outcome") - assert( - outcome.startsWith("rejected:"), - s"DROP COLUMN should be rejected while a branch is selected: $outcome") - assert( - columnNames.contains(Core.string0.columnName), - "DROP COLUMN should remain rejected while a branch is selected") - }) - } - - val branchingCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - val preparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = s"Three seed rows in a $format table with no branches or WAP configuration.") - - List( - preparation.test( - "branch.direct.isolation", - "Inserting directly into a created branch adds a row visible only when reading that " + - "branch, and main keeps its original 3 rows.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH b") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_b VALUES " + - coreRow(99, "branch")) - val branchRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'b'") - .collect()(0) - .getLong(0) - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - branchRowCount == 4, - s"branch b should have 4 rows, got $branchRowCount") - assert( - mainRowCount == 3, - s"main should be unchanged at 3 rows, got $mainRowCount") - }, - preparation.test( - "branch.wapConf.routing", - "With write.wap.enabled set and spark.wap.branch selecting a branch, an INSERT and the " + - "following read both route to that branch, leaving main at its original 3 rows.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH wapbr") - table.spark.conf.set("spark.wap.branch", "wapbr") - val branchRowCount = - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES ${coreRow(99, "wap")}") - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - } finally { - table.spark.conf.unset("spark.wap.branch") - } - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - branchRowCount == 4, - s"branch-routed read should see 4 rows, got $branchRowCount") - assert( - mainRowCount == 3, - s"branch-routed write changed main to $mainRowCount rows") - }, - preparation.test( - "wap.stagePublish", - "A staged INSERT under spark.wap.id leaves main at its original 3 rows until its " + - "snapshot is cherry-picked, after which main includes the inserted row.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.conf.set("spark.wap.id", "w1") - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES ${coreRow(99, "staged")}") - } finally { - table.spark.conf.unset("spark.wap.id") - } - val mainBeforePublish = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - assert( - mainBeforePublish == 3, - s"staged write changed main to $mainBeforePublish rows") - - val stagedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "WHERE summary['wap.id'] = 'w1'") - .collect()(0) - .getLong(0) - table.spark.sql( - "CALL openhouse.system.cherrypick_snapshot(" + - s"'${catalogRelative(table.name)}', $stagedSnapshotId)") - val mainAfterPublish = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - mainAfterPublish == 4, - s"publishing the staged write left main at $mainAfterPublish rows") - }, - preparation.test( - "branch.ddlLeak.addColumn", - "ALTER TABLE ADD COLUMN issued while spark.wap.branch selects a branch changes the " + - "table's global schema, visible on main.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH leakbr") - table.spark.conf.set("spark.wap.branch", "leakbr") - try { - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN leaked_col int") - } finally { - table.spark.conf.unset("spark.wap.branch") - } - val mainColumnNames = - table.spark.table(table.name).schema.fields.map(_.name).toSeq - - assert( - mainColumnNames.contains("leaked_col"), - "ADD COLUMN on a branch should change the table-global schema") - }, - preparation.test( - "branch.dml.updateDelete", - "UPDATE and DELETE issued while spark.wap.branch selects a branch change only that " + - "branch's rows and leave main at its original 3 rows.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH dmlbr") - table.spark.conf.set("spark.wap.branch", "dmlbr") - try { - table.spark.sql( - s"UPDATE ${table.name} " + - s"SET ${Core.string0.columnName} = 'br-upd' " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - s"DELETE FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 2") - } finally { - table.spark.conf.unset("spark.wap.branch") - } - val branchRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'dmlbr'") - .collect()(0) - .getLong(0) - val mainRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val branchValue = table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - "VERSION AS OF 'dmlbr' " + - s"WHERE ${Core.long0.columnName} = 1") - .collect()(0) - .getString(0) - - assert( - branchRowCount == 2, - s"branch should have 2 rows after delete, got $branchRowCount") - assert( - mainRowCount == 3, - s"branch DML changed main to $mainRowCount rows") - assert( - branchValue == "br-upd", - s"branch update returned $branchValue") - }, - preparation.test( - "branch.lifecycle.tag", - "A tag pins its snapshot through a later insert and snapshot expiration: the tag still " + - "reads 3 rows, main reads 4 rows including the new one, and the tagged snapshot is not " + - "expired.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE TAG mytag") - val taggedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.refs " + - "WHERE name = 'mytag' AND type = 'TAG'") - .collect()(0) - .getLong(0) - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'mytag'") - .collect()(0) - .getLong(0) == 3, - "the tag should read the snapshot captured before the insert") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "the main branch should include the inserted row") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.snapshots " + - s"WHERE snapshot_id = $taggedSnapshotId") - .collect()(0) - .getLong(0) == 1, - "snapshot expiration should retain the snapshot referenced by the tag") - }, - preparation.test( - "branch.lifecycle.dropBranch", - "CREATE BRANCH adds a ref that DROP BRANCH then removes.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH tmpbr") - val branchCountBeforeDrop = table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.refs " + - "WHERE name = 'tmpbr'") - .collect()(0) - .getLong(0) - assert( - branchCountBeforeDrop == 1, - "CREATE BRANCH did not create the branch ref") - - table.spark.sql( - s"ALTER TABLE ${table.name} DROP BRANCH tmpbr") - val branchCountAfterDrop = table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.refs " + - "WHERE name = 'tmpbr'") - .collect()(0) - .getLong(0) - - assert( - branchCountAfterDrop == 0, - "DROP BRANCH did not remove the branch ref") - }, - preparation.test( - "branch.neg.wapIdAndBranch", - "Setting both spark.wap.id and spark.wap.branch on a write is rejected with a validation " + - "error naming the conflict.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.wap.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH nb") - table.spark.conf.set("spark.wap.id", "w1") - table.spark.conf.set("spark.wap.branch", "nb") - try { - val exception = Check.intercept[ValidationException]( - table.spark.sql( - s"INSERT INTO ${table.name} VALUES ${coreRow(99, "x")}")) - assert( - exception.getMessage.contains("Cannot set both WAP ID and branch"), - s"unexpected validation message: ${exception.getMessage.take(140)}") - } finally { - table.spark.conf.unset("spark.wap.id") - table.spark.conf.unset("spark.wap.branch") - } - }, - preparation.test( - "branch.neg.insertNonexistentBranch", - "Inserting into a branch name that was never created is rejected with a validation error " + - "saying the branch does not exist.") { table => - val exception = Check.intercept[ValidationException]( - table.spark.sql( - s"INSERT INTO ${table.name}.branch_nope VALUES " + - coreRow(99, "x"))) - - assert( - exception.getMessage.contains("does not exist"), - s"unexpected validation message: ${exception.getMessage.take(140)}") - }) - } - - - - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorDmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorDmlScenarios.scala deleted file mode 100644 index f9cd4e96f..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorDmlScenarios.scala +++ /dev/null @@ -1,107 +0,0 @@ -package harness - -// The merge-on-read DML buckets. The mutation buckets are merge-on-read preparation lists crossed -// with the shared DML test-case lists that DmlScenarios names, so a merge-on-read table runs the -// same row-delta assertions as a copy-on-write one. The delete-file-mode bucket is the exception: -// it asserts the physical difference between the two write modes directly. -trait MorDmlScenarios extends MorScenarioKit { this: DmlScenarios => - import Rows._ - - lazy val morDmlCases: List[Plan.Case] = - preparedMorCoreTables.flatMap(preparation => rowMutationTestCases.map(_.runOn(preparation))) ++ - preparedNullStringMorCoreTables.flatMap(preparation => - nullStringRowTestCases.map(_.runOn(preparation))) - - lazy val rtasMorDmlCases: List[Plan.Case] = - preparedRtasMorCoreTables.flatMap(preparation => - rowMutationTestCases.map(_.runOn(preparation))) ++ - preparedNullStringRtasMorCoreTables.flatMap(preparation => - nullStringRowTestCases.map(_.runOn(preparation))) - - lazy val morReadDmlCases: List[Plan.Case] = - preparedMorReadCoreTables.flatMap(preparation => readTestCases.map(_.runOn(preparation))) - - // --- merge-on-read versus copy-on-write: prove the physical difference --- - // The rest of the merge-on-read preparations reuse the row-delta assertions, which hold - // identically whether the write was copy-on-write or merge-on-read. These two pin the physical - // difference: a merge-on-read delete adds a position-delete file, a copy-on-write delete rewrites - // the data file and adds none. Both are prepared with a single seed data file and delete a strict - // subset (one of three rows), so the write is a partial-file match and the outcome is - // deterministic across formats. - - private lazy val preparedSingleFileMorTables: List[TablePreparation[CoreTable.type]] = - morVerifyLayouts.map(layout => - TablePreparation( - layout.label, - createAndSeedSingleFile(layout, 3), - description = s"Three seed rows with keys 1, 2 and 3 written as one data file in " + - s"${layout.description}.")) - - private lazy val preparedSingleFileCowTables: List[TablePreparation[CoreTable.type]] = - cowVerifyLayouts.map(layout => - TablePreparation( - layout.label, - createAndSeedSingleFile(layout, 3), - description = s"Three seed rows with keys 1, 2 and 3 written as one data file in " + - s"${layout.description}.")) - - private lazy val morWritesDeleteFiles: DmlTestCase[CoreTable.type] = - DmlTestCase( - "mor.writesDeleteFiles", - s"A merge-on-read DELETE WHERE ${Core.long0.columnName} < 2 against a single data file removes " + - "the matching row, records the removal in at least one position-delete file, and commits one " + - "snapshot.", - table => { - val before = table.state - - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") - val after = table.state - val deleteFileCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.delete_files") - .collect()(0) - .getLong(0) - - assert( - after.rows == before.rows.filterNot(_.get(Core.long0) < 2), - s"strict-subset DELETE returned an unexpected row set: ${after.rows}") - assert( - deleteFileCount >= 1, - "merge-on-read DELETE should write a position-delete file") - assert( - after.snapshotCount == before.snapshotCount + 1, - "a merge-on-read DELETE commits one snapshot") - }) - - private lazy val cowWritesNoDeleteFiles: DmlTestCase[CoreTable.type] = - DmlTestCase( - "cow.writesNoDeleteFiles", - s"A copy-on-write DELETE WHERE ${Core.long0.columnName} < 2 against a single data file removes " + - "the matching row by rewriting that file, leaves the table with no delete files, and commits " + - "one snapshot.", - table => { - val before = table.state - - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") - val after = table.state - val deleteFileCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.delete_files") - .collect()(0) - .getLong(0) - - assert( - after.rows == before.rows.filterNot(_.get(Core.long0) < 2), - s"strict-subset DELETE returned an unexpected row set: ${after.rows}") - assert( - deleteFileCount == 0, - "copy-on-write DELETE should not write delete files") - assert( - after.snapshotCount == before.snapshotCount + 1, - "a copy-on-write DELETE commits one snapshot") - }) - - lazy val deleteFileModeCases: List[Plan.Case] = - preparedSingleFileMorTables.map(morWritesDeleteFiles.runOn) ++ - preparedSingleFileCowTables.map(cowWritesNoDeleteFiles.runOn) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorForkScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorForkScenarios.scala deleted file mode 100644 index f3258e62f..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorForkScenarios.scala +++ /dev/null @@ -1,72 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The fork behavior that only shows up on a merge-on-read delete. The delete-file replication factor -// is stamped onto the position-delete file this DELETE writes, so the case needs the merge-on-read -// write path. -trait MorForkScenarios extends MorScenarioKit { - import Rows._ - - private def showProps(spark: SparkSession, table: String): Map[String, String] = - spark.sql(s"SHOW TBLPROPERTIES $table").collect().toSeq.map(r => r.getString(0) -> r.getString(1)).toMap - - // Delete-file replication factor for merge-on-read deletes. - // The write.delete-file-replication table property is resolved into a replication factor that the - // delete-file write path stamps onto the position-delete file's output properties, which is what tells - // HDFS to set that file's block replication. The actual HDFS block replication is not observable on the - // local filesystem this harness runs on, so this test asserts the parts that are locally observable: - // the property round-trips through the catalog metadata, a merge-on-read DELETE physically writes a - // position-delete file, and the DML result and the property both survive the mutation. - private def forkDeleteFileReplication(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_delrepl" - spark.sql(s"DROP TABLE IF EXISTS $table") - // Merge-on-read, unpartitioned, distribution none, so one seed INSERT lands one data file; a partial - // DELETE against that file must then be satisfied with a position-delete file. - spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES (" + - s"'format-version'='2', 'write.distribution-mode'='none', 'write.delete.mode'='merge-on-read', " + - s"'write.update.mode'='merge-on-read', 'write.delete-file-replication'='2')") - // COALESCE(1) produces a single data file. Deleting a strict subset records a position-delete - // file while preserving the untouched rows in that data file. - spark.sql(s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM (VALUES (1L,'a'),(2L,'b'),(3L,'c')) AS s(id, s)") - - // (1) The property round-trips through the catalog metadata. - val p1 = showProps(spark, table) - assert(p1.get("write.delete-file-replication").contains("2"), - s"expected write.delete-file-replication=2 to round-trip, got ${p1.get("write.delete-file-replication")}") - - // (2) A merge-on-read DELETE writes a position-delete file. - spark.sql(s"DELETE FROM $table WHERE id = 1") - val delFiles = spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) - assert(delFiles >= 1, s"merge-on-read DELETE should write a position-delete file, got $delFiles") - - // (3) The DML result is correct; the replication factor never alters the logical row set. - val rows = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) - assert(rows == Seq(2L, 3L), s"expected [2,3] after the merge-on-read delete, got $rows") - - // (4) The property survives the mutation. - val p2 = showProps(spark, table) - assert(p2.get("write.delete-file-replication").contains("2"), "write.delete-file-replication lost after DELETE") - - println(s"fork.deleteFileReplication: prop=2 roundtrips=yes deleteFiles=$delFiles rows=${rows.mkString(",")}") - spark.sql(s"DROP TABLE IF EXISTS $table") - } - - val forkDeleteFileReplicationCases: List[Plan.Case] = - List( - Plan.Case( - "fork.deleteFileReplication @ mor", - forkDeleteFileReplication, - description = "The write.delete-file-replication table property round-trips through the " + - "catalog, a merge-on-read DELETE writes a position-delete file, the surviving rows are " + - "correct, and the property is still set after the delete.")) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorInteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorInteractionScenarios.scala deleted file mode 100644 index aaa746ecd..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorInteractionScenarios.scala +++ /dev/null @@ -1,59 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The merge-on-read interaction family. A table created with the default copy-on-write delete mode -// is switched to merge-on-read partway through its life, so the case composes the mode change with -// the mutations that run after it. -trait MorInteractionScenarios extends MorScenarioKit { - import Rows._ - - def interactionMorCases(format: String): List[Plan.Case] = { - val oneFilePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .sql("seed")(table => - s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM " + - s"(${RowGenerator.valuesClause(Core, 3)}) AS seed")(), - description = s"Three seed rows written as one data file in a $format table.") - - List( - oneFilePreparation.test( - "interact.mor.alterToMor", - "Switching a table's delete mode to merge-on-read partway through its life makes a " + - "subsequent partial-file DELETE write a position-delete file while preserving the " + - "untouched rows in the data file.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('write.delete.mode'='merge-on-read')") - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val deleteFileCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.all_delete_files") - .collect()(0) - .getLong(0) - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - deleteFileCount == 1, - s"ALTER-to-MoR should create one delete file, got $deleteFileCount") - assert( - rowCount == 2, - s"ALTER-to-MoR delete should leave 2 rows, got $rowCount") - }) - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala deleted file mode 100644 index 9e6c17f15..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorMaintScenarios.scala +++ /dev/null @@ -1,480 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The merge-on-read coexistence, maintenance, metadata and hazard families. Each case operates -// on a table that already carries a live position-delete file, so it exercises the surface where -// data files and delete files coexist. -trait MorMaintScenarios extends MorScenarioKit { - import Rows._ - - // Merge-on-read delete-file coexistence. - // A read or insert on a delete-free merge-on-read table is byte-identical to copy-on-write, - // since there are no delete files to apply and an append is mode-independent. The cases below - // instead operate on a table that already carries a live position-delete file, so they exercise - // the genuinely MoR-specific surface: data-file and delete-file coexistence. - // `createAndSeedMorDeleted` leaves 2 rows (keys 2 and 3) with a live delete for key 1; these - // cases then act on that state. - lazy val morCoexistCases: List[Plan.Case] = - morVerifyLayouts - .map(layout => - TablePreparation( - layout.label, - createAndSeedMorDeleted(layout, 3), - description = s"Two live rows with keys 2 and 3 in ${layout.description}, with a live " + - "position-delete file removing key 1.")) - .flatMap { preparation => - List( - preparation.test( - "coexist.append", - "INSERT INTO over a table with a live position-delete file adds the new row without " + - "resurrecting the deleted one.") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 3, - "append over a live delete file returned the wrong row count") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - .collect()(0) - .getLong(0) == 0, - "append resurrected the deleted row") - }, - preparation.test( - "coexist.secondDelete", - "A second DELETE on a table that already has a live position-delete file removes the " + - "targeted row and leaves delete files present.") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 1, - "second delete returned the wrong row count") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}.all_delete_files") - .collect()(0) - .getLong(0) >= 1, - "delete files are missing after the second delete") - }, - preparation.test( - "coexist.update", - "UPDATE on a table with a live position-delete file changes the targeted row's value " + - "without changing the row count.") { table => - table.spark.sql( - s"UPDATE ${table.name} " + - s"SET ${Core.string0.columnName} = 'cx' " + - s"WHERE ${Core.long0.columnName} = 3") - - assert( - table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 3") - .collect()(0) - .getString(0) == "cx", - "update over a live delete file failed") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "update over a live delete file changed the row count") - }, - preparation.test( - "coexist.readFilter", - "A filtered read over a table with a live position-delete file does not return the " + - "deleted row.") { table => - val keys = table.spark - .sql( - s"SELECT ${Core.long0.columnName} FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2 " + - s"ORDER BY ${Core.long0.columnName}") - .collect() - .toSeq - .map(_.getLong(0)) - - assert( - keys == Seq(2L), - s"filter did not apply the position delete: $keys") - }, - preparation.test( - "coexist.compactDeletes", - "rewrite_position_delete_files compacts the live position-delete file while preserving " + - "the 2 live rows.") { table => - table.spark.sql( - "CALL openhouse.system.rewrite_position_delete_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('rewrite-all', 'true'))") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "position-delete compaction changed the row set") - }, - preparation.test( - "coexist.merge", - "MERGE INTO on a table with a live position-delete file updates the matched row " + - "without changing the row count.") { table => - table.spark.sql( - s"MERGE INTO ${table.name} target " + - "USING (SELECT CAST(3 AS BIGINT) key) source " + - s"ON target.${Core.long0.columnName} = source.key " + - "WHEN MATCHED THEN UPDATE " + - s"SET ${Core.string0.columnName} = 'mg'") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "merge over a live delete file changed the row count") - assert( - table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 3") - .collect()(0) - .getString(0) == "mg", - "merge over a live delete file failed") - }) - } - - // Maintenance on a merge-on-read table that carries a live position-delete file. - // `createAndSeedMorDeleted` leaves keys 2 and 3 live with a live delete for key 1. These cases - // check whether each maintenance procedure handles the delete file correctly: folding it away, - // preserving it, or leaving the deleted row gone. - - // rewrite_data_files applies the live delete to the rewritten data (key 1 is physically gone and - // the row set is correct), but it does not remove the now-dangling position-delete reference from - // the current snapshot. The compacted table keeps a live delete-file reference that points at - // data already removed until rewrite_position_delete_files or expire_snapshots runs; reads stay - // correct throughout. This is exercised across all 3 MoR formats to confirm the behavior is - // format-consistent, since the delete decode differs per format. - lazy val maintenanceMorFoldCases: List[Plan.Case] = - morVerifyLayouts - .map(layout => - TablePreparation( - layout.label, - createAndSeedMorDeleted(layout, 3), - description = s"Two live rows with keys 2 and 3 in ${layout.description}, with a live " + - "position-delete file removing key 1.")) - .flatMap { preparation => - List( - preparation.test( - "maint.mor.rewriteDataFilesDanglingDelete", - "rewrite_data_files applies the live delete into the compacted data (key 1 stays gone, " + - "2 rows read back correctly) but leaves the now-dangling position-delete file in " + - "place.") { table => - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('rewrite-all', 'true'))") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "rewrite_data_files changed the live row set") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - .collect()(0) - .getLong(0) == 0, - "rewrite_data_files resurrected the deleted row") - - val deleteFileCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.delete_files") - .collect()(0) - .getLong(0) - val keys = table.spark - .sql( - s"SELECT ${Core.long0.columnName} FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2 " + - s"ORDER BY ${Core.long0.columnName}") - .collect() - .toSeq - .map(_.getLong(0)) - - assert( - deleteFileCount == 1, - "rewrite_data_files should leave one dangling position delete, " + - s"got $deleteFileCount") - assert( - keys == Seq(2L), - s"read after rewrite_data_files returned incorrect keys: $keys") - }, - preparation.test( - "maint.mor.rewritePositionDeleteFolds", - "After rewrite_data_files leaves a dangling position delete, rewrite_position_delete_files " + - "folds it away (the delete-file count drops to zero) while the live row set stays " + - "correct.") { table => - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('rewrite-all', 'true'))") - val deleteFilesBefore = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.delete_files") - .collect()(0) - .getLong(0) - - table.spark.sql( - "CALL openhouse.system.rewrite_position_delete_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('rewrite-all', 'true'))") - val deleteFilesAfter = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.delete_files") - .collect()(0) - .getLong(0) - - println( - "DIAG maint.mor.rewritePositionDeleteFolds: " + - s"delete_files before=$deleteFilesBefore after=$deleteFilesAfter") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "rewrite_position_delete_files changed the live row set") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - .collect()(0) - .getLong(0) == 0, - "rewrite_position_delete_files resurrected the deleted row") - assert( - deleteFilesBefore == 1 && deleteFilesAfter == 0, - "rewrite_position_delete_files should fold the dangling delete: " + - s"before=$deleteFilesBefore after=$deleteFilesAfter") - }) - } - - // Metadata-only maintenance over a live delete does not decode the delete file, so its behavior - // does not vary by format; this runs against a single MoR layout. Each case must preserve the - // delete (2 live rows, key 1 still gone). - lazy val maintenanceMorMetaCases: List[Plan.Case] = - morVerifyLayouts - .filter(layout => - layout.label == "mor-verify/parquet" || - layout.label == "mor-verify/orc") - .map(layout => - TablePreparation( - layout.label, - createAndSeedMorDeleted(layout, 3), - description = s"Two live rows with keys 2 and 3 in ${layout.description}, with a live " + - "position-delete file removing key 1.")) - .flatMap { preparation => - List( - preparation.test( - "maint.mor.expireSnapshots", - "expire_snapshots over a table with a live position-delete file leaves the 2 live " + - "rows unchanged and does not resurrect the deleted row.") { table => - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "expire_snapshots changed the live row set") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - .collect()(0) - .getLong(0) == 0, - "expire_snapshots resurrected the deleted row") - }, - preparation.test( - "maint.mor.rewriteManifests", - "rewrite_manifests over a table with a live position-delete file leaves the 2 live " + - "rows unchanged.") { table => - table.spark.sql( - "CALL openhouse.system.rewrite_manifests(" + - s"table => '${catalogRelative(table.name)}', " + - "use_caching => false)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "rewrite_manifests changed the live row set") - }, - preparation.test( - "maint.mor.removeOrphanFiles", - "remove_orphan_files over a table with a live position-delete file leaves the 2 live " + - "rows unchanged.") { table => - table.spark.sql( - "CALL openhouse.system.remove_orphan_files(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2020-01-01 00:00:00')") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "remove_orphan_files changed the live row set") - }, - preparation.test( - "maint.mor.compactThenExpire", - "Running rewrite_position_delete_files followed by expire_snapshots leaves the 2 live " + - "rows unchanged and does not resurrect the deleted row.") { table => - table.spark.sql( - "CALL openhouse.system.rewrite_position_delete_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('rewrite-all', 'true'))") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "compact-then-expire changed the live row set") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - .collect()(0) - .getLong(0) == 0, - "compact-then-expire resurrected the deleted row") - }) - } - - // A live position delete is snapshot-scoped state. These cases check that it is resolved - // correctly across history and restore: a delete must not be retroactive (pre-delete snapshots - // still show the row), rollback must undo it, and it must survive expiration of older snapshots. - // Time travel and rollback select snapshots. One MoR layout covers this format-independent - // behavior. - lazy val morHazardCases: List[Plan.Case] = - morVerifyLayouts - .filter(layout => - layout.label == "mor-verify/parquet" || - layout.label == "mor-verify/orc") - .map(layout => - TablePreparation( - layout.label, - createAndSeedMorDeleted(layout, 3), - description = s"Two live rows with keys 2 and 3 in ${layout.description}, with a live " + - "position-delete file removing key 1.")) - .flatMap { preparation => - List( - preparation.test( - "hazard.mor.timeTravelBeforeDelete", - "The current read applies the live position delete (2 rows), while VERSION AS OF the " + - "snapshot before the delete still shows the deleted row.") { table => - val seedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "ORDER BY committed_at LIMIT 1") - .collect()(0) - .getLong(0) - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "current merge-on-read state should apply the delete") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF $seedSnapshotId") - .collect()(0) - .getLong(0) == 3, - "the snapshot before the delete should still contain the row") - }, - preparation.test( - "hazard.mor.rollbackUndoesDelete", - "rollback_to_snapshot to before the position delete restores the deleted row and the " + - "full 3-row set.") { table => - val seedSnapshotId = table.spark - .sql( - s"SELECT snapshot_id FROM ${table.name}.snapshots " + - "ORDER BY committed_at LIMIT 1") - .collect()(0) - .getLong(0) - - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"table => '${catalogRelative(table.name)}', " + - s"snapshot_id => ${seedSnapshotId}L)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 3, - "rollback did not undo the merge-on-read delete") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - .collect()(0) - .getLong(0) == 1, - "rollback did not restore the deleted row") - }, - preparation.test( - "hazard.mor.expireThenDeleteHolds", - "After snapshot expiration, a read still excludes the position-deleted row.") { table => - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - val keys = table.spark - .sql( - s"SELECT ${Core.long0.columnName} FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2 " + - s"ORDER BY ${Core.long0.columnName}") - .collect() - .toSeq - .map(_.getLong(0)) - - assert( - keys == Seq(2L), - s"delete did not survive snapshot expiration: $keys") - }) - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorReaderWriterScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorReaderWriterScenarios.scala deleted file mode 100644 index 6d3def686..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorReaderWriterScenarios.scala +++ /dev/null @@ -1,194 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The merge-on-read reader and writer families. Each case is the merge-on-read counterpart of a -// copy-on-write changelog case: the same operation runs on a format-version 2 table whose delete, -// update and merge modes are merge-on-read, so the changelog it produces is read from the position -// delete files that mutation wrote. The cases run on parquet and orc. -trait MorReaderWriterScenarios extends MorScenarioKit { - import Rows._ - - private def morCreate(t: String, fmt: String): String = - s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES (${morPropsFmt(fmt)})" - - private def morPreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => morCreate(table, format))() - .insert(3)(), - description = s"Three seed rows in a merge-on-read $format table.") - - // The changelog view over an append on a merge-on-read table. - def morReaderWriterChangelogAppendCases(format: String): List[Plan.Case] = - List( - morPreparation(format).test( - "readerWriter.changelog.append.mor", - "On a merge-on-read table, a changelog view over an appended row reports exactly one " + - "INSERT and no DELETE.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.append.mor: $changeTypes") - assert( - changeTypes.getOrElse("INSERT", 0L) == 1 && - !changeTypes.contains("DELETE"), - s"MoR append changelog must contain one INSERT and no DELETE: $changeTypes") - }) - - // The changelog view over an INSERT OVERWRITE on a merge-on-read table. - def morReaderWriterChangelogOverwriteCases(format: String): List[Plan.Case] = - List( - morPreparation(format).test( - "readerWriter.changelog.overwrite.mor", - "On a merge-on-read table, a changelog view over an INSERT OVERWRITE that drops one row " + - "reports exactly that row as a DELETE.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT OVERWRITE ${table.name} " + - s"SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.overwrite.mor: $changeTypes") - assert( - changeTypes == Map("DELETE" -> 1L), - s"MoR overwrite changelog must contain the one removed row: $changeTypes") - }) - - // The changelog view over a position-delete DELETE. - def morReaderWriterChangelogDeleteCases(format: String): List[Plan.Case] = - List( - morPreparation(format).test( - "readerWriter.changelog.delete.mor", - "On a merge-on-read table, a changelog view over a DELETE reports exactly one DELETE and " + - "no INSERT.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.delete.mor: $changeTypes") - assert( - changeTypes.getOrElse("DELETE", 0L) == 1 && - !changeTypes.contains("INSERT"), - s"MoR delete changelog must contain one DELETE and no INSERT: $changeTypes") - }) - - // The changelog view over a merge-on-read UPDATE. - def morReaderWriterChangelogUpdateCases(format: String): List[Plan.Case] = - List( - morPreparation(format).test( - "readerWriter.changelog.update.mor", - "On a merge-on-read table, reading a changelog view over an UPDATE is rejected because " + - "position-delete files are not supported in changelog scans.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + - s"WHERE ${Core.long0.columnName} = 2") - val exception = Check.intercept[Exception] { - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - table.spark.sql(s"SELECT * FROM $view").collect() - } - - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage) - .exists(_.contains("Delete files are currently not supported"))), - "MoR update changelog should reject position-delete files") - println( - "DIAG changelog.update.mor: " + - "REJECTED (delete files unsupported in changelog scans)") - }) - - // The changelog view over a merge-on-read MERGE. - def morReaderWriterChangelogMergeCases(format: String): List[Plan.Case] = - List( - morPreparation(format).test( - "readerWriter.changelog.merge.mor", - "On a merge-on-read table, reading a changelog view over a MERGE is rejected because " + - "position-delete files are not supported in changelog scans.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"MERGE INTO ${table.name} target " + - "USING (SELECT CAST(2 AS BIGINT) key " + - "UNION ALL SELECT CAST(9 AS BIGINT)) source " + - s"ON target.${Core.long0.columnName} = source.key " + - s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + - "WHEN NOT MATCHED THEN INSERT " + - s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + - s"${Core.string0.columnName}, ${Core.double0.columnName}, " + - s"${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + - "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") - val exception = Check.intercept[Exception] { - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - table.spark.sql(s"SELECT * FROM $view").collect() - } - - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage) - .exists(_.contains("Delete files are currently not supported"))), - "MoR merge changelog should reject position-delete files") - println( - "DIAG changelog.merge.mor: " + - "REJECTED (delete files unsupported in changelog scans)") - }) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorScenarioKit.scala deleted file mode 100644 index 09d14c2a5..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorScenarioKit.scala +++ /dev/null @@ -1,137 +0,0 @@ -package harness - -// The merge-on-read preparation kit. A merge-on-read table is format version 2 with the delete, -// update and merge modes set to merge-on-read, so a mutation records position-delete files while -// preserving the untouched data files. This layer sits above RTAS, so it also owns the replace-lineage -// merge-on-read preparations. The members are lazy so they initialize on first read, after every -// trait mixed into `object Scenarios` has been constructed. -trait MorScenarioKit extends RtasScenarioKit { - - // Merge-on-read layouts use the standard shapes and record DELETE, UPDATE and MERGE changes in - // position-delete files. Only mutation operations run against these format-version 2 layouts. - private def morLayout(partitioning: Partitioning, format: String): Layout = - Layout( - s"mor-${partitioning.label}/$format", - s"a merge-on-read format-version 2 $format table ${partitioning.description}", - table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource ${partitioning.clause} " + - s"TBLPROPERTIES ('write.format.default'='$format', 'format-version'='2', " + - s"'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read', 'write.merge.mode'='merge-on-read')") - - lazy val morLayouts: List[Layout] = - for { - format <- fileFormats - partitioning <- partitionings - } yield morLayout(partitioning, format) - - lazy val unpartitionedMorLayouts: List[Layout] = - fileFormats.map(format => morLayout(unpartitioned, format)) - - // Layouts that pin how a DELETE is written physically. Both set `write.distribution-mode=none` - // and stay unpartitioned so a single seed INSERT lands every row in ONE data file. Deleting a - // strict subset is then a partial-file match, which Iceberg satisfies by writing a position - // delete under merge-on-read and by rewriting the data file under copy-on-write. The general - // `morLayouts` seed spreads rows over several files, where a delete aligned with a file boundary - // is satisfied by dropping that whole file, so these layouts are what make the physical outcome - // deterministic across formats. - lazy val morVerifyLayouts: List[Layout] = - fileFormats.map(format => Layout( - s"mor-verify/$format", - s"a merge-on-read format-version 2 $format table with no partitioning that writes one data file per insert", - table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'format-version'='2', 'write.distribution-mode'='none', " + - s"'write.delete.mode'='merge-on-read')")) - - lazy val cowVerifyLayouts: List[Layout] = - fileFormats.map(format => Layout( - s"cow-verify/$format", - s"a copy-on-write format-version 2 $format table with no partitioning that writes one data file per insert", - table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'format-version'='2', 'write.distribution-mode'='none', " + - s"'write.delete.mode'='copy-on-write')")) - - lazy val preparedMorCoreTables: List[TablePreparation[CoreTable.type]] = - morLayouts.map(layout => - TablePreparation( - layout.label, - createAndSeed(layout, 3), - description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, " + - "so a mutation writes position-delete files.")) - - // Seed every row into ONE data file. A plain seed INSERT spreads the rows over a couple of files, - // where a delete aligned with a file boundary is satisfied by dropping that whole file. The - // `COALESCE(1)` hint forces a single write task and so a single data file, which makes a - // strict-subset delete a partial-file match: merge-on-read writes a position delete for it, and - // copy-on-write rewrites the data file. - def createAndSeedSingleFile(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = - TableTest(Core).sql("create")(layout.create)() - .sql(s"seed($numberOfRows, one-file)")(table => - s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM (${RowGenerator.valuesClause(Core, numberOfRows)}) AS seed")( - view => assert(view.after.size == numberOfRows, - s"single-file seed expected $numberOfRows rows, got ${view.after.size}")) - - // Seed one data file on a merge-on-read layout, then delete a strict subset, which leaves a live - // position-delete file. A table in this state exercises the scan path where the reader applies a - // position delete, so the read cases run against rows that survive that filtering. - def createAndSeedMorDeleted(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = - createAndSeedSingleFile(layout, numberOfRows) - .step("prep.morDelete") { (spark, table) => - spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1") // a strict subset, so Iceberg writes a position delete - } { view => - assert(view.after.size == numberOfRows - 1, s"MoR prep delete failed: ${view.after.size}") - val deleteFiles = view.spark.sql(s"SELECT count(*) FROM ${view.table}.all_delete_files").collect()(0).getLong(0) - assert(deleteFiles == 1, s"MoR prep must leave a live position-delete file, got $deleteFiles") - } - - lazy val preparedMorReadCoreTables: List[TablePreparation[CoreTable.type]] = - morVerifyLayouts.map { layout => - TablePreparation( - layout.label, - createAndSeedMorDeleted(layout, 3), - "prep.morRead:", - description = s"Three seed rows written as one data file in ${layout.description}, then the " + - "row with key 1 deleted merge-on-read, so keys 2 and 3 remain behind a live position-delete " + - "file that the reader applies at scan time.") - } - - // RTAS preparation on a MERGE-ON-READ table: the replace re-specifies the MoR delete, update, and - // merge modes, so the mutation cases exercise the MoR write path on a replace-lineage table. - protected def morPropsFmt(format: String) = s"'write.format.default'='$format', 'format-version'='2', " + - "'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read', 'write.merge.mode'='merge-on-read'" - - def createAndSeedRtasMor(partitioning: Partitioning, numberOfRows: Int, format: String): TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource ${partitioning.clause} " + - s"TBLPROPERTIES (${morPropsFmt(format)}, 'replace.enabled'='true')")() - .insert(numberOfRows)() - .sql("prep.rtasMor")(t => s"CREATE OR REPLACE TABLE $t USING $dataSource ${partitioning.clause} " + - s"TBLPROPERTIES (${morPropsFmt(format)}) AS SELECT * FROM $t")() - // The OpenHouse user guide requires REFRESH TABLE after a replace, so the Spark session - // reads the committed metadata pointer before the preparation returns. - .sql("prep.rtasMor.refresh")(t => s"REFRESH TABLE $t")() - - lazy val preparedRtasMorCoreTables: List[TablePreparation[CoreTable.type]] = - fileFormats.map { format => - TablePreparation( - s"mor-${unpartitioned.label}/$format", - createAndSeedRtasMor(unpartitioned, 3, format), - "prep.rtasMor:", - description = s"Three seed rows with keys 1, 2 and 3 in a merge-on-read format-version 2 " + - s"$format table ${unpartitioned.description}, then replaced by CREATE OR REPLACE TABLE AS " + - "SELECT re-specifying the merge-on-read modes, so mutations run on replace lineage.") - } - - lazy val preparedNullStringMorCoreTables: List[TablePreparation[CoreTable.type]] = - preparedMorCoreTables.map(withNullStringRow) - - lazy val preparedNullStringRtasMorCoreTables: List[TablePreparation[CoreTable.type]] = - preparedRtasMorCoreTables.map(withNullStringRow) - - lazy val morReadLayoutFormatPreparations: List[TablePreparation[CoreTable.type]] = - preparedMorReadCoreTables - - def morReadLayoutFormatCases: List[Plan.Case] = - layoutFormatCasesFor(morReadLayoutFormatPreparations) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorSurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorSurfaceScenarios.scala deleted file mode 100644 index 5fb0cdc3d..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MorSurfaceScenarios.scala +++ /dev/null @@ -1,76 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The merge-on-read surface families. Both cases start from a table whose delete mode is -// merge-on-read and whose seed is one data file, so a strict-subset DELETE leaves a live -// position-delete file. One compacts that file through rewrite_position_delete_files, the other -// reads it back through the position_deletes metadata table. The cases run on parquet and orc. -trait MorSurfaceScenarios extends MorScenarioKit { - import Rows._ - - private def surfaceMergeOnReadPreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', " + - "'write.delete.mode'='merge-on-read')")() - .sql("seed")(table => - s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM " + - s"(${RowGenerator.valuesClause(Core, 3)}) AS seed")(), - description = s"Three seed rows written as one data file in a merge-on-read $format " + - "table.") - - // The rewrite procedure that compacts position-delete files. - def morSurfaceRewriteProcedureCases(format: String): List[Plan.Case] = - List( - surfaceMergeOnReadPreparation(format).test( - "surface.proc.rewritePositionDeletes", - "After a MoR DELETE creates one position-delete file, rewrite_position_delete_files " + - "compacts it while the 2 surviving rows remain readable.") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.all_delete_files") == "1", - "MoR delete should create one position-delete file") - - table.spark.sql( - "CALL openhouse.system.rewrite_position_delete_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('rewrite-all', 'true'))") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "2", - "rewrite_position_delete_files should preserve live rows") - }) - - // The position_deletes metadata table. - def morSurfaceMetadataCases(format: String): List[Plan.Case] = - List( - surfaceMergeOnReadPreparation(format).test( - "surface.meta.positionDeletes", - "After a MoR DELETE, the position_deletes metadata table reports exactly the one " + - "position-delete entry it created.") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.position_deletes") == "1", - "position_deletes should expose the MoR position delete") - }) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala index ae16fef80..12e3a1659 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala @@ -1,11 +1,6 @@ package harness -/** Mixes the scenario-owned case lists and shared preparation kits into one catalog source. - * - * The traits are listed bottom-up in the feature stack: the standard framework first, then RTAS, - * then merge-on-read, then branch and write-audit-publish. A feature layer's traits extend that - * layer's kit, so removing a layer's files and the traits below removes the layer entirely. - */ +/** Mixes the standard scenario-owned case lists into one catalog source. */ object Scenarios extends DmlScenarios with NestedTypesScenarios @@ -16,20 +11,3 @@ object Scenarios with SurfaceScenarios with HazardReaderWriterScenarios with ImplementationPinScenarios - with RtasDmlScenarios - with RtasDdlScenarios - with RtasInteractionScenarios - with RtasSurfaceScenarios - with RtasHazardScenarios - with MorDmlScenarios - with MorMaintScenarios - with MorReaderWriterScenarios - with MorInteractionScenarios - with MorSurfaceScenarios - with MorForkScenarios - with BranchDmlScenarios - with BranchWapScenarios - with BranchInteractionScenarios - with BranchSurfaceScenarios - with BranchHazardScenarios - with BranchMorScenarios diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala index 47a99eafc..5aea715df 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala @@ -28,10 +28,6 @@ object Plan { crossedFormats.flatMap { format => List( Scenarios.interactionDdlCases(format), - Scenarios.interactionRtasCases(format), - Scenarios.interactionBranchCases(format), - Scenarios.interactionBranchFlagCases(format), - Scenarios.interactionMorCases(format), Scenarios.interactionMiscellaneousCases(format) ).flatten } @@ -39,21 +35,13 @@ object Plan { private def surfaceContributions: List[Case] = crossedFormats.flatMap { format => List( - Scenarios.surfaceBranchMaintenanceCases(format), - Scenarios.surfaceMessageCases(format), - Scenarios.surfaceBranchCases(format), Scenarios.surfaceReaderCases(format), Scenarios.surfaceRewriteProcedureCases(format), - Scenarios.morSurfaceRewriteProcedureCases(format), - Scenarios.surfaceBranchPublishCases(format), Scenarios.surfaceSnapshotProcedureCases(format), Scenarios.surfaceMetadataCases(format), - Scenarios.morSurfaceMetadataCases(format), Scenarios.surfaceConcurrencyCases(format), - Scenarios.surfaceRtasConcurrencyCases(format), Scenarios.surfaceSchemaCases(format), Scenarios.surfaceWriteCases(format), - Scenarios.surfaceBranchWriteCases(format), Scenarios.surfacePinCases(format) ).flatten } @@ -62,8 +50,6 @@ object Plan { crossedFormats.flatMap { format => List( Scenarios.hazardReaderCases(format), - Scenarios.hazardRtasCases(format), - Scenarios.hazardBranchCases(format), Scenarios.hazardWriterCases(format) ).flatten } @@ -72,15 +58,10 @@ object Plan { crossedFormats.flatMap { format => List( Scenarios.readerWriterChangelogAppendCases(format), - Scenarios.morReaderWriterChangelogAppendCases(format), Scenarios.readerWriterChangelogOverwriteCases(format), - Scenarios.morReaderWriterChangelogOverwriteCases(format), Scenarios.readerWriterChangelogDeleteCases(format), - Scenarios.morReaderWriterChangelogDeleteCases(format), Scenarios.readerWriterChangelogUpdateCases(format), - Scenarios.morReaderWriterChangelogUpdateCases(format), Scenarios.readerWriterChangelogMergeCases(format), - Scenarios.morReaderWriterChangelogMergeCases(format), Scenarios.readerWriterIncrementalAndStreamCases(format) ).flatten } @@ -91,7 +72,6 @@ object Plan { Scenarios.ddlConsumerPreparations.flatMap { preparation => List( Scenarios.ddlConsumerDataCases(preparation), - Scenarios.branchDdlConsumerCases(preparation), Scenarios.ddlConsumerCompactionCases(preparation) ).flatten } @@ -100,8 +80,6 @@ object Plan { List( Scenarios.coreDmlCases, Scenarios.partitionedDmlCases, - Scenarios.morDmlCases, - Scenarios.deleteFileModeCases, Scenarios.nestedCases, Scenarios.typesCases, Scenarios.partitionTransformCases, @@ -111,49 +89,26 @@ object Plan { Scenarios.negativeCases, Scenarios.createSchemaCases, Scenarios.layoutFormatCases, - Scenarios.rtasLayoutFormatCases, - Scenarios.branchLayoutFormatCases, - Scenarios.morReadLayoutFormatCases, Scenarios.ddlSchemaCases, Scenarios.ddlNegativeCases, Scenarios.ddlPropertyCases, Scenarios.ddlMiscellaneousCases, Scenarios.ddlPolicyCases, - Scenarios.ddlCtasRtasCases, Scenarios.ddlTagAclFeatureCases, Scenarios.maintenanceCases, - Scenarios.controlPlaneCases, - Scenarios.branchingCases + Scenarios.controlPlaneCases ).flatten ++ interactionContributions ++ - Scenarios.interactionContextCases ++ surfaceContributions ++ hazardContributions ++ Scenarios.hazardContextCases ++ - List( - Scenarios.branchDmlCases, - Scenarios.branchDdlCases, - Scenarios.wapStagedCases, - Scenarios.branchPartitionedDmlCases, - Scenarios.branchMorDmlCases, - Scenarios.rtasDmlCases, - Scenarios.rtasPartitionedDmlCases, - Scenarios.rtasMorDmlCases, - Scenarios.morReadDmlCases, - Scenarios.morCoexistCases - ).flatten ++ ddlConsumerContributions ++ readerWriterContributions ++ List( Scenarios.orderedDmlCases, Scenarios.evolvedDmlCases, - Scenarios.maintenanceMorFoldCases, - Scenarios.maintenanceMorMetaCases, - Scenarios.morHazardCases, - Scenarios.morBranchMergeCases, Scenarios.encryptionPinCases, Scenarios.forkColumnDefaultAndDistributionCases, - Scenarios.forkDeleteFileReplicationCases, Scenarios.forkFileAndCompactionCases ).flatten } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDdlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDdlScenarios.scala deleted file mode 100644 index ea07d3d26..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDdlScenarios.scala +++ /dev/null @@ -1,76 +0,0 @@ -package harness - -import org.apache.iceberg.exceptions.BadRequestException - -// The CTAS and RTAS DDL family. CREATE TABLE AS SELECT copies a seeded table into a new one, and -// CREATE OR REPLACE TABLE AS SELECT replaces a table's content in place. The replace path is gated -// on the replace.enabled table property and is rejected outright while a replication policy is set, -// so both the enabled and the rejected outcomes are pinned here. -trait RtasDdlScenarios extends RtasScenarioKit { - - lazy val ddlCtasRtasCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => - List( - preparation.test( - "ddl.ctas", - "CREATE TABLE AS SELECT from the seeded table produces a new table with the same 3 " + - "rows.") { table => - val targetTable = s"${table.name}_ctas" - - table.spark.sql(s"DROP TABLE IF EXISTS $targetTable") - table.spark.sql( - s"CREATE TABLE $targetTable USING $dataSource AS SELECT * FROM ${table.name}") - - assert( - table.spark.sql(s"SELECT count(*) FROM $targetTable").collect()(0).getLong(0) == 3, - "CTAS lost rows") - - table.spark.sql(s"DROP TABLE IF EXISTS $targetTable") - }, - preparation.test( - "ddl.rtas.enabled", - "With replace.enabled=true, CREATE OR REPLACE TABLE AS SELECT replaces the table's " + - "content, leaving only the 2 rows selected by the replacement query.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('replace.enabled'='true')") - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} WHERE ${Core.long0.columnName} <= 2") - - assert( - table.spark.sql(s"SELECT count(*) FROM ${table.name}").collect()(0).getLong(0) == 2, - "RTAS did not replace") - }, - preparation.test( - "ddl.rtas.disabled", - "Without replace.enabled set, CREATE OR REPLACE TABLE AS SELECT is rejected with a " + - "BadRequestException stating RTAS is not enabled.") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name}")) - - assert( - exception.getMessage.contains("REPLACE TABLE AS SELECT is not enabled"), - s"msg: ${exception.getMessage.take(160)}") - }, - preparation.test( - "ddl.rtas.replicationConflict", - "With replace.enabled=true but a replication policy also set, CREATE OR REPLACE TABLE " + - "AS SELECT is rejected with a BadRequestException about replication being enabled.") { - table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('replace.enabled'='true')") - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") - - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name}")) - - assert( - exception.getMessage.contains("while replication is enabled"), - s"msg: ${exception.getMessage.take(160)}") - }) - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDmlScenarios.scala deleted file mode 100644 index b225e557f..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasDmlScenarios.scala +++ /dev/null @@ -1,16 +0,0 @@ -package harness - -// The RTAS DML buckets. Each bucket is a replace-lineage preparation list crossed with one of the -// shared DML test-case lists that DmlScenarios names, so a replaced table runs the same operations -// and the same assertions as a freshly created one. -trait RtasDmlScenarios extends RtasScenarioKit { this: DmlScenarios => - - lazy val rtasDmlCases: List[Plan.Case] = - preparedRtasCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) ++ - preparedNullStringRtasCoreTables.flatMap(preparation => - nullStringRowTestCases.map(_.runOn(preparation))) - - lazy val rtasPartitionedDmlCases: List[Plan.Case] = - preparedRtasPartitionedCoreTables.flatMap(preparation => - partitionedTableTestCases.map(_.runOn(preparation))) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasHazardScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasHazardScenarios.scala deleted file mode 100644 index 1d7291dc7..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasHazardScenarios.scala +++ /dev/null @@ -1,57 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The RTAS hazard family. A column tag policy is set on a table, the table is then replaced through -// CREATE OR REPLACE TABLE AS SELECT, and the case reads the policy back. The cases run on parquet -// and orc. -trait RtasHazardScenarios extends RtasScenarioKit { this: HazardReaderWriterScenarios => - import Rows._ - - def hazardRtasCases(format: String): List[Plan.Case] = { - val taggedReplacePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => cowCreate(table, format))() - .insert(3)() - .sql("enableReplace")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")() - .sql("tagPii")(table => - s"ALTER TABLE $table MODIFY COLUMN " + - s"${Core.string0.columnName} SET TAG = (PII)")(), - description = s"Three seed rows in a $format table with replace.enabled set and the string " + - "column tagged PII.") - - List( - taggedReplacePreparation.test( - "hazard.rtas.preservesColumnTags", - "CREATE OR REPLACE TABLE AS SELECT preserves the PII column tag policy that was set " + - "before the replace.") { table => - val policiesBefore = - tableProps(table.spark, table.name).getOrElse("policies", "") - assert( - policiesBefore.toLowerCase.contains("pii") || - policiesBefore.toLowerCase.contains("columntags"), - s"PII tag was not stored before RTAS: $policiesBefore") - - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val policiesAfter = - tableProps(table.spark, table.name).getOrElse("policies", "") - - assert( - policiesAfter == policiesBefore, - s"RTAS should preserve the PII column tag: $policiesAfter") - }) - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasInteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasInteractionScenarios.scala deleted file mode 100644 index 60557fa13..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasInteractionScenarios.scala +++ /dev/null @@ -1,390 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The RTAS interaction family. Each case composes CREATE OR REPLACE TABLE AS SELECT with another -// table state or another operation: an evolved schema, a snapshot reference, a maintenance -// procedure, or a REST lock. The cases run on parquet and orc. -trait RtasInteractionScenarios extends RtasScenarioKit { - import Rows._ - - def interactionRtasCases(format: String): List[Plan.Case] = { - val basePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = s"Three seed rows in a $format table.") - val replacePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableReplace")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")(), - description = s"Three seed rows in a $format table with replace.enabled set to true.") - val userPropertyPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', " + - "'replace.enabled'='true', 'user.key'='v1')")() - .insert(3)(), - description = s"Three seed rows in a $format table with replace.enabled set to true and a " + - "user property user.key=v1.") - val retentionPolicyPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.datePartition.columnName}) " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', 'replace.enabled'='true')")() - .insert(3)() - .sql("setRetention")(table => - s"ALTER TABLE $table SET POLICY " + - s"(RETENTION = 30d ON COLUMN ${Core.datePartition.columnName} " + - "WHERE pattern = 'yyyy-MM-dd-HH')")(), - description = s"Three seed rows in a $format table partitioned by datepartition, with " + - "replace.enabled set to true and a 30-day retention policy on datepartition.") - - List( - replacePreparation.test( - "interact.rtas.historyPreserved", - "CREATE OR REPLACE TABLE AS SELECT keeps the pre-replace snapshot in history: two " + - "snapshots exist afterward and the pre-replace one still reads 3 rows.") { table => - val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val snapshotCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.snapshots") - .collect()(0) - .getLong(0) - val historicalRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF $preReplaceSnapshotId") - .collect()(0) - .getLong(0) - - assert( - snapshotCount == 2, - s"replace should retain two snapshots, got $snapshotCount") - assert( - historicalRowCount == 3, - s"pre-replace snapshot should contain 3 rows, got $historicalRowCount") - }, - replacePreparation.test( - "interact.rtas.restoreRejected", - "Rolling back to a snapshot from before CREATE OR REPLACE TABLE AS SELECT is rejected " + - "because it is not an ancestor of the current snapshot.") { table => - val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val exception = Check.intercept[ValidationException]( - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $preReplaceSnapshotId)")) - - assert( - exception.getMessage.contains("not an ancestor"), - "rollback across replacement should reject the old lineage") - }, - replacePreparation.test( - "interact.rtas.setCurrentRecovery", - "set_current_snapshot to a pre-replace snapshot recovers the pre-replace 3 rows.") { table => - val preReplaceSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - table.spark.sql( - "CALL openhouse.system.set_current_snapshot(" + - s"'${catalogRelative(table.name)}', $preReplaceSnapshotId)") - val recoveredRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - recoveredRowCount == 3, - s"set_current_snapshot should recover 3 rows, got $recoveredRowCount") - }, - replacePreparation.test( - "interact.rtas.writeAfter", - "A table replaced by CREATE OR REPLACE TABLE AS SELECT accepts an insert immediately " + - "afterward, and the row count reflects both the replacement's rows and the new insert.") { table => - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - rowCount == 3, - s"replaced table should contain 3 rows after insert, got $rowCount") - }, - replacePreparation.test( - "interact.rtas.partitionSpecChange", - "CREATE OR REPLACE TABLE AS SELECT with a new PARTITIONED BY clause replaces the " + - "partition specification and preserves all 3 rows.") { table => - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"PARTITIONED BY (${Core.datePartition.columnName}) " + - s"AS SELECT * FROM ${table.name}") - val description = table.spark - .sql(s"DESCRIBE TABLE ${table.name}") - .collect() - .toSeq - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - description.exists(_.getString(0) == "# Partition Information") && - description.count( - _.getString(0) == Core.datePartition.columnName) == 2, - "RTAS should replace the partition specification") - assert( - rowCount == 3, - s"partition-spec replacement should preserve 3 rows, got $rowCount") - }, - basePreparation.test( - "interact.rtas.dropsColumn", - "CREATE OR REPLACE TABLE AS SELECT with a narrower column list projects a separate table " + - "down to those two columns while preserving all 3 rows.") { table => - val sideTable = s"${table.name}_dropcol" - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - try { - table.spark.sql( - s"CREATE TABLE $sideTable USING $dataSource " + - "TBLPROPERTIES ('replace.enabled'='true') " + - s"AS SELECT * FROM ${table.name}") - table.spark.sql( - s"CREATE OR REPLACE TABLE $sideTable USING $dataSource AS " + - s"SELECT ${Core.long0.columnName}, ${Core.string0.columnName} " + - s"FROM $sideTable") - val columns = table.spark - .sql(s"SELECT * FROM $sideTable LIMIT 1") - .columns - .toSeq - val rowCount = table.spark - .sql(s"SELECT count(*) FROM $sideTable") - .collect()(0) - .getLong(0) - - assert( - columns == Seq(Core.long0.columnName, Core.string0.columnName), - s"RTAS should project the table to two columns, got $columns") - assert( - rowCount == 3, - s"column-drop RTAS should preserve 3 rows, got $rowCount") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - } - }, - userPropertyPreparation.test( - "interact.rtas.props.userSurvival", - "CREATE OR REPLACE TABLE AS SELECT with no TBLPROPERTIES clause preserves the existing " + - "user.key and replace.enabled properties.") { table => - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val properties = tableProps(table.spark, table.name) - - assert( - properties.get("user.key").contains("v1"), - s"user.key did not survive RTAS: ${properties.get("user.key")}") - assert( - properties.get("replace.enabled").contains("true"), - "replace.enabled did not survive RTAS") - }, - userPropertyPreparation.test( - "interact.rtas.props.statementWins", - "CREATE OR REPLACE TABLE AS SELECT with a TBLPROPERTIES clause overrides the matching " + - "existing property while properties absent from the statement survive unchanged.") { table => - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - "TBLPROPERTIES ('user.key'='v2') " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val properties = tableProps(table.spark, table.name) - - assert( - properties.get("user.key").contains("v2"), - s"statement property should win, got ${properties.get("user.key")}") - assert( - properties.get("replace.enabled").contains("true"), - "properties omitted from RTAS should survive") - }, - replacePreparation.test( - "interact.rtas.props.createDefaulting", - "CREATE OR REPLACE TABLE AS SELECT with write.format.default=orc sets that property, " + - "keeps format-version at 2, and the replaced table remains writable.") { table => - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - "TBLPROPERTIES ('write.format.default'='orc') " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val properties = tableProps(table.spark, table.name) - - assert( - properties.get("write.format.default").contains("orc"), - "RTAS should set write.format.default to orc") - assert( - properties.get("format-version").forall(_ == "2"), - s"format-version drifted: ${properties.get("format-version")}") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 3, - "RTAS table using ORC should remain writable") - }, - retentionPolicyPreparation.test( - "interact.rtas.props.preservesRetentionPolicy", - "CREATE OR REPLACE TABLE AS SELECT with a new partition spec preserves the table's UUID " + - "and its retention policy.") { table => - val tableUuidBefore = tableProps(table.spark, table.name) - .getOrElse("openhouse.tableUUID", "") - val policiesBefore = tableProps(table.spark, table.name) - .getOrElse("policies", "") - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"PARTITIONED BY (${Core.datePartition.columnName}) " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val properties = tableProps(table.spark, table.name) - - assert( - properties.getOrElse("openhouse.tableUUID", "") == - tableUuidBefore, - "table UUID should survive RTAS") - assert( - properties.getOrElse("policies", "") == policiesBefore, - "RTAS should preserve the retention policy") - }, - replacePreparation - .test( - "interact.rtas.withBranch", - "CREATE OR REPLACE TABLE AS SELECT while a branch exists is rejected because branching " + - "is enabled, and both main and the branch remain exactly as they were before the " + - "attempt.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} CREATE BRANCH keepbr") - table.spark.sql( - s"INSERT INTO ${table.name}.branch_keepbr VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val mainStateBefore = table.state - val branchRowCountBefore = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'keepbr'") - .collect()(0) - .getLong(0) - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2")) - val branchRowCountAfter = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} VERSION AS OF 'keepbr'") - .collect()(0) - .getLong(0) - - assert( - exception.getMessage.contains("while branching is enabled"), - s"msg: ${exception.getMessage.take(160)}") - assert( - table.state == mainStateBefore, - "rejected RTAS should not change the main table") - assert( - branchRowCountAfter == branchRowCountBefore, - "rejected RTAS should not change the branch") - } - .copy(knownBugReason = Some( - "The guide documents CREATE OR REPLACE TABLE AS SELECT as incompatible with an " + - "existing branch. The current product accepts the statement. This case keeps the " + - "documented rejection as the contract so the gap is visible; it is skipped until the " + - "product enforces the rejection."))) - } - - // The REST lock has no SQL surface, so this case runs directly against a Ctx like the other - // control-plane cases. The lock must reject both a normal write and RTAS. - def interactRtasOnLockedTable(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_lockrtas" - val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) - spark.sql(s"DROP TABLE IF EXISTS $table") - spark.sql(coreCreateParquet(table)) - spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") - spark.sql(s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')") - try { - val (lockStatus, lockBody) = Rest.post(ctx, s"/v1/databases/$db/tables/$tbl/lock", """{"locked":true}""") - assert(lockStatus >= 200 && lockStatus < 300, s"lock POST failed: $lockStatus $lockBody") - val blocked = Check.intercept[Exception](spark.sql( - s"UPDATE $table SET ${Core.string0.columnName} = 'x' WHERE ${Core.long0.columnName} = 1")) - assert(Exceptions.causeChain(blocked).exists(t => Option(t.getMessage).exists(_.toLowerCase.contains("locked"))), - s"lock not enforced on UPDATE: ${blocked.getMessage.take(160)}") - val rowCountBefore = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) - val snapshotCountBefore = - spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) - val replaceFailure = Check.intercept[BadRequestException]( - spark.sql( - s"CREATE OR REPLACE TABLE $table USING $dataSource " + - s"AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= 2")) - - assert( - replaceFailure.getMessage.toLowerCase.contains("locked"), - s"RTAS rejection did not identify the lock: ${replaceFailure.getMessage.take(160)}") - assert( - spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) == rowCountBefore, - "rejected RTAS changed the table rows") - assert( - spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) == - snapshotCountBefore, - "rejected RTAS committed a snapshot") - } finally { - Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/lock") - spark.sql(s"DROP TABLE IF EXISTS $table") - } - } - - val interactionContextCases: List[Plan.Case] = - List( - Plan.Case( - "interact.rtas.onLockedTable @ embedded", - interactRtasOnLockedTable, - description = "While a table is REST-locked, both UPDATE and CREATE OR REPLACE TABLE AS " + - "SELECT are rejected, and the table keeps the same rows and snapshots.")) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasScenarioKit.scala deleted file mode 100644 index faf85a98e..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasScenarioKit.scala +++ /dev/null @@ -1,56 +0,0 @@ -package harness - -// The RTAS preparation kit. A replace-lineage table is created, seeded, and then re-specified by -// CREATE OR REPLACE TABLE AS SELECT, so every case that runs on one of these preparations exercises -// the replace path. The members are lazy so they initialize on first read, after every trait mixed -// into `object Scenarios` has been constructed. -trait RtasScenarioKit extends ScenarioKit { - - def createAndSeedRtas(partitioning: Partitioning, numberOfRows: Int, format: String): TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(t => s"CREATE TABLE $t ($columnDefinitions) USING $dataSource ${partitioning.clause} " + - s"TBLPROPERTIES ('write.format.default'='$format', 'replace.enabled'='true')")() - .insert(numberOfRows)() - .sql("prep.rtas")(t => s"CREATE OR REPLACE TABLE $t USING $dataSource ${partitioning.clause} " + - s"TBLPROPERTIES ('write.format.default'='$format') AS SELECT * FROM $t")() - // The OpenHouse user guide requires REFRESH TABLE after a replace: the Spark session caches - // the table state it read before the replace, and REFRESH re-reads the committed metadata - // pointer so later statements in the session see the replaced table. - .sql("prep.rtas.refresh")(t => s"REFRESH TABLE $t")() - - // Create and seed, then CREATE OR REPLACE ... AS SELECT * re-specifying the same shape, so the - // table holds the same three rows and was reached through the replace path. The cases run on all - // six layouts, so a replaced table supports the same operations as a freshly created one. - private def rtasPreparationDescription(partitioning: Partitioning, format: String): String = - s"Three seed rows with keys 1, 2 and 3 in a $format table ${partitioning.description}, then " + - "replaced by CREATE OR REPLACE TABLE AS SELECT re-specifying the same shape, so the table " + - "holds the same three rows on replace lineage." - - lazy val preparedRtasCoreTables: List[TablePreparation[CoreTable.type]] = - for { - partitioning <- partitionings - format <- fileFormats - } yield TablePreparation( - s"${partitioning.label}/$format", - createAndSeedRtas(partitioning, 3, format), - "prep.rtas:", - description = rtasPreparationDescription(partitioning, format)) - - lazy val preparedRtasPartitionedCoreTables: List[TablePreparation[CoreTable.type]] = - fileFormats.map { format => - TablePreparation( - s"${partitionedByDate.label}/$format", - createAndSeedRtas(partitionedByDate, 3, format), - "prep.rtas:", - description = rtasPreparationDescription(partitionedByDate, format)) - } - - lazy val preparedNullStringRtasCoreTables: List[TablePreparation[CoreTable.type]] = - preparedRtasCoreTables.map(withNullStringRow) - - lazy val rtasLayoutFormatPreparations: List[TablePreparation[CoreTable.type]] = - preparedRtasCoreTables - - def rtasLayoutFormatCases: List[Plan.Case] = - layoutFormatCasesFor(rtasLayoutFormatPreparations) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasSurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasSurfaceScenarios.scala deleted file mode 100644 index 470063891..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RtasSurfaceScenarios.scala +++ /dev/null @@ -1,120 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The RTAS surface families. One case reads back the message the catalog returns when replace is -// disabled, alongside the other rejection messages a user meets; the other races a replace against -// an append. Both drive CREATE OR REPLACE TABLE AS SELECT, so they belong to the RTAS layer. The -// seeded preparation and the concurrency helpers come from the standard surface trait. The cases run -// on parquet and orc. -trait RtasSurfaceScenarios extends RtasScenarioKit { this: SurfaceScenarios => - import Rows._ - - // A rejection a SQL user reads back is readable when the message is non-empty, carries no - // internal-error marker, carries no raw stack frames, and starts with something other than - // java.lang.NullPointerException. - private def assertReadableMessage(context: String)(e: Throwable): Unit = { - val m = Option(e.getMessage).getOrElse("") - assert(m.nonEmpty, s"$context: empty error message (worst possible readability)") - assert(!m.contains("[INTERNAL_ERROR]"), s"$context: internal error surfaced to the user: ${m.take(160)}") - assert(!m.contains("\n\tat ") && !m.contains("\tat java."), s"$context: stacktrace frames in the user-facing message: ${m.take(160)}") - assert(!m.startsWith("java.lang.NullPointerException"), s"$context: bare NPE surfaced: ${m.take(160)}") - } - - private def surfaceReplacePreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("enableReplace")(table => - s"ALTER TABLE $table SET TBLPROPERTIES ('replace.enabled'='true')")(), - description = s"Three seed rows in an unpartitioned $format table with " + - "replace.enabled=true.") - - // The rejection messages a user reads back from the catalog. - def surfaceMessageCases(format: String): List[Plan.Case] = - List( - surfaceBasePreparation(format).test( - "surface.msg.readabilityGuard", - "Rejection messages for a dropped column, a reserved property, disabled RTAS and " + - "CREATE NAMESPACE are all non-empty, free of internal-error markers, free of raw " + - "stack frames, and not a bare NullPointerException.") { table => - assertReadableMessage("dropColumn")( - Check.intercept[Exception]( - table.spark.sql( - s"ALTER TABLE ${table.name} " + - s"DROP COLUMN ${Core.int0.columnName}"))) - assertReadableMessage("reservedProp")( - Check.intercept[Exception]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES " + - "('openhouse.tableUUID'='x')"))) - assertReadableMessage("rtasDisabled")( - Check.intercept[Exception]( - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name}"))) - assertReadableMessage("createNamespace")( - Check.intercept[Exception]( - table.spark.sql("CREATE NAMESPACE openhouse.nope_ns"))) - }) - - // A replace racing an append. The outcome is either a commit or a typed commit conflict. - def surfaceRtasConcurrencyCases(format: String): List[Plan.Case] = - List( - surfaceReplacePreparation(format).test( - "surface.conc.rtasVsAppend", - "A concurrent CREATE OR REPLACE TABLE AS SELECT racing an INSERT settles at either 2 " + - "rows (replace won) or 3 rows (append also landed), with any failure being a typed " + - "commit conflict.") { table => - def replaceTable(): Unit = - try { - table.spark.sql( - s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + - s"AS SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - } catch { - case exception: Throwable => - assert( - isTypedCommitConflict(exception), - s"RTAS race failed with ${exception.getClass.getName}") - } - def appendRow(): Unit = - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(30 AS BIGINT), 30, 'row-30', 30.5, " + - "true, '2024-01-09-01')") - } catch { - case exception: Throwable => - assert( - isTypedCommitConflict(exception), - s"append race failed with ${exception.getClass.getName}") - } - val threadErrors = - runConcurrently(Seq(() => replaceTable(), () => appendRow())) - - assert( - threadErrors.isEmpty, - s"racing thread failed with a non-conflict error: $threadErrors") - table.spark.sql(s"REFRESH TABLE ${table.name}") - val rowCount = countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}").toLong - assert( - rowCount == 2 || rowCount == 3, - s"RTAS and append race settled at $rowCount rows") - println(s"DIAG conc.rtasVsAppend: settled at $rowCount rows") - }) -} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/BranchDmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/BranchDmlCaseCatalogTest.scala deleted file mode 100644 index aae1a2eaf..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/BranchDmlCaseCatalogTest.scala +++ /dev/null @@ -1,83 +0,0 @@ -package harness - -import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} -import org.junit.jupiter.api.Test - -/** - * Pins the shape of the branch DML buckets: each one is a branch-routed preparation list crossed - * with a DML test-case list the standard layer names. Reading these lists does not execute a case - * or start Spark. - */ -final class BranchDmlCaseCatalogTest { - - @Test - def eachBucketIsThePreparationListCrossedWithItsTestCaseList(): Unit = { - assertEquals( - caseIds(Scenarios.preparedBranchCoreTables, Scenarios.allDmlTestCases) ++ - caseIds(Scenarios.preparedNullStringBranchCoreTables, Scenarios.nullStringRowTestCases), - Scenarios.branchDmlCases.map(_.id), - "branchDmlCases is not its named preparations crossed with its named test cases") - assertEquals( - caseIds(Scenarios.preparedPartitionedBranchCoreTables, Scenarios.partitionedTableTestCases), - Scenarios.branchPartitionedDmlCases.map(_.id), - "branchPartitionedDmlCases is not its named preparations crossed with its named test cases") - assertEquals( - caseIds(Scenarios.preparedBranchMorCoreTables, Scenarios.rowMutationTestCases) ++ - caseIds(Scenarios.preparedNullStringBranchMorCoreTables, Scenarios.nullStringRowTestCases), - Scenarios.branchMorDmlCases.map(_.id), - "branchMorDmlCases is not its named preparations crossed with its named test cases") - } - - @Test - def everyBranchPreparationDescribesTheRoutingItSetsUp(): Unit = { - val describedPreparations = - Scenarios.preparedBranchCoreTables ++ - Scenarios.preparedPartitionedBranchCoreTables ++ - Scenarios.preparedBranchMorCoreTables - - describedPreparations.foreach { preparation => - assertTrue( - preparation.description.contains("spark.wap.branch"), - s"${preparation.label} does not describe the branch routing it sets up") - } - } - - @Test - def theLayoutFormatCasesRunOnTheBranchPreparations(): Unit = - assertEquals( - caseIds(Scenarios.branchLayoutFormatPreparations, "format.materialization"), - Scenarios.branchLayoutFormatCases.map(_.id)) - - @Test - def everyBranchCaseCarriesItsOwnDescriptionAndItsPreparationDescription(): Unit = { - val describedBuckets = List( - Scenarios.branchDmlCases, - Scenarios.branchPartitionedDmlCases, - Scenarios.branchMorDmlCases, - Scenarios.branchLayoutFormatCases).flatten - - describedBuckets.foreach { testCase => - assertTrue( - testCase.description.trim.nonEmpty, - s"${testCase.id} has no description of the operation it runs") - assertTrue( - testCase.preparationDescription.trim.nonEmpty, - s"${testCase.id} has no description of the state it starts from") - } - } - - private def caseIds( - preparations: List[TablePreparation[CoreTable.type]], - testCases: List[DmlTestCase[CoreTable.type]] - ): List[String] = - preparations.flatMap(preparation => - testCases.map(testCase => - s"${preparation.casePrefix}${testCase.id} @ ${preparation.label}")) - - private def caseIds( - preparations: List[TablePreparation[CoreTable.type]], - testCaseId: String - ): List[String] = - preparations.map(preparation => - s"${preparation.casePrefix}$testCaseId @ ${preparation.label}") -} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala index bab8e1efc..9622c9d4b 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala @@ -7,9 +7,9 @@ import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} import org.junit.jupiter.api.Test final class CaseCatalogTest { - private val expectedCaseCount = 2572 + private val expectedCaseCount = 1181 private val expectedCatalogSha256 = - "ffa5fde92303f703e2f9f7febddfe9e912323c3ade8f95339fd073f89a8028c3" + "377f65959e3034c51e078fea72491444b06a6055f37c051184bdc379234b3d57" @Test def orderedCaseCatalogMatchesBaseline(): Unit = { diff --git a/integrations/spark/delta-harness/src/test/scala/harness/MorDmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/MorDmlCaseCatalogTest.scala deleted file mode 100644 index 9768ddb2b..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/MorDmlCaseCatalogTest.scala +++ /dev/null @@ -1,96 +0,0 @@ -package harness - -import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} -import org.junit.jupiter.api.Test - -/** - * Pins the shape of the merge-on-read DML buckets: each one is a merge-on-read preparation list - * crossed with a DML test-case list the standard layer names, plus the pair of cases that assert - * the physical difference between the two write modes. Reading these lists does not execute a case - * or start Spark. - */ -final class MorDmlCaseCatalogTest { - - @Test - def eachBucketIsThePreparationListCrossedWithItsTestCaseList(): Unit = { - assertEquals( - caseIds(Scenarios.preparedMorCoreTables, Scenarios.rowMutationTestCases) ++ - caseIds(Scenarios.preparedNullStringMorCoreTables, Scenarios.nullStringRowTestCases), - Scenarios.morDmlCases.map(_.id), - "morDmlCases is not its named preparations crossed with its named test cases") - assertEquals( - caseIds(Scenarios.preparedRtasMorCoreTables, Scenarios.rowMutationTestCases) ++ - caseIds(Scenarios.preparedNullStringRtasMorCoreTables, Scenarios.nullStringRowTestCases), - Scenarios.rtasMorDmlCases.map(_.id), - "rtasMorDmlCases is not its named preparations crossed with its named test cases") - assertEquals( - caseIds(Scenarios.preparedMorReadCoreTables, Scenarios.readTestCases), - Scenarios.morReadDmlCases.map(_.id), - "morReadDmlCases is not its named preparations crossed with its named test cases") - } - - @Test - def theDeleteFileModeBucketPairsOneMergeOnReadCaseWithOneCopyOnWriteCase(): Unit = - assertEquals( - Scenarios.morVerifyLayouts.map(layout => s"mor.writesDeleteFiles @ ${layout.label}") ++ - Scenarios.cowVerifyLayouts.map(layout => s"cow.writesNoDeleteFiles @ ${layout.label}"), - Scenarios.deleteFileModeCases.map(_.id)) - - @Test - def theLayoutFormatCasesRunOnTheMergeOnReadReadPreparations(): Unit = - assertEquals( - caseIds(Scenarios.morReadLayoutFormatPreparations, "format.materialization"), - Scenarios.morReadLayoutFormatCases.map(_.id)) - - @Test - def everyMergeOnReadLayoutDescribesTheTableItCreates(): Unit = { - val describedLayouts = - Scenarios.morLayouts ++ - Scenarios.unpartitionedMorLayouts ++ - Scenarios.morVerifyLayouts ++ - Scenarios.cowVerifyLayouts - - describedLayouts.foreach { layout => - assertTrue( - layout.description.trim.nonEmpty, - s"layout ${layout.label} has no description") - assertTrue( - layout.description != layout.label, - s"layout ${layout.label} repeats its label; the description must explain the table") - } - } - - @Test - def everyMergeOnReadCaseCarriesItsOwnDescriptionAndItsPreparationDescription(): Unit = { - val describedBuckets = List( - Scenarios.morDmlCases, - Scenarios.rtasMorDmlCases, - Scenarios.morReadDmlCases, - Scenarios.deleteFileModeCases, - Scenarios.morReadLayoutFormatCases).flatten - - describedBuckets.foreach { testCase => - assertTrue( - testCase.description.trim.nonEmpty, - s"${testCase.id} has no description of the operation it runs") - assertTrue( - testCase.preparationDescription.trim.nonEmpty, - s"${testCase.id} has no description of the state it starts from") - } - } - - private def caseIds( - preparations: List[TablePreparation[CoreTable.type]], - testCases: List[DmlTestCase[CoreTable.type]] - ): List[String] = - preparations.flatMap(preparation => - testCases.map(testCase => - s"${preparation.casePrefix}${testCase.id} @ ${preparation.label}")) - - private def caseIds( - preparations: List[TablePreparation[CoreTable.type]], - testCaseId: String - ): List[String] = - preparations.map(preparation => - s"${preparation.casePrefix}$testCaseId @ ${preparation.label}") -} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/RtasDmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/RtasDmlCaseCatalogTest.scala deleted file mode 100644 index f8b331656..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/RtasDmlCaseCatalogTest.scala +++ /dev/null @@ -1,72 +0,0 @@ -package harness - -import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} -import org.junit.jupiter.api.Test - -/** - * Pins the shape of the RTAS DML buckets: each one is a replace-lineage preparation list crossed - * with a DML test-case list the standard layer names. Reading these lists does not execute a case - * or start Spark. - */ -final class RtasDmlCaseCatalogTest { - - @Test - def eachBucketIsThePreparationListCrossedWithItsTestCaseList(): Unit = { - assertEquals( - caseIds(Scenarios.preparedRtasCoreTables, Scenarios.allDmlTestCases) ++ - caseIds(Scenarios.preparedNullStringRtasCoreTables, Scenarios.nullStringRowTestCases), - Scenarios.rtasDmlCases.map(_.id), - "rtasDmlCases is not its named preparations crossed with its named test cases") - assertEquals( - caseIds(Scenarios.preparedRtasPartitionedCoreTables, Scenarios.partitionedTableTestCases), - Scenarios.rtasPartitionedDmlCases.map(_.id), - "rtasPartitionedDmlCases is not its named preparations crossed with its named test cases") - } - - @Test - def theReplaceLineagePreparationsDescribeTheReplaceTheyPerform(): Unit = { - Scenarios.preparedRtasCoreTables.foreach { preparation => - assertTrue( - preparation.description.contains("CREATE OR REPLACE TABLE AS SELECT"), - s"${preparation.label} does not describe the replace it performs") - } - } - - @Test - def theLayoutFormatCasesRunOnTheReplaceLineagePreparations(): Unit = - assertEquals( - caseIds(Scenarios.rtasLayoutFormatPreparations, "format.materialization"), - Scenarios.rtasLayoutFormatCases.map(_.id)) - - @Test - def everyRtasCaseCarriesItsOwnDescriptionAndItsPreparationDescription(): Unit = { - val describedBuckets = List( - Scenarios.rtasDmlCases, - Scenarios.rtasPartitionedDmlCases, - Scenarios.rtasLayoutFormatCases).flatten - - describedBuckets.foreach { testCase => - assertTrue( - testCase.description.trim.nonEmpty, - s"${testCase.id} has no description of the operation it runs") - assertTrue( - testCase.preparationDescription.trim.nonEmpty, - s"${testCase.id} has no description of the state it starts from") - } - } - - private def caseIds( - preparations: List[TablePreparation[CoreTable.type]], - testCases: List[DmlTestCase[CoreTable.type]] - ): List[String] = - preparations.flatMap(preparation => - testCases.map(testCase => - s"${preparation.casePrefix}${testCase.id} @ ${preparation.label}")) - - private def caseIds( - preparations: List[TablePreparation[CoreTable.type]], - testCaseId: String - ): List[String] = - preparations.map(preparation => - s"${preparation.casePrefix}$testCaseId @ ${preparation.label}") -} From 2afbafc37a93f13edd70b4dfad8945a06a1a8e66 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Thu, 27 Aug 2026 18:02:46 -0700 Subject: [PATCH 08/24] refactor(delta-harness): document cases in source Keep runtime case metadata limited to stable identifiers and execution state. Put preparation and test explanations beside their Scala behavior so reviewers can read each case without tracing string registries. Generate a fresh UUID for every table and begin cleanup only after the preparation creates it, which preserves any pre-existing table on a name conflict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/openhouse/DmlScenarios.scala | 1132 ++++++++++------ .../main/scala/harness/openhouse/Env.scala | 6 - .../harness/openhouse/ForkScenarios.scala | 194 ++- .../scala/harness/openhouse/Framework.scala | 71 +- .../HazardReaderWriterScenarios.scala | 1017 ++++++++------- .../ImplementationPinScenarios.scala | 59 +- .../openhouse/InteractionScenarios.scala | 344 ++--- .../openhouse/MaintControlScenarios.scala | 338 ++--- .../openhouse/NegativeDdlScenarios.scala | 995 ++++++++------ .../openhouse/NestedTypesScenarios.scala | 767 ++++++----- .../main/scala/harness/openhouse/Plan.scala | 14 +- .../scala/harness/openhouse/ScenarioKit.scala | 163 ++- .../harness/openhouse/SurfaceScenarios.scala | 1161 +++++++++-------- .../test/scala/harness/CaseCatalogTest.scala | 3 - .../scala/harness/DmlCaseCatalogTest.scala | 96 +- .../scala/harness/TablePreparationTest.scala | 83 +- .../test/scala/harness/TableTestTest.scala | 23 + 17 files changed, 3652 insertions(+), 2814 deletions(-) create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala index 52c959e77..cd0ee3d0d 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala @@ -18,11 +18,13 @@ trait DmlScenarios extends ScenarioKit { // the row change and the snapshot delta that operation caused. Deltas are relative, so a case // holds on any preparation regardless of how many snapshots the preparation itself committed. - val readTestCases: List[DmlTestCase[CoreTable.type]] = List( + /** + * SELECT of foo_col_string alone returns that column for every prepared row in key order and + * leaves the table state unchanged. + */ + private val readProjection: DmlTestCase[CoreTable.type] = DmlTestCase( "read.projection", - s"SELECT of ${Core.string0.columnName} alone returns that column for every prepared row in " + - "key order and leaves the table state unchanged.", table => { val before = table.state val projected = table.spark @@ -38,11 +40,15 @@ trait DmlScenarios extends ScenarioKit { projected == before.rows.sortBy(_.get(Core.long0)).map(_.get(Core.string0)), s"projection returned $projected") assert(after == before, "a read leaves the rows and the snapshot count unchanged") - }), + }) + + /** + * SELECT with a foo_col_long >= 2 predicate returns exactly the prepared rows whose key is 2 or + * greater and leaves the table state unchanged. + */ + private val readFilter: DmlTestCase[CoreTable.type] = DmlTestCase( "read.filter", - s"SELECT with a ${Core.long0.columnName} >= 2 predicate returns exactly the prepared rows " + - "whose key is 2 or greater and leaves the table state unchanged.", table => { val before = table.state val selected = table.spark @@ -58,15 +64,23 @@ trait DmlScenarios extends ScenarioKit { selected == before.rows.map(_.get(Core.long0)).filter(_ >= 2).sorted, s"filtered read returned $selected") assert(after == before, "a read leaves the rows and the snapshot count unchanged") - })) + }) - // The DELETE for the preparations that already hold a row with a null string. It is its own list - // because it only means something against those starting states. - val nullStringRowTestCases: List[DmlTestCase[CoreTable.type]] = List( + /** + * The reads. They select columns by name and write nothing, so they run on any preparation that + * starts from the three seed rows, including one whose column list has grown past that shape. + */ + val readTestCases: List[DmlTestCase[CoreTable.type]] = List( + readProjection, + readFilter) + + /** + * DELETE WHERE foo_col_string IS NULL removes exactly the prepared row whose string is null, + * leaves every other row unchanged, and commits one snapshot. + */ + private val deleteByNullCondition: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.byNullCondition", - s"DELETE WHERE ${Core.string0.columnName} IS NULL removes exactly the prepared row whose " + - "string is null, leaves every other row byte for byte as it was, and commits one snapshot.", table => { val before = table.state @@ -80,13 +94,22 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "DELETE by a null condition commits one snapshot") - })) + }) + /** + * The DELETE that selects a null string. It applies to a preparation that already holds a row + * whose string column is null, and it removes exactly that row. + */ + val nullStringRowTestCases: List[DmlTestCase[CoreTable.type]] = List( + deleteByNullCondition) + + /** + * DELETE WHERE datepartition = '2024-01-01-00' removes the rows in that partition value, keeps + * the rest, and commits one snapshot. + */ private val deleteByPartitionPredicate: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.byPartitionPredicate", - s"DELETE WHERE ${Core.datePartition.columnName} = '2024-01-01-00' removes the rows in that " + - "partition value, keeps the rest, and commits one snapshot.", table => { val before = table.state @@ -103,11 +126,13 @@ trait DmlScenarios extends ScenarioKit { "DELETE by a partition predicate commits one snapshot") }) - private val deleteTestCases: List[DmlTestCase[CoreTable.type]] = List( + /** + * DELETE WHERE foo_col_long < 2 removes the rows below key 2, leaves every other row unchanged, + * and commits one snapshot. + */ + private val deleteByPredicate: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.byPredicate", - s"DELETE WHERE ${Core.long0.columnName} < 2 removes the rows below key 2, keeps every other " + - "row untouched, and commits one snapshot.", table => { val before = table.state @@ -121,11 +146,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "DELETE by a predicate commits one snapshot") - }), + }) + + /** + * DELETE WHERE foo_col_long IN (1, 3) removes keys 1 and 3, leaves every other row exactly as + * prepared, and commits one snapshot. + */ + private val deleteByInList: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.byInList", - s"DELETE WHERE ${Core.long0.columnName} IN (1, 3) removes keys 1 and 3, leaves every " + - "other row exactly as prepared, and commits one snapshot.", table => { val before = table.state @@ -139,11 +168,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "DELETE by an IN list commits one snapshot") - }), + }) + + /** + * DELETE WHERE foo_col_long IN (subquery yielding 2) removes key 2, leaves every other row + * unchanged, and commits one snapshot. + */ + private val deleteByInSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.byInSubquery", - s"DELETE WHERE ${Core.long0.columnName} IN (subquery yielding 2) removes key 2, leaves " + - "every other row exactly as prepared, and commits one snapshot.", table => { val before = table.state @@ -158,11 +191,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "DELETE by an IN subquery commits one snapshot") - }), + }) + + /** + * DELETE WHERE foo_col_long NOT IN (subquery yielding 2) removes every key other than 2, leaves + * the row for key 2 unchanged, and commits one snapshot. + */ + private val deleteByNotInSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.byNotInSubquery", - s"DELETE WHERE ${Core.long0.columnName} NOT IN (subquery yielding 2) removes every key other " + - "than 2 and leaves the row for key 2 exactly as prepared, in one snapshot.", table => { val before = table.state @@ -177,11 +214,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "DELETE by a NOT IN subquery commits one snapshot") - }), + }) + + /** + * DELETE WHERE EXISTS (correlated subquery matching foo_col_long = 2) removes key 2, leaves + * every other row unchanged, and commits one snapshot. + */ + private val deleteByExistsSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.byExistsSubquery", - s"DELETE WHERE EXISTS (correlated subquery matching ${Core.long0.columnName} = 2) removes " + - "key 2, leaves every other row exactly as prepared, and commits one snapshot.", table => { val before = table.state @@ -197,11 +238,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "DELETE by an EXISTS subquery commits one snapshot") - }), + }) + + /** + * DELETE WHERE NOT EXISTS (correlated subquery matching foo_col_long = 2) removes every key + * other than 2, leaves the row for key 2 unchanged, and commits one snapshot. + */ + private val deleteByNotExistsSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.byNotExistsSubquery", - s"DELETE WHERE NOT EXISTS (correlated subquery matching ${Core.long0.columnName} = 2) removes " + - "every key other than 2 and leaves the row for key 2 exactly as prepared, in one snapshot.", table => { val before = table.state @@ -217,11 +262,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "DELETE by a NOT EXISTS subquery commits one snapshot") - }), + }) + + /** + * DELETE WHERE foo_col_long = (scalar subquery yielding 2) removes key 2, leaves every other row + * unchanged, and commits one snapshot. + */ + private val deleteByScalarSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.byScalarSubquery", - s"DELETE WHERE ${Core.long0.columnName} = (scalar subquery yielding 2) removes key 2, " + - "leaves every other row exactly as prepared, and commits one snapshot.", table => { val before = table.state @@ -236,10 +285,12 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "DELETE by a scalar subquery commits one snapshot") - }), + }) + + /** DELETE FROM without a predicate empties the table and commits one snapshot. */ + private val deleteAll: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.all", - "DELETE FROM without a predicate empties the table and commits one snapshot.", table => { val before = table.state @@ -250,11 +301,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "an unconditional DELETE commits one snapshot") - }), + }) + + /** + * DELETE WHERE foo_col_long = 999 matches no row, leaves every row unchanged, and still commits + * one snapshot. + */ + private val deleteNone: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.none", - s"DELETE WHERE ${Core.long0.columnName} = 999 matches no row, keeps every row, and still " + - "commits one snapshot.", table => { val before = table.state @@ -266,12 +321,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a no-match DELETE with a real predicate still commits one snapshot") - }), - deleteByPartitionPredicate, + }) + + /** + * DELETE FROM
AS x WHERE x.foo_col_long < 2 resolves the alias, removes the rows below + * key 2, and commits one snapshot. + */ + private val deleteWithAlias: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.withAlias", - s"DELETE FROM
AS x WHERE x.${Core.long0.columnName} < 2 resolves the alias, removes " + - "the rows below key 2, and commits one snapshot.", table => { val before = table.state @@ -285,10 +343,14 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "DELETE through an alias commits one snapshot") - }), + }) + + /** + * DELETE WHERE false is optimized away: the rows stay as they are and no snapshot is committed. + */ + private val deleteWhereFalseNoSnapshot: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.whereFalse.noSnapshot", - "DELETE WHERE false is optimized away: the rows stay as they are and no snapshot is committed.", table => { val before = table.state @@ -299,10 +361,12 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount, "DELETE WHERE false must not commit a snapshot") - }), + }) + + /** TRUNCATE TABLE empties the table and commits one snapshot. */ + private val deleteTruncate: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.truncate", - "TRUNCATE TABLE empties the table and commits one snapshot.", table => { val before = table.state @@ -313,11 +377,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "TRUNCATE commits one snapshot") - }), + }) + + /** + * DELETE against a snapshot-pinned identifier is rejected with an IllegalArgumentException + * naming that snapshot, and the rows and the snapshot count stay unchanged. + */ + private val deleteAtSnapshotRejected: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.atSnapshot.rejected", - "DELETE against a snapshot-pinned identifier is rejected with an IllegalArgumentException " + - "naming that snapshot, and the table state stays exactly as prepared.", table => { val before = table.state val snapshotId = table.spark @@ -338,13 +406,36 @@ trait DmlScenarios extends ScenarioKit { s"Cannot delete from table at a specific snapshot: $snapshotId", s"unexpected rejection message: ${exception.getMessage}") assert(after == before, "a rejected DELETE leaves the rows and the snapshot count unchanged") - })) + }) - private val updateTestCases: List[DmlTestCase[CoreTable.type]] = List( + /** + * The DELETE operations. They select rows by column name and write no new row, so they run on + * any preparation that starts from the three seed rows, including one whose column list has + * grown past that shape. + */ + private val deleteTestCases: List[DmlTestCase[CoreTable.type]] = List( + deleteByPredicate, + deleteByInList, + deleteByInSubquery, + deleteByNotInSubquery, + deleteByExistsSubquery, + deleteByNotExistsSubquery, + deleteByScalarSubquery, + deleteAll, + deleteNone, + deleteByPartitionPredicate, + deleteWithAlias, + deleteWhereFalseNoSnapshot, + deleteTruncate, + deleteAtSnapshotRejected) + + /** + * UPDATE SET foo_col_string = 'X' WHERE foo_col_long = 2 rewrites that column for key 2 only, + * leaves every other row unchanged, and commits one snapshot. + */ + private val updateByPredicate: DmlTestCase[CoreTable.type] = DmlTestCase( "update.byPredicate", - s"UPDATE SET ${Core.string0.columnName} = 'X' WHERE ${Core.long0.columnName} = 2 rewrites " + - "that column for key 2 only, leaves every other key's value alone, and commits one snapshot.", table => { val before = table.state @@ -360,11 +451,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "UPDATE by a predicate commits one snapshot") - }), + }) + + /** + * UPDATE SET foo_col_string = 'Z' without a WHERE clause rewrites that column for every row and + * commits one snapshot. + */ + private val updateWithoutCondition: DmlTestCase[CoreTable.type] = DmlTestCase( "update.withoutCondition", - s"UPDATE SET ${Core.string0.columnName} = 'Z' without a WHERE clause rewrites that column " + - "for every row and commits one snapshot.", table => { val before = table.state @@ -378,11 +473,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "an unconditional UPDATE commits one snapshot") - }), + }) + + /** + * UPDATE ... WHERE foo_col_long = 99 matches no row, leaves every row unchanged, and still + * commits one snapshot. + */ + private val updateNoMatch: DmlTestCase[CoreTable.type] = DmlTestCase( "update.noMatch", - s"UPDATE ... WHERE ${Core.long0.columnName} = 99 matches no row, leaves every value as it " + - "was, and still commits one snapshot.", table => { val before = table.state @@ -397,11 +496,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a no-match UPDATE still commits one snapshot") - }), + }) + + /** + * UPDATE ... WHERE foo_col_long IN (subquery yielding 2) rewrites key 2 only and commits one + * snapshot. + */ + private val updateByInSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "update.byInSubquery", - s"UPDATE ... WHERE ${Core.long0.columnName} IN (subquery yielding 2) rewrites key 2 only and " + - "commits one snapshot.", table => { val before = table.state @@ -418,11 +521,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "UPDATE by an IN subquery commits one snapshot") - }), + }) + + /** + * UPDATE ... WHERE foo_col_long NOT IN (subquery yielding 2) rewrites every key other than 2 and + * commits one snapshot. + */ + private val updateByNotInSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "update.byNotInSubquery", - s"UPDATE ... WHERE ${Core.long0.columnName} NOT IN (subquery yielding 2) rewrites every key " + - "other than 2 and commits one snapshot.", table => { val before = table.state @@ -439,11 +546,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "UPDATE by a NOT IN subquery commits one snapshot") - }), + }) + + /** + * UPDATE ... WHERE EXISTS (correlated subquery matching foo_col_long = 2) rewrites key 2 only + * and commits one snapshot. + */ + private val updateByExistsSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "update.byExistsSubquery", - s"UPDATE ... WHERE EXISTS (correlated subquery matching ${Core.long0.columnName} = 2) " + - "rewrites key 2 only and commits one snapshot.", table => { val before = table.state @@ -460,11 +571,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "UPDATE by an EXISTS subquery commits one snapshot") - }), + }) + + /** + * UPDATE ... WHERE NOT EXISTS (correlated subquery matching foo_col_long = 2) rewrites every key + * other than 2 and commits one snapshot. + */ + private val updateByNotExistsSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "update.byNotExistsSubquery", - s"UPDATE ... WHERE NOT EXISTS (correlated subquery matching ${Core.long0.columnName} = 2) " + - "rewrites every key other than 2 and commits one snapshot.", table => { val before = table.state @@ -481,11 +596,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "UPDATE by a NOT EXISTS subquery commits one snapshot") - }), + }) + + /** + * UPDATE ... WHERE foo_col_long = (scalar subquery yielding 2) rewrites key 2 only and commits + * one snapshot. + */ + private val updateByScalarSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "update.byScalarSubquery", - s"UPDATE ... WHERE ${Core.long0.columnName} = (scalar subquery yielding 2) rewrites key 2 " + - "only and commits one snapshot.", table => { val before = table.state @@ -502,11 +621,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "UPDATE by a scalar subquery commits one snapshot") - }), + }) + + /** + * UPDATE
AS x SET x.foo_col_string ... WHERE x.foo_col_long = 2 resolves the alias on + * both sides, rewrites key 2 only, and commits one snapshot. + */ + private val updateWithAlias: DmlTestCase[CoreTable.type] = DmlTestCase( "update.withAlias", - s"UPDATE
AS x SET x.${Core.string0.columnName} ... WHERE x.${Core.long0.columnName} " + - "= 2 resolves the alias on both sides, rewrites key 2 only, and commits one snapshot.", table => { val before = table.state @@ -522,12 +645,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "UPDATE through an alias commits one snapshot") - }), + }) + + /** + * UPDATE SET foo_col_string = 'X', foo_col_int = 99 WHERE foo_col_long = 2 rewrites both columns + * of key 2 in one statement and commits one snapshot. + */ + private val updateMultipleColumns: DmlTestCase[CoreTable.type] = DmlTestCase( "update.multipleColumns", - s"UPDATE SET ${Core.string0.columnName} = 'X', ${Core.int0.columnName} = 99 WHERE " + - s"${Core.long0.columnName} = 2 rewrites both columns of key 2 in one statement and commits " + - "one snapshot.", table => { val before = table.state @@ -545,12 +671,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a multi-column UPDATE commits one snapshot") - }), + }) + + /** + * UPDATE SET foo_col_long = foo_col_long + 10 WHERE foo_col_long = 2 moves key 2 to key 12, + * leaves every other row unchanged, and commits one snapshot. + */ + private val updateByExpression: DmlTestCase[CoreTable.type] = DmlTestCase( "update.byExpression", - s"UPDATE SET ${Core.long0.columnName} = ${Core.long0.columnName} + 10 WHERE " + - s"${Core.long0.columnName} = 2 moves key 2 to key 12, leaves the other keys alone, and " + - "commits one snapshot.", table => { val before = table.state @@ -567,12 +696,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "UPDATE by an expression commits one snapshot") - }), + }) + + /** + * UPDATE SET datepartition = '2099-12-31-23' WHERE foo_col_long = 2 moves key 2 to another + * partition value, leaves every other row unchanged, and commits one snapshot. + */ + private val updateMovePartition: DmlTestCase[CoreTable.type] = DmlTestCase( "update.movePartition", - s"UPDATE SET ${Core.datePartition.columnName} = '2099-12-31-23' WHERE " + - s"${Core.long0.columnName} = 2 moves key 2 to another partition value, leaves the other " + - "rows in their partitions, and commits one snapshot.", table => { val before = table.state @@ -591,11 +723,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a partition-moving UPDATE commits one snapshot") - }), + }) + + /** + * UPDATE SET foo_col_string = NULL WHERE foo_col_long = 2 stores a null in that column for key 2 + * only and commits one snapshot. + */ + private val updateNullAssignment: DmlTestCase[CoreTable.type] = DmlTestCase( "update.nullAssignment", - s"UPDATE SET ${Core.string0.columnName} = NULL WHERE ${Core.long0.columnName} = 2 stores a " + - "null in that column for key 2 only and commits one snapshot.", table => { val before = table.state @@ -611,14 +747,35 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "an UPDATE assigning null commits one snapshot") - })) + }) - private val mergeTestCases: List[DmlTestCase[CoreTable.type]] = List( + /** + * The UPDATE operations. They assign columns by name, so they run on any preparation that starts + * from the three seed rows, including one whose column list has grown past that shape. + */ + private val updateTestCases: List[DmlTestCase[CoreTable.type]] = List( + updateByPredicate, + updateWithoutCondition, + updateNoMatch, + updateByInSubquery, + updateByNotInSubquery, + updateByExistsSubquery, + updateByNotExistsSubquery, + updateByScalarSubquery, + updateWithAlias, + updateMultipleColumns, + updateByExpression, + updateMovePartition, + updateNullAssignment) + + /** + * MERGE with only a WHEN NOT MATCHED THEN INSERT * clause appends the two source rows (keys 4 + * and 5) with every source column value, leaves the prepared rows unchanged, and commits one + * snapshot. + */ + private val mergeInsertNotMatched: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.insertNotMatched", - "MERGE with only a WHEN NOT MATCHED THEN INSERT * clause appends the two source rows (keys 4 " + - "and 5) with every source column value, leaves the prepared rows exactly as they were, and " + - "commits one snapshot.", table => { val before = table.state @@ -640,11 +797,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a MERGE that inserts commits one snapshot") - }), + }) + + /** + * MERGE with only a WHEN MATCHED THEN UPDATE clause rewrites the matched key 2, leaves the + * unmatched rows unchanged, and commits one snapshot. + */ + private val mergeUpdateMatched: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.updateMatched", - "MERGE with only a WHEN MATCHED THEN UPDATE clause rewrites the matched key 2, leaves the " + - "unmatched rows alone, and commits one snapshot.", table => { val before = table.state @@ -664,11 +825,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a MERGE that updates commits one snapshot") - }), + }) + + /** + * MERGE with only a WHEN MATCHED THEN DELETE clause removes the matched keys 1 and 3, keeps the + * unmatched rows, and commits one snapshot. + */ + private val mergeDeleteMatched: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.deleteMatched", - "MERGE with only a WHEN MATCHED THEN DELETE clause removes the matched keys 1 and 3, keeps " + - "the unmatched rows, and commits one snapshot.", table => { val before = table.state @@ -686,11 +851,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a MERGE that deletes commits one snapshot") - }), + }) + + /** + * MERGE with both an UPDATE clause and an INSERT clause rewrites the matched key 2 and appends + * the unmatched key 7 in a single statement, and commits one snapshot. + */ + private val mergeUpsert: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.upsert", - "MERGE with both an UPDATE clause and an INSERT clause rewrites the matched key 2 and appends " + - "the unmatched key 7 in a single statement, and commits one snapshot.", table => { val before = table.state @@ -715,11 +884,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "an upsert MERGE commits one snapshot") - }), + }) + + /** + * MERGE with a WHEN NOT MATCHED BY SOURCE THEN DELETE clause removes every row the source does + * not carry, keeps the matched key 2, and commits one snapshot. + */ + private val mergeDeleteNotMatchedBySource: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.deleteNotMatchedBySource", - "MERGE with a WHEN NOT MATCHED BY SOURCE THEN DELETE clause removes every row the source does " + - "not carry, keeps the matched key 2, and commits one snapshot.", table => { val before = table.state @@ -737,12 +910,16 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a not-matched-by-source MERGE commits one snapshot") - }), + }) + + /** + * MERGE with a WHEN MATCHED AND THEN UPDATE clause rewrites only the matched row + * that also satisfies the condition (key 2), leaves matched key 3 unchanged, and commits one + * snapshot. + */ + private val mergeConditionalUpdate: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.conditionalUpdate", - "MERGE with a WHEN MATCHED AND THEN UPDATE clause rewrites only the matched row " + - "that also satisfies the condition (key 2), leaves matched key 3 as it was, and commits one " + - "snapshot.", table => { val before = table.state @@ -763,11 +940,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a conditional-update MERGE commits one snapshot") - }), + }) + + /** + * MERGE with two MATCHED clauses applies the first matching clause per row: key 2 is updated by + * the conditional clause and key 3 falls through to the DELETE clause, in one snapshot. + */ + private val mergeMultipleMatchedClauses: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.multipleMatchedClauses", - "MERGE with two MATCHED clauses applies the first matching clause per row: key 2 is updated " + - "by the conditional clause and key 3 falls through to the DELETE clause, in one snapshot.", table => { val before = table.state @@ -791,11 +972,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a multi-clause MERGE commits one snapshot") - }), + }) + + /** + * MERGE with a WHEN NOT MATCHED AND THEN INSERT clause appends only the source row + * that satisfies the condition (key 4), skips key 5, and commits one snapshot. + */ + private val mergeConditionalInsert: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.conditionalInsert", - "MERGE with a WHEN NOT MATCHED AND THEN INSERT clause appends only the source row " + - "that satisfies the condition (key 4), skips key 5, and commits one snapshot.", table => { val before = table.state @@ -815,11 +1000,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a conditional-insert MERGE commits one snapshot") - }), + }) + + /** + * MERGE carrying UPDATE, INSERT, and NOT MATCHED BY SOURCE DELETE clauses updates key 2, inserts + * key 4, deletes the rows the source omits, and commits one snapshot. + */ + private val mergeAllClauses: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.allClauses", - "MERGE carrying UPDATE, INSERT, and NOT MATCHED BY SOURCE DELETE clauses updates key 2, " + - "inserts key 4, deletes the rows the source omits, and commits one snapshot.", table => { val before = table.state @@ -846,11 +1035,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a MERGE with every clause commits one snapshot") - }), + }) + + /** + * MERGE with WHEN MATCHED THEN UPDATE SET * copies every source column onto the matched key 2, + * leaves the unmatched rows unchanged, and commits one snapshot. + */ + private val mergeUpdateStar: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.updateStar", - "MERGE with WHEN MATCHED THEN UPDATE SET * copies every source column onto the matched key 2, " + - "leaves the unmatched rows exactly as prepared, and commits one snapshot.", table => { val before = table.state @@ -872,11 +1065,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "an UPDATE SET * MERGE commits one snapshot") - }), + }) + + /** + * MERGE whose INSERT clause names a column subset appends key 7 with the named values, leaves + * the unnamed columns null, and commits one snapshot. + */ + private val mergeInsertExplicitColumns: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.insertExplicitColumns", - "MERGE whose INSERT clause names a column subset appends key 7 with the named values, leaves " + - "the unnamed columns null, and commits one snapshot.", table => { val before = table.state @@ -896,11 +1093,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "an explicit-column MERGE insert commits one snapshot") - }), + }) + + /** + * MERGE whose source is a common table expression appends the key 8 that CTE yields, with null + * in every column the CTE does not supply, and commits one snapshot. + */ + private val mergeSourceCTE: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.sourceCTE", - "MERGE whose source is a common table expression appends the key 8 that CTE yields, with " + - "null in every column the CTE does not supply, and commits one snapshot.", table => { val before = table.state @@ -921,11 +1122,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a MERGE from a CTE source commits one snapshot") - }), + }) + + /** + * MERGE whose source is a UNION ALL appends both keys the set operation yields (8 and 9), with + * null in every column the source does not supply, and commits one snapshot. + */ + private val mergeSourceSetOp: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.sourceSetOp", - "MERGE whose source is a UNION ALL appends both keys the set operation yields (8 and 9), with " + - "null in every column the source does not supply, and commits one snapshot.", table => { val before = table.state @@ -947,11 +1152,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a MERGE from a set-operation source commits one snapshot") - }), + }) + + /** + * After the table is emptied, MERGE with a NOT MATCHED INSERT clause inserts both source rows + * (keys 4 and 5) into the empty target and commits one snapshot. + */ + private val mergeIntoEmptyTarget: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.intoEmptyTarget", - "After the table is emptied, MERGE with a NOT MATCHED INSERT clause inserts both source rows " + - "(keys 4 and 5) into the empty target and commits one snapshot.", table => { table.spark.sql(s"DELETE FROM ${table.name}") val before = table.state @@ -976,11 +1185,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a MERGE into an empty target commits one snapshot") - }), + }) + + /** + * MERGE whose source carries a null join key matches no target row on that key: only the + * non-null key 2 is updated, no row is added or removed, and one snapshot is committed. + */ + private val mergeNullJoinKey: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.nullJoinKey", - "MERGE whose source carries a null join key matches no target row on that key: only the " + - "non-null key 2 is updated, no row is added or removed, and one snapshot is committed.", table => { val before = table.state @@ -1001,12 +1214,16 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a MERGE with a null join key commits one snapshot") - }), + }) + + /** + * MERGE with INSERT * resolves the source columns by name: key 7 lands with every source value + * in its named column when the source lists its columns in another order, and one snapshot is + * committed. + */ + private val mergeResolveByName: DmlTestCase[CoreTable.type] = DmlTestCase( "merge.resolveByName", - "MERGE with INSERT * resolves the source columns by name: key 7 lands with every source " + - "value in its named column when the source lists its columns in another order, and one " + - "snapshot is committed.", table => { val before = table.state @@ -1031,13 +1248,37 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a name-resolved MERGE insert commits one snapshot") - })) + }) - private val insertAndOverwriteTestCases: List[DmlTestCase[CoreTable.type]] = List( + /** + * The MERGE operations. Their INSERT clauses write a whole seed-shaped row, so they run on a + * preparation whose column list is still the seed shape. + */ + private val mergeTestCases: List[DmlTestCase[CoreTable.type]] = List( + mergeInsertNotMatched, + mergeUpdateMatched, + mergeDeleteMatched, + mergeUpsert, + mergeDeleteNotMatchedBySource, + mergeConditionalUpdate, + mergeMultipleMatchedClauses, + mergeConditionalInsert, + mergeAllClauses, + mergeUpdateStar, + mergeInsertExplicitColumns, + mergeSourceCTE, + mergeSourceSetOp, + mergeIntoEmptyTarget, + mergeNullJoinKey, + mergeResolveByName) + + /** + * INSERT INTO ... VALUES appends the two literal rows (keys 4 and 5), leaves the prepared rows + * unchanged, and commits one snapshot. + */ + private val insertInto: DmlTestCase[CoreTable.type] = DmlTestCase( "insert.into", - "INSERT INTO ... VALUES appends the two literal rows (keys 4 and 5), keeps the prepared rows, " + - "and commits one snapshot.", table => { val before = table.state @@ -1055,11 +1296,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "INSERT INTO commits one snapshot") - }), + }) + + /** + * INSERT INTO naming a subset of the columns is rejected by the engine with a message naming the + * omitted data, and the rows and the snapshot count stay unchanged. + */ + private val insertExplicitColumns: DmlTestCase[CoreTable.type] = DmlTestCase( "insert.explicitColumns", - "INSERT INTO naming a subset of the columns is rejected by the engine with a message naming " + - "the omitted data, and the table state stays exactly as prepared.", table => { val before = table.state @@ -1078,11 +1323,15 @@ trait DmlScenarios extends ScenarioKit { "expected a partial-INSERT rejection naming the omitted column " + s"(engine limitation), got: ${exceptionMessage.take(200)}") assert(after == before, "a rejected INSERT leaves the rows and the snapshot count unchanged") - }), + }) + + /** + * INSERT INTO ... SELECT appends the row the SELECT produces (key 6), leaves the prepared rows + * unchanged, and commits one snapshot. + */ + private val insertIntoSelect: DmlTestCase[CoreTable.type] = DmlTestCase( "insert.intoSelect", - "INSERT INTO ... SELECT appends the row the SELECT produces (key 6), keeps the prepared rows, " + - "and commits one snapshot.", table => { val before = table.state @@ -1098,11 +1347,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "INSERT INTO ... SELECT commits one snapshot") - }), + }) + + /** + * The DataFrame writeTo(...).append() path appends the frame's row (key 6), keeps the prepared + * rows, and commits one snapshot. + */ + private val appendDataFrame: DmlTestCase[CoreTable.type] = DmlTestCase( "append.dataFrame", - "The DataFrame writeTo(...).append() path appends the frame's row (key 6), keeps the prepared " + - "rows, and commits one snapshot.", table => { val before = table.state @@ -1121,11 +1374,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a DataFrame append commits one snapshot") - }), + }) + + /** + * INSERT OVERWRITE ... VALUES replaces the table contents with the two literal rows (keys 1 and + * 2) and commits one snapshot. + */ + private val insertOverwrite: DmlTestCase[CoreTable.type] = DmlTestCase( "insert.overwrite", - "INSERT OVERWRITE ... VALUES replaces the table contents with the two literal rows (keys 1 " + - "and 2) and commits one snapshot.", table => { val before = table.state @@ -1143,11 +1400,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "INSERT OVERWRITE commits one snapshot") - }), + }) + + /** + * The DataFrame writeTo(...).overwrite(lit(true)) path replaces every row with the frame's row + * (key 8) and commits one snapshot. + */ + private val overwriteDataFrame: DmlTestCase[CoreTable.type] = DmlTestCase( "overwrite.dataFrame", - "The DataFrame writeTo(...).overwrite(lit(true)) path replaces every row with the frame's row " + - "(key 8) and commits one snapshot.", table => { val before = table.state @@ -1166,15 +1427,28 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a DataFrame overwrite commits one snapshot") - })) + }) - // Partition-scoped writes: they only mean something on a table that is partitioned, so they are - // crossed with the partitioned preparations alone. - val partitionedTableTestCases: List[DmlTestCase[CoreTable.type]] = List( + /** + * The appends and the overwrites. They write whole seed-shaped rows, so they run on a + * preparation whose column list is still the seed shape. + */ + private val insertAndOverwriteTestCases: List[DmlTestCase[CoreTable.type]] = List( + insertInto, + insertExplicitColumns, + insertIntoSelect, + appendDataFrame, + insertOverwrite, + overwriteDataFrame) + + /** + * Under partitionOverwriteMode=dynamic, INSERT OVERWRITE with one row replaces only that row's + * partition (2024-01-01-00), leaves the rows of every other partition unchanged, and commits one + * snapshot. + */ + private val insertDynamicOverwrite: DmlTestCase[CoreTable.type] = DmlTestCase( "insert.dynamicOverwrite", - "Under partitionOverwriteMode=dynamic, INSERT OVERWRITE with one row replaces only that row's " + - "partition (2024-01-01-00), keeps the rows of every other partition, and commits one snapshot.", table => { val before = table.state @@ -1196,11 +1470,16 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a dynamic partition overwrite commits one snapshot") - }), + }) + + /** + * The DataFrame writeTo(...).overwritePartitions() path replaces only the partitions the frame + * carries (2024-01-01-00), leaves the rows of every other partition unchanged, and commits one + * snapshot. + */ + private val overwritePartitions: DmlTestCase[CoreTable.type] = DmlTestCase( "overwrite.partitions", - "The DataFrame writeTo(...).overwritePartitions() path replaces only the partitions the frame " + - "carries (2024-01-01-00), keeps the rows of every other partition, and commits one snapshot.", table => { val before = table.state @@ -1221,7 +1500,15 @@ trait DmlScenarios extends ScenarioKit { assert( after.snapshotCount == before.snapshotCount + 1, "a partition overwrite commits one snapshot") - })) + }) + + /** + * The partition-scoped writes. They replace whole partitions, so they apply to a preparation + * that partitions the table, and they cross with the partitioned preparations alone. + */ + val partitionedTableTestCases: List[DmlTestCase[CoreTable.type]] = List( + insertDynamicOverwrite, + overwritePartitions) // --- which cases a preparation is compatible with --- // Compatibility is a property of the starting state, so each list names the states it fits. @@ -1245,6 +1532,10 @@ trait DmlScenarios extends ScenarioKit { val testCasesCompatibleWithAnAddedColumn: List[DmlTestCase[CoreTable.type]] = readTestCases ++ deleteTestCases ++ updateTestCases + /** + * Every DML case, with the partition-predicate DELETE marked as a known bug: the Spark and + * Iceberg rewrite crashes on it when the table carries a write order. + */ val orderedDmlTestCases: List[DmlTestCase[CoreTable.type]] = allDmlTestCases.map { case testCase if testCase == deleteByPartitionPredicate => @@ -1257,29 +1548,43 @@ trait DmlScenarios extends ScenarioKit { // --- standard preparations crossed with the cases they are compatible with --- + /** + * Every DML case on the core preparations, plus the null-string DELETE on the same preparations + * extended with a null-string row. + */ val coreDmlCases: List[Plan.Case] = preparedCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) ++ preparedNullStringCoreTables.flatMap(preparation => nullStringRowTestCases.map(_.runOn(preparation))) + /** The partition-scoped writes on the partitioned preparations. */ val partitionedDmlCases: List[Plan.Case] = preparedPartitionedCoreTables.flatMap(preparation => partitionedTableTestCases.map(_.runOn(preparation))) + /** + * Every DML case on the write-ordered preparations, plus the null-string DELETE on their + * null-string form. + */ val orderedDmlCases: List[Plan.Case] = preparedOrderedCoreTables.flatMap(preparation => orderedDmlTestCases.map(_.runOn(preparation))) ++ preparedNullStringOrderedCoreTables.flatMap(preparation => nullStringRowTestCases.map(_.runOn(preparation))) + /** The cases that address columns by name, on the preparations that added a column. */ val evolvedDmlCases: List[Plan.Case] = preparedEvolvedCoreTables.flatMap(preparation => testCasesCompatibleWithAnAddedColumn.map(_.runOn(preparation))) // --- DDL consumers: a DDL evolves the table, then operations are run against it --- - // Each preparation is one layout evolved by one DDL. A consumer case then runs an operation - // against the evolved table. Plan walks this list so every consumer family lands on the same - // preparation before the next preparation starts. + /** + * One preparation per Parquet and ORC layout and per DDL: three seed rows with keys 1, 2 and 3, + * then one of ADD COLUMN cc int, which the seed rows read as null; foo_col_int widened from int + * to bigint; WRITE ORDERED BY foo_col_long, which gives the table that write sort order; or + * write.distribution-mode set to range, which range distributes later writes. Plan walks this + * list so every consumer family lands on one preparation before the next preparation starts. + */ val ddlConsumerPreparations: List[TablePreparation[CoreTable.type]] = parquetAndOrcLayouts.flatMap { layout => List( @@ -1287,156 +1592,169 @@ trait DmlScenarios extends ScenarioKit { layout.label, createAndSeed(layout, 3) .sql("ddl")(table => s"ALTER TABLE $table ADD COLUMN cc int")(), - "ddlConsume:addColumn.", - description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + - "ADD COLUMN cc int, so the table carries an added column the seed rows read as null."), + "ddlConsume:addColumn."), TablePreparation( layout.label, createAndSeed(layout, 3) .sql("ddl")(table => s"ALTER TABLE $table ALTER COLUMN ${Core.int0.columnName} TYPE bigint")(), - "ddlConsume:typeWiden.", - description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + - s"${Core.int0.columnName} widened from int to bigint."), + "ddlConsume:typeWiden."), TablePreparation( layout.label, createAndSeed(layout, 3) .sql("ddl")(table => s"ALTER TABLE $table WRITE ORDERED BY ${Core.long0.columnName}")(), - "ddlConsume:writeOrder.", - description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + - s"WRITE ORDERED BY ${Core.long0.columnName}, so the table carries that write sort order."), + "ddlConsume:writeOrder."), TablePreparation( layout.label, createAndSeed(layout, 3) .sql("ddl")(table => s"ALTER TABLE $table SET TBLPROPERTIES " + "('write.distribution-mode'='range')")(), - "ddlConsume:distMode.", - description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + - "write.distribution-mode set to range, so writes are range distributed.")) + "ddlConsume:distMode.")) + } + + /** A plain INSERT still lands on the table after the DDL, taking it to four rows. */ + private def dmlWriteCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("dmlWrite") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "table is not writable after DDL") + } + + /** A row-level DELETE still lands on the table after the DDL, taking it to two rows. */ + private def dmlMutateCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("dmlMutate") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 2, + "mutation failed after DDL") + } + + /** + * The seed snapshot from before the DDL is still readable through VERSION AS OF and returns its + * three rows. + */ + private def timeTravelCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("timeTravel") { table => + val seedSnapshotId = + snapshotIds(table.spark, table.name).head + + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF $seedSnapshotId") + .collect()(0) + .getLong(0) == 3, + "seed snapshot is not readable after DDL") + } + + /** + * rollback_to_snapshot back to the seed snapshot undoes an INSERT made after the DDL and returns + * the table to its three seed rows. + */ + private def restoreCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("restore") { table => + val seedSnapshotId = + snapshotIds(table.spark, table.name).head + + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $seedSnapshotId)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 3, + "restore across DDL failed") } - // The reads and writes a consumer runs against the evolved table. + /** + * expire_snapshots retaining only the newest snapshot leaves the table readable with its four + * current rows. + */ + private def expireCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("expire") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "table is unreadable after snapshot expiration") + } + + /** The reads and writes a consumer runs against the table this preparation evolved. */ def ddlConsumerDataCases( preparation: TablePreparation[CoreTable.type]): List[Plan.Case] = List( - preparation.test( - "dmlWrite", - "A plain INSERT still lands on the table after the DDL, taking it to four rows.") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "table is not writable after DDL") - }, - preparation.test( - "dmlMutate", - "A row-level DELETE still lands on the table after the DDL, taking it to two rows.") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "mutation failed after DDL") - }, - preparation.test( - "timeTravel", - "The seed snapshot from before the DDL is still readable through VERSION AS OF and " + - "returns its three rows.") { table => - val seedSnapshotId = - snapshotIds(table.spark, table.name).head - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF $seedSnapshotId") - .collect()(0) - .getLong(0) == 3, - "seed snapshot is not readable after DDL") - }, - preparation.test( - "restore", - "rollback_to_snapshot back to the seed snapshot undoes an INSERT made after the DDL " + - "and returns the table to its three seed rows.") { table => - val seedSnapshotId = - snapshotIds(table.spark, table.name).head + dmlWriteCase(preparation), + dmlMutateCase(preparation), + timeTravelCase(preparation), + restoreCase(preparation), + expireCase(preparation)) - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $seedSnapshotId)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 3, - "restore across DDL failed") - }, - preparation.test( - "expire", - "expire_snapshots retaining only the newest snapshot leaves the table readable with " + - "its four current rows.") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "table is unreadable after snapshot expiration") - }) + /** + * rewrite_data_files compacts the files written across the DDL and preserves the four current + * rows. + */ + private def compactCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("compact") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('min-input-files', '2'))") + + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "compaction changed rows after DDL") + } - // Compaction run against the files written across the DDL. + /** The compaction a consumer runs over the files written across this preparation's DDL. */ def ddlConsumerCompactionCases( preparation: TablePreparation[CoreTable.type]): List[Plan.Case] = List( - preparation.test( - "compact", - "rewrite_data_files compacts the files written across the DDL and preserves the four " + - "current rows.") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('min-input-files', '2'))") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "compaction changed rows after DDL") - }) + compactCase(preparation)) // --- DDL that changes the schema of a seeded table --- - val createSchemaCases: List[Plan.Case] = preparedEmptyCoreTables.map { preparation => - preparation.test( - "create.schema", - "The created table's schema is exactly CoreTable's columns, in declaration order and with " + - "their declared types, and the table holds no rows.") { table => + /** + * The created table's schema is exactly CoreTable's columns, in declaration order and with their + * declared types, and the table holds no rows. + */ + private def createSchemaCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("create.schema") { table => val actual = table.spark .table(table.name) .schema @@ -1448,110 +1766,138 @@ trait DmlScenarios extends ScenarioKit { assert(actual == expected, s"schema is $actual") assert(table.rows.isEmpty, "a table that was never seeded holds no rows") } + + /** The created-schema case on every unseeded preparation. */ + val createSchemaCases: List[Plan.Case] = preparedEmptyCoreTables.map { preparation => + createSchemaCase(preparation) } - val ddlSchemaCases: List[Plan.Case] = preparedCoreTables.flatMap { preparation => - List( - preparation.test( - "ddl.addColumn.single", - "ADD COLUMN adds the column to the schema, the existing rows read null for it, and the row " + - "count is unchanged.") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_int int") + /** + * ADD COLUMN adds the column to the schema, the existing rows read null for it, and the row + * count is unchanged. + */ + private def ddlAddColumnSingleCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.addColumn.single") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_int int") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + val nullCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name} WHERE added_int IS NULL") + .collect()(0) + .getLong(0) + + assert(columnNames.contains("added_int"), s"added_int missing: $columnNames") + assert( + nullCount == table.preparedRows.size, + s"existing rows should read null for added_int: $nullCount != ${table.preparedRows.size}") + assert(table.rows.size == table.preparedRows.size, "ADD COLUMN changed the row count") + } - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - val nullCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name} WHERE added_int IS NULL") - .collect()(0) - .getLong(0) + /** + * ADD COLUMNS with two columns in one statement adds both to the schema and leaves the row count + * unchanged. + */ + private def ddlAddColumnMultipleCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.addColumn.multiple") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMNS (added_a int, added_b string)") - assert(columnNames.contains("added_int"), s"added_int missing: $columnNames") - assert( - nullCount == table.preparedRows.size, - s"existing rows should read null for added_int: $nullCount != ${table.preparedRows.size}") - assert(table.rows.size == table.preparedRows.size, "ADD COLUMN changed the row count") - }, - preparation.test( - "ddl.addColumn.multiple", - "ADD COLUMNS with two columns in one statement adds both to the schema and leaves the row " + - "count unchanged.") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMNS (added_a int, added_b string)") + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + assert( + columnNames.contains("added_a") && columnNames.contains("added_b"), + s"added columns missing: $columnNames") + assert(table.rows.size == table.preparedRows.size, "ADD COLUMNS changed the row count") + } - assert( - columnNames.contains("added_a") && columnNames.contains("added_b"), - s"added columns missing: $columnNames") - assert(table.rows.size == table.preparedRows.size, "ADD COLUMNS changed the row count") - }, - preparation.test( - "ddl.addColumn.comment", - "ADD COLUMN ... COMMENT stores the comment on the added column and the reader sees it.") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_c int COMMENT 'a note'") - - val addedColumn = table.spark - .table(table.name) - .schema - .fields - .find(_.name == "added_c") - .getOrElse(throw new AssertionError("added_c missing")) - - assert( - addedColumn.getComment().contains("a note"), - s"comment not stored: ${addedColumn.getComment()}") - }, - preparation.test( - "ddl.addColumn.position", - s"ADD COLUMN ... AFTER ${Core.long0.columnName} places the added column directly after that " + - "column in the schema.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN added_after int AFTER ${Core.long0.columnName}") + /** ADD COLUMN ... COMMENT stores the comment on the added column and the reader sees it. */ + private def ddlAddColumnCommentCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.addColumn.comment") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_c int COMMENT 'a note'") + + val addedColumn = table.spark + .table(table.name) + .schema + .fields + .find(_.name == "added_c") + .getOrElse(throw new AssertionError("added_c missing")) + + assert( + addedColumn.getComment().contains("a note"), + s"comment not stored: ${addedColumn.getComment()}") + } + + /** + * ADD COLUMN ... AFTER foo_col_long places the added column directly after that column in the + * schema. + */ + private def ddlAddColumnPositionCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.addColumn.position") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN added_after int AFTER ${Core.long0.columnName}") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + + assert( + columnNames.indexOf("added_after") == columnNames.indexOf(Core.long0.columnName) + 1, + s"added_after not after long0: $columnNames") + } + + /** + * ALTER COLUMN foo_col_int TYPE bigint widens the column in the schema and the already-written + * values read back unchanged. + */ + private def ddlAlterColumnTypeWidenCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.alterColumn.typeWiden") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.int0.columnName} TYPE bigint") + + val liveColumns = table.spark.table(table.name).schema.fields.toSeq + .map(field => field.name -> field.dataType.simpleString) + .toMap + val values = table.spark + .sql( + s"SELECT ${Core.int0.columnName} FROM ${table.name} ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) + + assert( + liveColumns.get(Core.int0.columnName).contains("bigint"), + s"int0 not widened: ${liveColumns.get(Core.int0.columnName)}") + assert(values == Seq(1L, 2L, 3L), s"values not preserved after widening: $values") + } + + /** + * RENAME COLUMN renames the column in the schema: the new name is present, the old name is gone, + * and the row count is unchanged. + */ + private def ddlRenameColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation + .test("ddl.renameColumn") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN to_rename int") + table.spark.sql(s"ALTER TABLE ${table.name} RENAME COLUMN to_rename TO renamed_col") val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) assert( - columnNames.indexOf("added_after") == columnNames.indexOf(Core.long0.columnName) + 1, - s"added_after not after long0: $columnNames") - }, - preparation.test( - "ddl.alterColumn.typeWiden", - s"ALTER COLUMN ${Core.int0.columnName} TYPE bigint widens the column in the schema and the " + - "already-written values read back unchanged.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.int0.columnName} TYPE bigint") + columnNames.contains("renamed_col") && !columnNames.contains("to_rename"), + s"RENAME COLUMN silently no-oped: $columnNames") + assert(table.rows.size == table.preparedRows.size, "RENAME COLUMN changed the row count") + } + .copy(knownBugReason = Some( + "RENAME COLUMN is a silent no-op because server-side schema casing normalization " + + "restores the old name.")) - val liveColumns = table.spark.table(table.name).schema.fields.toSeq - .map(field => field.name -> field.dataType.simpleString) - .toMap - val values = table.spark - .sql( - s"SELECT ${Core.int0.columnName} FROM ${table.name} ORDER BY ${Core.long0.columnName}") - .collect() - .toSeq - .map(_.getLong(0)) - - assert( - liveColumns.get(Core.int0.columnName).contains("bigint"), - s"int0 not widened: ${liveColumns.get(Core.int0.columnName)}") - assert(values == Seq(1L, 2L, 3L), s"values not preserved after widening: $values") - }, - preparation - .test( - "ddl.renameColumn", - "RENAME COLUMN renames the column in the schema: the new name is present, the old name is " + - "gone, and the row count is unchanged.") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN to_rename int") - table.spark.sql(s"ALTER TABLE ${table.name} RENAME COLUMN to_rename TO renamed_col") - - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - - assert( - columnNames.contains("renamed_col") && !columnNames.contains("to_rename"), - s"RENAME COLUMN silently no-oped: $columnNames") - assert(table.rows.size == table.preparedRows.size, "RENAME COLUMN changed the row count") - } - .copy(knownBugReason = Some( - "RENAME COLUMN is a silent no-op because server-side schema casing normalization " + - "restores the old name."))) + /** The schema-changing DDL cases on the core preparations. */ + val ddlSchemaCases: List[Plan.Case] = preparedCoreTables.flatMap { preparation => + List( + ddlAddColumnSingleCase(preparation), + ddlAddColumnMultipleCase(preparation), + ddlAddColumnCommentCase(preparation), + ddlAddColumnPositionCase(preparation), + ddlAlterColumnTypeWidenCase(preparation), + ddlRenameColumnCase(preparation)) } } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala index b3e562c60..92734dce3 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala @@ -155,12 +155,6 @@ object Main { "" } println(f"${outcome.label}%-4s ${testCase.id}%-52s try=$attempts$note") - if (testCase.preparationDescription.nonEmpty) { - println(s" Preparation: ${testCase.preparationDescription}") - } - if (testCase.description.nonEmpty) { - println(s" Test: ${testCase.description}") - } } val failed = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala index f9672252f..8295977cc 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala @@ -10,14 +10,22 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal +// The fork cases. OpenHouse compiles and runs against LinkedIn's fork of Apache Iceberg, the +// com.linkedin.iceberg artifacts this module depends on, and a case here pins a behavior the +// Iceberg library decides rather than one the catalog exposes: the column-default path, the write +// distribution default for a partitioned write, the output-file replication key, the read split +// size, and the compaction plan. These behaviors have no catalog SQL surface of their own, so a +// case reaches them through the Iceberg API or a Spark configuration and asserts the result a +// caller can observe. trait ForkScenarios extends ScenarioKit { import Rows._ - // Column-default DDL path, format-parameterized. - // ALTER TABLE ... ADD COLUMN c int DEFAULT 5 is accepted at Spark parse time, but the connector does - // not wire the default into the write path: the default value is not written into the Iceberg schema, - // pre-existing rows read null for the new column, and an INSERT that omits the column is rejected - // with INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA because there is no default to fill it in with. + /** + * ALTER TABLE ADD COLUMN c int DEFAULT 5 parses, and the default value stops at the parser: the + * committed schema records no default for c, pre-existing rows read null for it, and an INSERT + * that omits c is rejected with INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA. The file format is + * the parameter. + */ private def forkColDefaultAddColumn(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_coldef_$fmt" @@ -51,14 +59,15 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $table") } - // Column-default API serialization at the schema level. - // NestedField carries initial-default and write-default, and SchemaParser serializes them into the - // schema JSON. toJson takes no format-version parameter, so the key serializes the same regardless of - // the table's format version. This runs against either artifact through reflection, since the builder - // API does not exist in every Iceberg release jar and a direct reference would fail to compile there: - // when NestedField.builder() is absent, the test records that the column-default API is unsupported; - // when it is present, the test builds a defaulted field, confirms SchemaParser emits initial-default, - // and confirms the value survives a fromJson then toJson round trip. + /** + * A NestedField built with an initial default serializes initial-default into the schema JSON, + * and that value survives a fromJson then toJson round trip. SchemaParser.toJson takes no + * format-version parameter, so the key serializes the same at every format version. On an + * artifact whose NestedField exposes no builder, the column-default API is absent entirely, down + * to the initialDefault and writeDefault accessors, and the case pins that absence. Reflection + * reaches the builder because some Iceberg release jars leave it out, which a direct reference + * would fail to compile against. + */ private def forkColDefaultApiSerialization(ctx: Ctx): Unit = { val nestedFieldCls = Class.forName("org.apache.iceberg.types.Types$NestedField") val builderM = scala.util.Try(nestedFieldCls.getMethod("builder")) @@ -104,8 +113,10 @@ trait ForkScenarios extends ScenarioKit { println("fork.colDefault.api: initial-default serialized with no format-version argument and round-trips") } - // Reflectively builds an optional int NestedField carrying initial-default=dflt. - // Returns None when the builder API is absent so callers can assert that absence directly. + /** + * Reflectively builds an optional int NestedField carrying the given initial default. Returns + * None when the builder API is absent, so a caller can assert that absence directly. + */ private def buildDefaultedIntField(id: Int, name: String, dflt: Int): Option[org.apache.iceberg.types.Types.NestedField] = { val nfCls = Class.forName("org.apache.iceberg.types.Types$NestedField") val bm = scala.util.Try(nfCls.getMethod("builder")) @@ -121,13 +132,13 @@ trait ForkScenarios extends ScenarioKit { Some(b.getClass.getMethod("build").invoke(b).asInstanceOf[org.apache.iceberg.types.Types.NestedField]) } - // Column-default persistence versus read-apply, over data files written before the default existed. - // A schema evolution that adds a defaulted column is committed directly through the low-level - // TableMetadata API, since the public UpdateSchema surface has no set-default operation. The test - // asserts the one deterministic half of this behavior: the default value persists into the committed - // schema. What the OSS Spark read path returns for pre-existing rows over that defaulted column is not - // part of this connector's documented read contract, so that value is recorded for reference rather - // than asserted. + /** + * A column default added after data files exist persists into the committed schema. The schema + * evolution goes through the low-level TableMetadata API because the public UpdateSchema surface + * has no set-default operation. The documented read contract covers schema persistence only. The + * case prints the OSS Spark read result for pre-existing rows as diagnostic output, while its + * assertions stop at the persisted schema. + */ private def forkColDefaultReadApplyProbe(ctx: Ctx): Unit = { val spark = ctx.spark val nfCls = Class.forName("org.apache.iceberg.types.Types$NestedField") @@ -179,13 +190,14 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $t") } - // Partitioned write distribution default, format-parameterized. - // The connector defaults write.distribution-mode to NONE for partitioned writes. With HASH, the - // writer shuffles rows so each partition is written by a - // single task, producing roughly one data file per partition. With NONE, no shuffle happens, so every - // input task writes every partition it holds, producing up to (input tasks times partitions) files. - // This test appends the same multi-task DataFrame into a 4-partition table twice, once under the - // default and once under an explicit hash distribution, and compares the resulting data file counts. + /** + * A partitioned write defaults write.distribution-mode to NONE, so every input task writes every + * partition it holds and one append produces up to (input tasks times partitions) data files. + * Under an explicit HASH distribution the writer shuffles rows so one task owns each partition, + * clustering the append to roughly one file per partition. Appending the same multi-task + * DataFrame into a 4-partition table under each mode therefore yields at least as many files + * under the default as under HASH. The file format is the parameter. + */ private def forkPartitionDistDefault(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark val nParts = 4 @@ -214,21 +226,21 @@ trait ForkScenarios extends ScenarioKit { s"(default=$nDefault hash=$nHash)") } - // (count, sumBytes) of the current data files, used by the compaction tests below. + /** Returns the count and the total byte size of the table's current data files. */ private def dataFileStats(spark: SparkSession, table: String): (Long, Long) = { val r = spark.sql(s"SELECT count(*), coalesce(sum(file_size_in_bytes), 0) FROM $table.data_files").collect()(0) (r.getLong(0), r.getLong(1)) } - // Output-file replication factor at the OutputFileFactory level. - // The property key that OutputFileFactory stamps into the per-output-file property map is - // FILE_REPLICATION_FACTOR, "file-replication-factor". It is not a settable table property; it is the - // key HDFS reads to set block replication on an output file when a replication factor is supplied to - // the factory. Only the delete-file write path feeds a replication factor to the factory; data-file - // factories never set it. This test builds a factory with an explicit replication factor and asserts - // the exact key it stamps into the output-file property map, then confirms writes still succeed and - // return correct rows afterward. Reflection is used because the builder method and getProperties are - // not part of the public compiled API on every Iceberg artifact this test runs against. + /** + * OutputFileFactory exposes FILE_REPLICATION_FACTOR as "file-replication-factor", and a factory + * built with a replication factor stamps that key into the property map of the output files it + * creates. Writes made through the table afterward still return the correct rows. It is not a + * settable table property; it is the key HDFS reads to set block replication on an output file + * when a replication factor is supplied to the factory, and the delete-file write path is the one + * path that supplies one. Reflection reaches the builder and getProperties because some Iceberg + * artifacts leave them out of the public compiled API. + */ private def forkFileReplicationFactor(ctx: Ctx): Unit = { val spark = ctx.spark val offCls = Class.forName("org.apache.iceberg.io.OutputFileFactory") @@ -272,11 +284,14 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $table") } - // Spark read split size, format-parameterized. - // spark.sql.iceberg.split-size controls how the read path combines or splits data files into read - // tasks. With several small files, a large split size combines them into fewer read tasks and a tiny - // split size splits them into more, visible through rdd.getNumPartitions, while the row set stays - // invariant. This test also checks the same knob at the planner level directly. + /** + * spark.sql.iceberg.split-size decides how the read path combines data files into read tasks. + * Over several small files, a large split size combines them into fewer read tasks and a tiny + * split size splits them into more, visible through rdd.getNumPartitions, and both reads return + * the same rows. The planner shows the same effect directly: a split size above the whole table + * plans one task group, and a split size below one file plans one group per file. The file format + * is the parameter. + */ private def forkSplitSize(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_splitsize_$fmt" @@ -334,11 +349,12 @@ trait ForkScenarios extends ScenarioKit { } } - // Bin-pack compaction weighted by data-file length. - // rewrite_data_files packs data files into rewrite groups weighted by file length. The weighting - // decision itself is an internal planner detail with no local SQL surface, so this test observes what - // is externally checkable: compacting a table with unevenly sized data files through - // rewrite_data_files preserves both the row count and every row's value. + /** + * rewrite_data_files packs data files into rewrite groups weighted by file length. Compacting a + * table whose data files are unevenly sized preserves the row count and every row's value, which + * is the observable result of that packing; the weighting itself is a planner decision that no + * SQL surface exposes. The file format is the parameter. + */ private def forkBinPackByLength(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_binpack_$fmt" @@ -366,13 +382,13 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $table") } - // Budgeted rewrite ordering by file-sequence-number. - // A budgeted rewrite orders candidate files by file-sequence-number when spending its rewrite budget. - // That ordering decision is metadata-level with no local SQL surface, and it shares its execution path - // with the bin-pack compaction test above, so this test checks the distinct, externally observable - // half: the ordering key, file_sequence_number on the entries metadata table, is exposed and increases - // monotonically across commits, and rewrite_data_files with rewrite-all preserves the row set. The - // Sequence numbers define the ordering, so a single format is sufficient here. + /** + * file_sequence_number is exposed on the live data-file entries of the entries metadata table and + * increases monotonically across commits, and rewrite_data_files with rewrite-all preserves the + * row count and the row set. A budgeted rewrite spends its budget in file-sequence-number order, + * so that column is the observable half of the ordering decision. Sequence numbers order commits + * the same way in every file format, so parquet alone covers this behavior. + */ private def forkCompactionOrder(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_compord" @@ -407,82 +423,52 @@ trait ForkScenarios extends ScenarioKit { spark.sql(s"DROP TABLE IF EXISTS $table") } + /** The column-default and write-distribution fork cases. */ val forkColumnDefaultAndDistributionCases: List[Plan.Case] = List( Plan.Case( "fork.colDefault.addColumnInert @ parquet", - forkColDefaultAddColumn("parquet"), - description = "ALTER TABLE ADD COLUMN ... DEFAULT is accepted on a parquet table, but the " + - "default is not written into the schema, pre-existing rows read null for it, and an insert " + - "that omits the column is rejected."), + forkColDefaultAddColumn("parquet")), Plan.Case( "fork.colDefault.addColumnInert @ orc", - forkColDefaultAddColumn("orc"), - description = "ALTER TABLE ADD COLUMN ... DEFAULT is accepted on an orc table, but the " + - "default is not written into the schema, pre-existing rows read null for it, and an insert " + - "that omits the column is rejected."), + forkColDefaultAddColumn("orc")), Plan.Case( "fork.colDefault.apiSerialization @ core", - forkColDefaultApiSerialization, - description = "A NestedField built with an initial default serializes 'initial-default' into " + - "the schema JSON and the value survives a fromJson/toJson round trip, on a build that carries " + - "the column-default API."), + forkColDefaultApiSerialization), Plan.Case( "fork.colDefault.readApplyProbe @ core", - forkColDefaultReadApplyProbe, - description = "A column default added after existing data files persists into the committed " + - "schema. The read path's returned value for pre-existing rows over that column is recorded " + - "for reference, since it is not part of this connector's documented read contract."), + forkColDefaultReadApplyProbe), Plan.Case( "fork.partitionDist.default @ parquet", - forkPartitionDistDefault("parquet"), - description = "Appending the same multi-task write to a 4-way partitioned parquet table " + - "produces at least as many data files under the default write distribution mode as under an " + - "explicit hash distribution, and hash distribution clusters to about one file per partition."), + forkPartitionDistDefault("parquet")), Plan.Case( "fork.partitionDist.default @ orc", - forkPartitionDistDefault("orc"), - description = "Appending the same multi-task write to a 4-way partitioned orc table produces " + - "at least as many data files under the default write distribution mode as under an explicit " + - "hash distribution, and hash distribution clusters to about one file per partition.")) + forkPartitionDistDefault("orc"))) - // The fork cases are two contribution lists. One more fork entry sits between them in the - // catalog; the layer that owns that entry supplies it and Plan keeps the order. + /** + * The output-file, split-size and compaction fork cases. They are the second of two fork + * contribution lists: one more fork entry sits between the two in the catalog, supplied by the + * layer that owns it, and Plan keeps that order. + */ val forkFileAndCompactionCases: List[Plan.Case] = List( Plan.Case( "fork.fileReplicationFactor @ core", - forkFileReplicationFactor, - description = "OutputFileFactory exposes the key 'file-replication-factor', a factory built " + - "with replication factor 2 stamps that key into its output-file properties, and writes made " + - "through the table afterward still produce the correct rows."), + forkFileReplicationFactor), Plan.Case( "fork.splitSize @ parquet", - forkSplitSize("parquet"), - description = "Reading a multi-file parquet table under a large spark.sql.iceberg.split-size " + - "and a tiny one returns the same rows both times, and the tiny split size does not decrease " + - "the read task count relative to the large one."), + forkSplitSize("parquet")), Plan.Case( "fork.splitSize @ orc", - forkSplitSize("orc"), - description = "Reading a multi-file orc table under a large spark.sql.iceberg.split-size and " + - "a tiny one returns the same rows both times, and the tiny split size does not decrease the " + - "read task count relative to the large one."), + forkSplitSize("orc")), Plan.Case( "fork.binPackByLength @ parquet", - forkBinPackByLength("parquet"), - description = "Compacting a parquet table with unevenly sized data files through " + - "rewrite_data_files preserves the row count and every row's value."), + forkBinPackByLength("parquet")), Plan.Case( "fork.binPackByLength @ orc", - forkBinPackByLength("orc"), - description = "Compacting an orc table with unevenly sized data files through " + - "rewrite_data_files preserves the row count and every row's value."), + forkBinPackByLength("orc")), Plan.Case( "fork.compactionOrder @ parquet", - forkCompactionOrder, - description = "File sequence numbers on live data-file entries are exposed and increase " + - "monotonically across commits, and rewrite_data_files with rewrite-all preserves the row " + - "count and the row set.")) + forkCompactionOrder)) } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala index a8a79b253..7515b87ac 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala @@ -4,6 +4,7 @@ import org.apache.spark.sql.{Row, SparkSession} import java.time.LocalDate import java.time.LocalDateTime import java.time.format.DateTimeFormatter +import java.util.UUID import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal @@ -285,10 +286,15 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste * Execute these steps as a reusable preparation, then hand the prepared table to one localized * test body. The fresh-table lifecycle covers both the preparation and the test body. */ - def prepare(ctx: Ctx)(use: PreparedTable[S] => Unit): Unit = withTable(ctx) { table => + def prepare(ctx: Ctx)(use: PreparedTable[S] => Unit): Unit = + withTable(ctx) { (table, markTableCreated) => val (preparedRows, preparedSnapshotCount) = - steps.foldLeft((Seq.empty[Row], 0L)) { case ((beforeRows, beforeSnapshots), step) => + steps.zipWithIndex.foldLeft((Seq.empty[Row], 0L)) { + case ((beforeRows, beforeSnapshots), (step, stepIndex)) => step.execute(ctx.spark, table, schema) + if (stepIndex == 0) { + markTableCreated() + } val afterRows = PreparedTable.currentRows(ctx.spark, table, schema) val afterSnapshots = PreparedTable.snapshotCount(ctx.spark, table) step.validate( @@ -305,28 +311,32 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste use(PreparedTable(ctx.spark, table, schema, preparedRows, preparedSnapshotCount)) } - // Gives the preparation a fresh table and drops it after the test. A test failure remains primary, - // and a cleanup failure is attached to it as a suppressed exception. - private def withTable(ctx: Ctx)(use: String => Unit): Unit = { - val table = s"${ctx.namespace}.t_${TableTest.counter.incrementAndGet()}" - ctx.spark.sql(s"DROP TABLE IF EXISTS $table") + // Gives the preparation a unique table name and drops that table after the test. Cleanup starts + // only after the first preparation step creates the table, so a name conflict preserves the + // pre-existing table. A test failure stays primary, and a cleanup failure is attached to it as a + // suppressed exception. + private def withTable(ctx: Ctx)(use: (String, () => Unit) => Unit): Unit = { + val table = TableTest.nextQualifiedTableName(ctx.namespace) + var tableCreated = false var testFailure: Option[Throwable] = None try { - use(table) + use(table, () => tableCreated = true) } catch { case failure: Throwable => testFailure = Some(failure) throw failure } finally { - try { - ctx.spark.sql(s"DROP TABLE IF EXISTS $table") - } catch { - case cleanupFailure: Throwable => - testFailure match { - case Some(failure) => failure.addSuppressed(cleanupFailure) - case None => throw cleanupFailure - } + if (tableCreated) { + try { + ctx.spark.sql(s"DROP TABLE IF EXISTS $table") + } catch { + case cleanupFailure: Throwable => + testFailure match { + case Some(failure) => failure.addSuppressed(cleanupFailure) + case None => throw cleanupFailure + } + } } } } @@ -335,8 +345,12 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste object TableTest { private val counter = new java.util.concurrent.atomic.AtomicInteger(0) + def apply[S <: Schema](schema: S): TableTest[S] = new TableTest(schema, Vector.empty) def seedCounter(value: Int): Unit = counter.set(value) + + private[harness] def nextQualifiedTableName(namespace: String): String = + s"$namespace.t_${UUID.randomUUID().toString.replace("-", "")}_${counter.incrementAndGet()}" } /** An immutable recipe that prepares one fresh table for each localized test case. */ @@ -344,15 +358,14 @@ final case class TablePreparation[S <: Schema]( label: String, preparation: TableTest[S], casePrefix: String = "", - afterTest: PreparedTable[S] => Unit = (_: PreparedTable[S]) => (), - description: String + afterTest: PreparedTable[S] => Unit = (_: PreparedTable[S]) => () ) { - require(description.trim.nonEmpty, s"table preparation $label needs a description") - - def test( - caseName: String, - testDescription: String - )(body: PreparedTable[S] => Unit): Plan.Case = + /** + * Build the case that runs `body` against one freshly prepared table. The case ID combines the + * preparation's prefix and label with `caseName`, so one test body yields a separate case on + * every preparation it runs on. + */ + def test(caseName: String)(body: PreparedTable[S] => Unit): Plan.Case = Plan.Case( s"$casePrefix$caseName @ $label", context => preparation.prepare(context) { table => @@ -372,21 +385,17 @@ final case class TablePreparation[S <: Schema]( } } } - }, - description = testDescription, - preparationDescription = description) + }) } final case class DmlTestCase[S <: Schema]( id: String, - description: String, run: PreparedTable[S] => Unit, knownBugReason: Option[String] = None ) { - require(description.trim.nonEmpty, s"DML test case $id needs a description") - + /** Build the case that runs this operation against a table `preparation` produces. */ def runOn(preparation: TablePreparation[S]): Plan.Case = preparation - .test(id, description)(run) + .test(id)(run) .copy(knownBugReason = knownBugReason) } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala index 5cf262846..35d61e487 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala @@ -13,547 +13,606 @@ import scala.util.control.NonFatal // The copy-on-write reader, writer and hazard families. The reader and writer cases pin the // changelog view, the incremental read and the structured-streaming reader and writer against a // plain copy-on-write table. The hazard cases pin what happens when two operations that can -// interfere are run against the same table. `cowCreate` states the standard copy-on-write table -// shape, so a feature layer reaches it through a self-type on this trait. +// interfere are run against the same table. Plan crosses every family here with the parquet and +// orc file formats. `cowCreate` states the standard copy-on-write table shape, so a feature layer +// reaches it through a self-type on this trait. trait HazardReaderWriterScenarios extends ScenarioKit { import Rows._ + /** The CREATE statement for a copy-on-write table in the given file format. */ protected def cowCreate(t: String, fmt: String): String = s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')" - // Every reader and writer family is crossed with parquet and orc. Each family builds its own copy - // of the preparation, so a family reads on its own. + /** + * Three seed rows in a copy-on-write table in the given file format. Each family builds its own + * table from this recipe, so a family reads on its own. + */ private def cowPreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation( format, TableTest(Core) .sql("create")(table => cowCreate(table, format))() - .insert(3)(), - description = s"Three seed rows in a copy-on-write $format table.") + .insert(3)()) + + /** A changelog view over an appended row reports exactly one INSERT and no DELETE. */ + private def readerWriterChangelogAppendCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.changelog.append") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.append: $changeTypes") + assert( + changeTypes.getOrElse("INSERT", 0L) == 1 && + !changeTypes.contains("DELETE"), + s"append changelog must contain one INSERT and no DELETE: $changeTypes") + } - // The changelog view over an append. + /** The changelog case for an append, on three seed rows in the given file format. */ def readerWriterChangelogAppendCases(format: String): List[Plan.Case] = List( - cowPreparation(format).test( - "readerWriter.changelog.append", - "A changelog view over an appended row reports exactly one INSERT and no DELETE.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.append: $changeTypes") - assert( - changeTypes.getOrElse("INSERT", 0L) == 1 && - !changeTypes.contains("DELETE"), - s"append changelog must contain one INSERT and no DELETE: $changeTypes") - }) + readerWriterChangelogAppendCase(format)) + + /** + * A changelog view over an INSERT OVERWRITE that drops one row reports exactly that row as a + * DELETE. + */ + private def readerWriterChangelogOverwriteCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.changelog.overwrite") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT OVERWRITE ${table.name} " + + s"SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.overwrite: $changeTypes") + assert( + changeTypes == Map("DELETE" -> 1L), + s"overwrite changelog must contain the one removed row: $changeTypes") + } - // The changelog view over an INSERT OVERWRITE. + /** The changelog case for an INSERT OVERWRITE, on three seed rows in the given file format. */ def readerWriterChangelogOverwriteCases(format: String): List[Plan.Case] = List( - cowPreparation(format).test( - "readerWriter.changelog.overwrite", - "A changelog view over an INSERT OVERWRITE that drops one row reports exactly that row " + - "as a DELETE.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT OVERWRITE ${table.name} " + - s"SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.overwrite: $changeTypes") - assert( - changeTypes == Map("DELETE" -> 1L), - s"overwrite changelog must contain the one removed row: $changeTypes") - }) + readerWriterChangelogOverwriteCase(format)) + + /** A changelog view over a DELETE reports exactly one DELETE and no INSERT. */ + private def readerWriterChangelogDeleteCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.changelog.delete") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.delete: $changeTypes") + assert( + changeTypes.getOrElse("DELETE", 0L) == 1 && + !changeTypes.contains("INSERT"), + s"delete changelog must contain one DELETE and no INSERT: $changeTypes") + } - // The changelog view over a DELETE. + /** The changelog case for a DELETE, on three seed rows in the given file format. */ def readerWriterChangelogDeleteCases(format: String): List[Plan.Case] = List( - cowPreparation(format).test( - "readerWriter.changelog.delete", - "A changelog view over a DELETE reports exactly one DELETE and no INSERT.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.delete: $changeTypes") - assert( - changeTypes.getOrElse("DELETE", 0L) == 1 && - !changeTypes.contains("INSERT"), - s"delete changelog must contain one DELETE and no INSERT: $changeTypes") - }) + readerWriterChangelogDeleteCase(format)) + + /** + * A changelog view over an UPDATE reports the old row as a DELETE and the new value as an + * INSERT. + */ + private def readerWriterChangelogUpdateCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.changelog.update") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.update: $changeTypes") + assert( + changeTypes == Map("DELETE" -> 1L, "INSERT" -> 1L), + s"update changelog must contain the old and new row versions: $changeTypes") + } - // The changelog view over an UPDATE. + /** The changelog case for an UPDATE, on three seed rows in the given file format. */ def readerWriterChangelogUpdateCases(format: String): List[Plan.Case] = List( - cowPreparation(format).test( - "readerWriter.changelog.update", - "A changelog view over an UPDATE reports the old row as a DELETE and the new value as " + - "an INSERT.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + - s"WHERE ${Core.long0.columnName} = 2") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.update: $changeTypes") - assert( - changeTypes == Map("DELETE" -> 1L, "INSERT" -> 1L), - s"update changelog must contain the old and new row versions: $changeTypes") - }) + readerWriterChangelogUpdateCase(format)) + + /** + * A changelog view over a MERGE that updates one row and inserts another reports one DELETE and + * two INSERTs. + */ + private def readerWriterChangelogMergeCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.changelog.merge") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"MERGE INTO ${table.name} target " + + "USING (SELECT CAST(2 AS BIGINT) key " + + "UNION ALL SELECT CAST(9 AS BIGINT)) source " + + s"ON target.${Core.long0.columnName} = source.key " + + s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + + "WHEN NOT MATCHED THEN INSERT " + + s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + + s"${Core.string0.columnName}, ${Core.double0.columnName}, " + + s"${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + + "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + val changeTypes = table.spark + .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + println(s"DIAG changelog.merge: $changeTypes") + assert( + changeTypes == Map("DELETE" -> 1L, "INSERT" -> 2L), + s"merge changelog must contain one update and one insert: $changeTypes") + } - // The changelog view over a MERGE. + /** The changelog case for a MERGE, on three seed rows in the given file format. */ def readerWriterChangelogMergeCases(format: String): List[Plan.Case] = List( - cowPreparation(format).test( - "readerWriter.changelog.merge", - "A changelog view over a MERGE that updates one row and inserts another reports one " + - "DELETE and two INSERTs.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"MERGE INTO ${table.name} target " + - "USING (SELECT CAST(2 AS BIGINT) key " + - "UNION ALL SELECT CAST(9 AS BIGINT)) source " + - s"ON target.${Core.long0.columnName} = source.key " + - s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + - "WHEN NOT MATCHED THEN INSERT " + - s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + - s"${Core.string0.columnName}, ${Core.double0.columnName}, " + - s"${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + - "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.merge: $changeTypes") - assert( - changeTypes == Map("DELETE" -> 1L, "INSERT" -> 2L), - s"merge changelog must contain one update and one insert: $changeTypes") - }) + readerWriterChangelogMergeCase(format)) + + /** An incremental scan spanning an appended row returns exactly that one row. */ + private def readerWriterIncrementalAppendCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.incremental.append") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.append: added=$addedRowCount") + assert( + addedRowCount == 1, + s"append incremental scan should contain one row, got $addedRowCount") + } - // Incremental reads between two snapshots, and the structured-streaming reader and writer. - def readerWriterIncrementalAndStreamCases(format: String): List[Plan.Case] = - List( - cowPreparation(format).test( - "readerWriter.incremental.append", - "An incremental scan spanning an appended row returns exactly that one row.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head + /** An incremental scan spanning a DELETE-only snapshot returns no rows. */ + private def readerWriterIncrementalDeleteCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.incremental.delete") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.delete: added=$addedRowCount") + assert( + addedRowCount == 0, + s"delete-only incremental scan must not return appended rows: $addedRowCount") + } + + /** An incremental scan spanning an INSERT OVERWRITE that only removes rows returns no rows. */ + private def readerWriterIncrementalOverwriteCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.incremental.overwrite") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"INSERT OVERWRITE ${table.name} " + + s"SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.overwrite: added=$addedRowCount") + assert( + addedRowCount == 0, + s"overwrite-only incremental scan must not return appended rows: $addedRowCount") + } + + /** An incremental scan spanning an UPDATE-only snapshot returns no rows. */ + private def readerWriterIncrementalUpdateCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.incremental.update") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2") + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", seedSnapshotId) + .option("end-snapshot-id", currentSnapshotId) + .load(table.name) + .count() + + println(s"DIAG incremental.update: added=$addedRowCount") + assert( + addedRowCount == 0, + s"update-only incremental scan must not return appended rows: $addedRowCount") + } + + /** + * A streaming read of the table delivers the seed rows on first run and the newly inserted row + * after restart, into a destination table. + */ + private def readerWriterStreamAppendCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.stream.append") { table => + val destination = s"${table.name}_s" + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + table.spark.sql(cowCreate(destination, format)) + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-rw").toString + def runStream(): Unit = { + val query = table.spark.readStream + .table(table.name) + .writeStream + .format("iceberg") + .outputMode("append") + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .toTable(destination) + assert(query.awaitTermination(120000), "stream did not finish") + query.stop() + } + + try { + runStream() + assert( + countOf(table.spark, s"SELECT count(*) FROM $destination") == "3", + "initial stream did not deliver the seed") table.spark.sql( s"INSERT INTO ${table.name} VALUES " + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() - - println(s"DIAG incremental.append: added=$addedRowCount") + runStream() assert( - addedRowCount == 1, - s"append incremental scan should contain one row, got $addedRowCount") - }, - cowPreparation(format).test( - "readerWriter.incremental.delete", - "An incremental scan spanning a DELETE-only snapshot returns no rows.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head + countOf(table.spark, s"SELECT count(*) FROM $destination") == "4", + "stream restart did not deliver the appended row") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + } + } + + /** + * An append-only stream restarted after a DELETE snapshot was written fails, with an error + * mentioning delete or overwrite. + */ + private def readerWriterStreamDeleteRejectedCase(format: String): Plan.Case = + cowPreparation(format).test("readerWriter.stream.deleteRejected") { table => + val destination = s"${table.name}_sd" + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + table.spark.sql(cowCreate(destination, format)) + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-rwd").toString + def runStream(): Unit = { + val query = table.spark.readStream + .table(table.name) + .writeStream + .format("iceberg") + .outputMode("append") + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .toTable(destination) + assert(query.awaitTermination(120000), "stream did not finish") + query.stop() + } + + try { + runStream() table.spark.sql( s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() + val exception = Check.intercept[Exception](runStream()) - println(s"DIAG incremental.delete: added=$addedRowCount") + println( + "DIAG stream.afterDelete: " + + s"${exception.getClass.getSimpleName} :: " + + Option(exception.getMessage).getOrElse("").take(140)) assert( - addedRowCount == 0, - s"delete-only incremental scan must not return appended rows: $addedRowCount") - }, - cowPreparation(format).test( - "readerWriter.incremental.overwrite", - "An incremental scan spanning an INSERT OVERWRITE that only removes rows returns no " + - "rows.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT OVERWRITE ${table.name} " + - s"SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage).exists(message => + message.toLowerCase.contains("delete") || + message.toLowerCase.contains("overwrite"))), + "append-only stream should reject a delete snapshot") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + } + } + + /** + * The incremental reads between two snapshots and the structured-streaming reader and writer, on + * three seed rows in the given file format. + */ + def readerWriterIncrementalAndStreamCases(format: String): List[Plan.Case] = + List( + readerWriterIncrementalAppendCase(format), + readerWriterIncrementalDeleteCase(format), + readerWriterIncrementalOverwriteCase(format), + readerWriterIncrementalUpdateCase(format), + readerWriterStreamAppendCase(format), + readerWriterStreamDeleteRejectedCase(format)) + + /** + * A streaming read that resumes after its earliest offset snapshot has been expired fails, with + * an error naming the expired or missing snapshot. + */ + private def hazardStreamExpiredCheckpointCase( + format: String, + basePreparation: TablePreparation[CoreTable.type]): Plan.Case = + basePreparation.test("hazard.stream.expiredCheckpoint") { table => + val destination = s"${table.name}_sink" + table.spark.sql(s"DROP TABLE IF EXISTS $destination") + table.spark.sql(cowCreate(destination, format)) + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-hazard").toString + def runStream(): Unit = { + val query = table.spark.readStream + .table(table.name) + .writeStream .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() + .outputMode("append") + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .toTable(destination) + assert(query.awaitTermination(120000), "stream did not finish") + query.stop() + } + + try { + runStream() + assert( + countOf( + table.spark, + s"SELECT count(*) FROM $destination") == "3", + "initial stream should deliver the seed") - println(s"DIAG incremental.overwrite: added=$addedRowCount") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + runStream() assert( - addedRowCount == 0, - s"overwrite-only incremental scan must not return appended rows: $addedRowCount") - }, - cowPreparation(format).test( - "readerWriter.incremental.update", - "An incremental scan spanning an UPDATE-only snapshot returns no rows.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head + countOf( + table.spark, + s"SELECT count(*) FROM $destination") == "4", + "control restart should deliver one incremental row") + table.spark.sql( - s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + - s"WHERE ${Core.long0.columnName} = 2") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + val exception = Check.intercept[Exception](runStream()) - println(s"DIAG incremental.update: added=$addedRowCount") assert( - addedRowCount == 0, - s"update-only incremental scan must not return appended rows: $addedRowCount") - }, - cowPreparation(format).test( - "readerWriter.stream.append", - "A streaming read of the table delivers the seed rows on first run and the newly " + - "inserted row after restart, into a destination table.") { table => - val destination = s"${table.name}_s" + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage).exists(message => + message.contains("expired or removed") || + message.contains("Cannot load current offset") || + message.contains("Cannot find snapshot"))), + "stream restart should report the expired checkpoint offset") + } finally { table.spark.sql(s"DROP TABLE IF EXISTS $destination") - table.spark.sql(cowCreate(destination, format)) - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-rw").toString - def runStream(): Unit = { - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("iceberg") - .outputMode("append") - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .toTable(destination) - assert(query.awaitTermination(120000), "stream did not finish") - query.stop() - } + } + } + /** + * After expire_snapshots removes a changelog start point, create_changelog_view over that start + * point either throws or reports fewer changes than the table's history holds, and any message it + * throws leaves expiration unnamed. The case covers three start points: an expired snapshot ID, a + * timestamp older than the whole history, and a timestamp inside the expired range. + */ + private def hazardCdcExpiredRangeCase( + basePreparation: TablePreparation[CoreTable.type]): Plan.Case = + basePreparation.test("hazard.cdc.expiredRange") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + val snapshots = snapshotIds(table.spark, table.name) + val firstTimestamp = table.spark + .sql( + s"SELECT committed_at FROM ${table.name}.snapshots " + + "ORDER BY committed_at LIMIT 1") + .collect()(0) + .getTimestamp(0) + val middleTimestamp = table.spark + .sql( + s"SELECT committed_at FROM ${table.name}.snapshots " + + s"WHERE snapshot_id = ${snapshots(1)}") + .collect()(0) + .getTimestamp(0) + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + def changelog( + optionKey: String, + optionValue: String, + trueChangeCount: Long): String = try { - runStream() - assert( - countOf(table.spark, s"SELECT count(*) FROM $destination") == "3", - "initial stream did not deliver the seed") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - runStream() - assert( - countOf(table.spark, s"SELECT count(*) FROM $destination") == "4", - "stream restart did not deliver the appended row") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - } - }, - cowPreparation(format).test( - "readerWriter.stream.deleteRejected", - "An append-only stream restarted after a DELETE snapshot was written fails, with an " + - "error mentioning delete or overwrite.") { table => - val destination = s"${table.name}_sd" - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - table.spark.sql(cowCreate(destination, format)) - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-rwd").toString - def runStream(): Unit = { - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("iceberg") - .outputMode("append") - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .toTable(destination) - assert(query.awaitTermination(120000), "stream did not finish") - query.stop() + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('$optionKey', '$optionValue'))") + .collect()(0) + .getString(0) + val actualChangeCount = table.spark + .sql(s"SELECT count(*) FROM $view") + .collect()(0) + .getLong(0) + if (actualChangeCount < trueChangeCount) { + s"SILENT under-report: $actualChangeCount of " + + s"$trueChangeCount true changes" + } else { + s"FULL: $actualChangeCount of $trueChangeCount" + } + } catch { + case exception: Throwable => + s"TYPED: ${exception.getClass.getSimpleName} :: " + + Option(exception.getMessage).getOrElse("").take(140) } - - try { - runStream() - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val exception = Check.intercept[Exception](runStream()) - - println( - "DIAG stream.afterDelete: " + - s"${exception.getClass.getSimpleName} :: " + - Option(exception.getMessage).getOrElse("").take(140)) + val explicitSnapshotOutcome = + changelog("start-snapshot-id", snapshots.head.toString, 5) + val beforeHistoryOutcome = + changelog( + "start-timestamp", + (firstTimestamp.getTime - 1000).toString, + 5) + val middleHistoryOutcome = + changelog( + "start-timestamp", + (middleTimestamp.getTime - 1).toString, + 2) + + println(s"DIAG cdc.explicitExpiredId: $explicitSnapshotOutcome") + println(s"DIAG cdc.tsBeforeHistory: $beforeHistoryOutcome") + println(s"DIAG cdc.tsMidExpired: $middleHistoryOutcome") + Seq( + "explicitId" -> explicitSnapshotOutcome, + "tsBeforeHistory" -> beforeHistoryOutcome, + "tsMidExpired" -> middleHistoryOutcome).foreach { + case (label, outcome) => assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage).exists(message => - message.toLowerCase.contains("delete") || - message.toLowerCase.contains("overwrite"))), - "append-only stream should reject a delete snapshot") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - } - }) - + !outcome.startsWith("FULL"), + s"expired-lineage changelog returned full truth for $label") + assert( + !outcome.toLowerCase.contains("expir"), + s"expired-lineage message now names expiration for $label") + } + } - // The hazards a reader or a consumer meets when maintenance or a schema change lands underneath - // it. Every case starts from a plain copy-on-write table. + /** + * The hazards a reader or a consumer meets when maintenance lands underneath it. Every case + * starts from three seed rows in a copy-on-write table in the given file format. + */ def hazardReaderCases(format: String): List[Plan.Case] = { val basePreparation = TablePreparation( format, TableTest(Core) .sql("create")(table => cowCreate(table, format))() - .insert(3)(), - description = s"Three seed rows in a copy-on-write $format table.") + .insert(3)()) List( - basePreparation.test( - "hazard.stream.expiredCheckpoint", - "A streaming read that resumes after its earliest offset snapshot has been expired fails, " + - "with an error naming the expired or missing snapshot.") { table => - val destination = s"${table.name}_sink" - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - table.spark.sql(cowCreate(destination, format)) - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-hazard").toString - def runStream(): Unit = { - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("iceberg") - .outputMode("append") - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .toTable(destination) - assert(query.awaitTermination(120000), "stream did not finish") - query.stop() - } - - try { - runStream() - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $destination") == "3", - "initial stream should deliver the seed") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - runStream() - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $destination") == "4", - "control restart should deliver one incremental row") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - val exception = Check.intercept[Exception](runStream()) - - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage).exists(message => - message.contains("expired or removed") || - message.contains("Cannot load current offset") || - message.contains("Cannot find snapshot"))), - "stream restart should report the expired checkpoint offset") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - } - }, - basePreparation.test( - "hazard.cdc.expiredRange", - "A changelog view whose start point has been removed by snapshot expiration does not " + - "silently under-report the true change count or return successfully; it fails with a " + - "typed error.") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - val snapshots = snapshotIds(table.spark, table.name) - val firstTimestamp = table.spark - .sql( - s"SELECT committed_at FROM ${table.name}.snapshots " + - "ORDER BY committed_at LIMIT 1") - .collect()(0) - .getTimestamp(0) - val middleTimestamp = table.spark - .sql( - s"SELECT committed_at FROM ${table.name}.snapshots " + - s"WHERE snapshot_id = ${snapshots(1)}") - .collect()(0) - .getTimestamp(0) - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - def changelog( - optionKey: String, - optionValue: String, - trueChangeCount: Long): String = - try { - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('$optionKey', '$optionValue'))") - .collect()(0) - .getString(0) - val actualChangeCount = table.spark - .sql(s"SELECT count(*) FROM $view") - .collect()(0) - .getLong(0) - if (actualChangeCount < trueChangeCount) { - s"SILENT under-report: $actualChangeCount of " + - s"$trueChangeCount true changes" - } else { - s"FULL: $actualChangeCount of $trueChangeCount" - } - } catch { - case exception: Throwable => - s"TYPED: ${exception.getClass.getSimpleName} :: " + - Option(exception.getMessage).getOrElse("").take(140) - } - val explicitSnapshotOutcome = - changelog("start-snapshot-id", snapshots.head.toString, 5) - val beforeHistoryOutcome = - changelog( - "start-timestamp", - (firstTimestamp.getTime - 1000).toString, - 5) - val middleHistoryOutcome = - changelog( - "start-timestamp", - (middleTimestamp.getTime - 1).toString, - 2) - - println(s"DIAG cdc.explicitExpiredId: $explicitSnapshotOutcome") - println(s"DIAG cdc.tsBeforeHistory: $beforeHistoryOutcome") - println(s"DIAG cdc.tsMidExpired: $middleHistoryOutcome") - Seq( - "explicitId" -> explicitSnapshotOutcome, - "tsBeforeHistory" -> beforeHistoryOutcome, - "tsMidExpired" -> middleHistoryOutcome).foreach { - case (label, outcome) => - assert( - !outcome.startsWith("FULL"), - s"expired-lineage changelog returned full truth for $label") - assert( - !outcome.toLowerCase.contains("expir"), - s"expired-lineage message now names expiration for $label") - } - }) + hazardStreamExpiredCheckpointCase(format, basePreparation), + hazardCdcExpiredRangeCase(basePreparation)) } - // The hazard an explicit-column writer meets after a column is added. + /** + * An explicit-column INSERT that worked before ADD COLUMN is rejected afterward, with an error + * naming the new column. + */ + private def hazardAddColumnBreaksWritersCase( + basePreparation: TablePreparation[CoreTable.type]): Plan.Case = + basePreparation.test("hazard.addColumn.breaksWriters") { table => + val allColumns = + Core.tableColumns.map(_.columnName).mkString(", ") + val writerStatement = + s"INSERT INTO ${table.name} ($allColumns) VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')" + table.spark.sql(writerStatement) + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "4", + "explicit-column writer should work before schema evolution") + + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + val exception = Check.intercept[AnalysisException]( + table.spark.sql(writerStatement)) + assert( + exception.getMessage.contains("extra_col") && + (exception.getMessage.contains("CANNOT_FIND_DATA") || + exception.getMessage.toLowerCase.contains("cannot find data")), + "pre-evolution explicit-column writer should fail after ADD COLUMN") + } + + /** + * The hazard an explicit-column writer meets after a column is added. The case starts from three + * seed rows in a copy-on-write table in the given file format. + */ def hazardWriterCases(format: String): List[Plan.Case] = { val basePreparation = TablePreparation( format, TableTest(Core) .sql("create")(table => cowCreate(table, format))() - .insert(3)(), - description = s"Three seed rows in a copy-on-write $format table.") + .insert(3)()) List( - basePreparation.test( - "hazard.addColumn.breaksWriters", - "An explicit-column INSERT that worked before ADD COLUMN is rejected afterward, with an " + - "error naming the new column.") { table => - val allColumns = - Core.tableColumns.map(_.columnName).mkString(", ") - val writerStatement = - s"INSERT INTO ${table.name} ($allColumns) VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')" - table.spark.sql(writerStatement) - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "4", - "explicit-column writer should work before schema evolution") - - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - val exception = Check.intercept[AnalysisException]( - table.spark.sql(writerStatement)) - assert( - exception.getMessage.contains("extra_col") && - (exception.getMessage.contains("CANNOT_FIND_DATA") || - exception.getMessage.toLowerCase.contains("cannot find data")), - "pre-evolution explicit-column writer should fail after ADD COLUMN") - }) + hazardAddColumnBreaksWritersCase(basePreparation)) } - // While a table is locked through the REST lock endpoint, every maintenance commit is blocked, not - // just table replacement. + /** + * While a table is REST-locked, an expire_snapshots call is rejected and snapshots keep + * accumulating. After the lock is deleted, expire_snapshots succeeds and the snapshot count + * drops, so the lock blocks every maintenance commit while it is held. + */ def hazardLockStarvesMaintenance(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_lockmaint" @@ -585,13 +644,11 @@ trait HazardReaderWriterScenarios extends ScenarioKit { } } + /** The hazard cases the embedded REST server drives. */ val hazardContextCases: List[Plan.Case] = List( Plan.Case( "hazard.lock.starvesMaintenance @ embedded", - hazardLockStarvesMaintenance, - description = "While a table is REST-locked, an expire_snapshots call is rejected and " + - "snapshots keep accumulating; after unlocking, expire_snapshots succeeds and the snapshot " + - "count drops.")) + hazardLockStarvesMaintenance)) } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala index e025a3130..b1b325f02 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala @@ -16,13 +16,34 @@ import scala.util.control.NonFatal trait ImplementationPinScenarios extends ScenarioKit { import Rows._ - // OpenHouse delegates table-data encryption to an external KMS plugin. The OSS build never wires - // a KeyManagementClient into the catalog, so customer tables use the default - // PlaintextEncryptionManager and data is written unencrypted. A Parquet file's footer magic bytes - // are "PAR1" when unencrypted and "PARE" under modular encryption regardless of compression, so - // this case checks that magic value to confirm the OSS write path produces plaintext data files. - // An off-the-shelf KMS plugin alone would not change this result, because nothing in the - // OpenHouse write path invokes the encryption hook without that wiring. + /** + * A data file's Parquet footer magic bytes are the plaintext PAR1 marker, confirming OSS writes + * table data in plaintext. OpenHouse delegates table-data encryption to an external KMS plugin + * and the OSS build wires no KeyManagementClient into the catalog, so tables use the default + * PlaintextEncryptionManager. A Parquet footer reads PAR1 for plaintext and PARE under modular + * encryption regardless of compression, so that magic value settles which path wrote the file. + */ + private def surfacePinDataPlaintextCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("surface.pin.dataPlaintext") { table => + val dataFilePath = table.spark + .sql(s"SELECT file_path FROM ${table.name}.data_files LIMIT 1") + .collect()(0) + .getString(0) + .stripPrefix("file:") + val bytes = java.nio.file.Files.readAllBytes( + java.nio.file.Paths.get(dataFilePath)) + + assert( + bytes.length >= 8, + s"data file is too small to inspect: ${bytes.length} bytes") + val footerMagic = new String(bytes.takeRight(4), "US-ASCII") + assert( + footerMagic == "PAR1", + s"expected plaintext Parquet footer PAR1, got $footerMagic") + } + + /** The encryption pin, starting from three seed rows in a parquet table. */ lazy val encryptionPinCases: List[Plan.Case] = { val preparation = TablePreparation( "parquet", @@ -30,29 +51,9 @@ trait ImplementationPinScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + "TBLPROPERTIES ('write.format.default'='parquet')")() - .insert(3)(), - description = "Three seed rows in a parquet table.") + .insert(3)()) List( - preparation.test( - "surface.pin.dataPlaintext", - "A data file's Parquet footer magic bytes are the unencrypted PAR1 marker, confirming " + - "OSS writes table data in plaintext.") { table => - val dataFilePath = table.spark - .sql(s"SELECT file_path FROM ${table.name}.data_files LIMIT 1") - .collect()(0) - .getString(0) - .stripPrefix("file:") - val bytes = java.nio.file.Files.readAllBytes( - java.nio.file.Paths.get(dataFilePath)) - - assert( - bytes.length >= 8, - s"data file is too small to inspect: ${bytes.length} bytes") - val footerMagic = new String(bytes.takeRight(4), "US-ASCII") - assert( - footerMagic == "PAR1", - s"expected plaintext Parquet footer PAR1, got $footerMagic") - }) + surfacePinDataPlaintextCase(preparation)) } } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala index 10dfe1084..f4aa03212 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala @@ -16,6 +16,138 @@ import scala.util.control.NonFatal trait InteractionScenarios extends ScenarioKit { import Rows._ + /** + * After ADD COLUMN and an insert into the new column, time travel to the pre-DDL snapshot reads + * the old schema with 3 rows, while a current read sees the new column. + */ + private def interactDdlTtAfterAddColumnCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("interact.ddl.ttAfterAddColumn") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + val currentColumns = table.spark + .sql(s"SELECT * FROM ${table.name} LIMIT 1") + .columns + .toSeq + val historicalColumns = table.spark + .sql( + s"SELECT * FROM ${table.name} " + + s"VERSION AS OF $seedSnapshotId LIMIT 1") + .columns + .toSeq + val historicalRowCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF $seedSnapshotId") + .collect()(0) + .getLong(0) + + assert( + currentColumns.contains("extra_col"), + s"current read is missing the evolved column: $currentColumns") + assert( + !historicalColumns.contains("extra_col") && + historicalColumns.size == Core.tableColumns.size, + s"time travel should use the snapshot schema: $historicalColumns") + assert( + historicalRowCount == 3, + s"pre-DDL snapshot should contain 3 rows, got $historicalRowCount") + } + + /** + * Rolling back to the pre-DDL snapshot after ADD COLUMN and an insert keeps the evolved schema, + * restores 3 rows reading null for the new column, and the table still accepts writes into that + * column. + */ + private def interactDdlRestoreAfterAddColumnCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("interact.ddl.restoreAfterAddColumn") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $seedSnapshotId)") + val currentColumns = table.spark + .sql(s"SELECT * FROM ${table.name} LIMIT 1") + .columns + .toSeq + val currentRowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val nonNullEvolvedValueCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + "WHERE extra_col IS NOT NULL") + .collect()(0) + .getLong(0) + + assert( + currentColumns.contains("extra_col"), + s"rollback should retain the evolved schema: $currentColumns") + assert( + currentRowCount == 3, + s"rollback should restore 3 rows, got $currentRowCount") + assert( + nonNullEvolvedValueCount == 0, + "rolled-back rows should read the evolved column as null") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert10") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 4, + "the rolled-back table should accept evolved-schema writes") + } + + /** + * DROP COLUMN on a column that holds data is rejected, the column's data remains readable, and + * the table remains writable. + */ + private def interactDdlDropColAfterDataCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("interact.ddl.dropColAfterData") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} DROP COLUMN extra_col")) + + assert( + exception.getMessage.contains("not found in newSchema"), + s"drop rejection message changed: ${exception.getMessage.take(200)}") + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} WHERE extra_col = 42") + .collect()(0) + .getLong(0) == 1, + "rejected drop should leave the column data readable") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert10") + assert( + table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) == 5, + "rejected drop should leave the table writable") + } + + /** + * The DDL interactions. Every case starts from three seed rows in a table in the given file + * format. + */ def interactionDdlCases(format: String): List[Plan.Case] = { val preparation = TablePreparation( format, @@ -23,128 +155,61 @@ trait InteractionScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = s"Three seed rows in a $format table.") + .insert(3)()) List( - preparation.test( - "interact.ddl.ttAfterAddColumn", - "After ADD COLUMN and an insert into the new column, time travel to the pre-DDL snapshot " + - "reads the old schema with 3 rows, while a current read sees the new column.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - val currentColumns = table.spark - .sql(s"SELECT * FROM ${table.name} LIMIT 1") - .columns - .toSeq - val historicalColumns = table.spark - .sql( - s"SELECT * FROM ${table.name} " + - s"VERSION AS OF $seedSnapshotId LIMIT 1") - .columns - .toSeq - val historicalRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF $seedSnapshotId") - .collect()(0) - .getLong(0) - - assert( - currentColumns.contains("extra_col"), - s"current read is missing the evolved column: $currentColumns") - assert( - !historicalColumns.contains("extra_col") && - historicalColumns.size == Core.tableColumns.size, - s"time travel should use the snapshot schema: $historicalColumns") - assert( - historicalRowCount == 3, - s"pre-DDL snapshot should contain 3 rows, got $historicalRowCount") - }, - preparation.test( - "interact.ddl.restoreAfterAddColumn", - "Rolling back to the pre-DDL snapshot after ADD COLUMN and an insert keeps the evolved " + - "schema, restores 3 rows reading null for the new column, and the table still accepts " + - "writes into that column.") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $seedSnapshotId)") - val currentColumns = table.spark - .sql(s"SELECT * FROM ${table.name} LIMIT 1") - .columns - .toSeq - val currentRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val nonNullEvolvedValueCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - "WHERE extra_col IS NOT NULL") - .collect()(0) - .getLong(0) - - assert( - currentColumns.contains("extra_col"), - s"rollback should retain the evolved schema: $currentColumns") - assert( - currentRowCount == 3, - s"rollback should restore 3 rows, got $currentRowCount") - assert( - nonNullEvolvedValueCount == 0, - "rolled-back rows should read the evolved column as null") + interactDdlTtAfterAddColumnCase(preparation), + interactDdlRestoreAfterAddColumnCase(preparation), + interactDdlDropColAfterDataCase(preparation)) + } - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert10") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "the rolled-back table should accept evolved-schema writes") - }, - preparation.test( - "interact.ddl.dropColAfterData", - "DROP COLUMN on a column that holds data is rejected, the column's data remains readable, " + - "and the table remains writable.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} DROP COLUMN extra_col")) - - assert( - exception.getMessage.contains("not found in newSchema"), - s"drop rejection message changed: ${exception.getMessage.take(200)}") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} WHERE extra_col = 42") - .collect()(0) - .getLong(0) == 1, - "rejected drop should leave the column data readable") + /** + * Compacting a table after an ADD COLUMN and inserts into the new column preserves all rows, the + * new column's non-null values, and null for rows written before the column was added. + */ + private def interactMaintCompactEvolvedCase( + basePreparation: TablePreparation[CoreTable.type]): Plan.Case = + basePreparation.test("interact.maint.compactEvolved") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert10") + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}')") + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + val evolvedValueCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + "WHERE extra_col IN (42, 43)") + .collect()(0) + .getLong(0) + val nullValueCount = table.spark + .sql( + s"SELECT count(*) FROM ${table.name} WHERE extra_col IS NULL") + .collect()(0) + .getLong(0) - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert10") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 5, - "rejected drop should leave the table writable") - }) - } + assert( + rowCount == 5, + s"compaction should preserve 5 rows, got $rowCount") + assert( + evolvedValueCount == 2, + s"compaction should preserve two evolved values, got $evolvedValueCount") + assert( + nullValueCount == 3, + s"pre-evolution rows should remain null, got $nullValueCount") + } + /** + * The maintenance interactions. The case starts from three seed rows in a table in the given + * file format. + */ def interactionMiscellaneousCases( format: String): List[Plan.Case] = { val basePreparation = TablePreparation( @@ -153,50 +218,9 @@ trait InteractionScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = s"Three seed rows in a $format table.") - + .insert(3)()) List( - basePreparation.test( - "interact.maint.compactEvolved", - "Compacting a table after an ADD COLUMN and inserts into the new column preserves all " + - "rows, the new column's non-null values, and null for rows written before the column " + - "was added.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert10") - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}')") - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val evolvedValueCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - "WHERE extra_col IN (42, 43)") - .collect()(0) - .getLong(0) - val nullValueCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} WHERE extra_col IS NULL") - .collect()(0) - .getLong(0) - - assert( - rowCount == 5, - s"compaction should preserve 5 rows, got $rowCount") - assert( - evolvedValueCount == 2, - s"compaction should preserve two evolved values, got $evolvedValueCount") - assert( - nullValueCount == 3, - s"pre-evolution rows should remain null, got $nullValueCount") - }) + interactMaintCompactEvolvedCase(basePreparation)) } } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala index 2521677ab..41bd7e848 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala @@ -13,176 +13,206 @@ import scala.util.control.NonFatal trait MaintControlScenarios extends ScenarioKit { import Rows._ - // Time travel and restore/rollback. - // A two-snapshot base: seed 3 rows (snapshot A), then insert 2 more (snapshot B). Format is a - // parameter so each case below runs against every supported file format. + /** + * A five-row table across two snapshots in the given file format: a 3-row seed commit, then a + * 2-row insert committed at a later timestamp. Time travel, restore and maintenance all start + * from this state. + */ + private def twoSnapshotPreparation(format: String): TablePreparation[CoreTable.type] = + TablePreparation(format, coreTwoSnapshots(format)) + + /** + * VERSION AS OF the first snapshot ID reads the 3 rows the seed commit wrote, and VERSION AS OF + * the second reads all 5 rows. + */ + private def timeTravelVersionAsOfCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("timeTravel.versionAsOf") { table => + val snapshots = snapshotIds(table.spark, table.name) + + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF ${snapshots(0)}") + .collect()(0) + .getLong(0) == 3) + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"VERSION AS OF ${snapshots(1)}") + .collect()(0) + .getLong(0) == 5) + } + + /** TIMESTAMP AS OF the first commit's time reads the 3 rows that commit wrote. */ + private def timeTravelTimestampAsOfCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("timeTravel.timestampAsOf") { table => + val firstCommitTimestamp = table.spark + .sql( + s"SELECT CAST(committed_at AS STRING) FROM ${table.name}.snapshots " + + "ORDER BY committed_at LIMIT 1") + .collect()(0) + .getString(0) + + assert( + table.spark + .sql( + s"SELECT count(*) FROM ${table.name} " + + s"TIMESTAMP AS OF '$firstCommitTimestamp'") + .collect()(0) + .getLong(0) == 3) + } + + /** + * The snapshots and history metadata tables each report the table's 2 snapshots, and the files + * and manifests metadata tables report at least 1 row. + */ + private def timeTravelMetadataTablesCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("timeTravel.metadataTables") { table => + def metadataRowCount(metadataTable: String): Long = + table.spark + .sql( + s"SELECT count(*) FROM ${table.name}.$metadataTable") + .collect()(0) + .getLong(0) + + assert(metadataRowCount("snapshots") == 2) + assert(metadataRowCount("history") == 2) + assert( + metadataRowCount("files") >= 1 && + metadataRowCount("manifests") >= 1) + } + + /** An incremental read spanning both snapshots returns the 2 rows the second commit added. */ + private def timeTravelIncrementalReadCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("timeTravel.incrementalRead") { table => + val snapshots = snapshotIds(table.spark, table.name) + val addedRowCount = table.spark.read + .format("iceberg") + .option("start-snapshot-id", snapshots(0)) + .option("end-snapshot-id", snapshots(1)) + .load(table.name) + .count() + + assert(addedRowCount == 2) + } + /** Time travel across both snapshots of the two-snapshot table, in parquet and in orc. */ val timeTravelCases: List[Plan.Case] = List("parquet", "orc").flatMap { format => - val preparation = TablePreparation( - format, - coreTwoSnapshots(format), - description = s"Five seed rows across two snapshots in a $format table.") + val preparation = twoSnapshotPreparation(format) List( - preparation.test( - "timeTravel.versionAsOf", - "VERSION AS OF the first snapshot ID reads 3 rows and VERSION AS OF the second reads " + - "5 rows.") { table => - val snapshots = snapshotIds(table.spark, table.name) - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF ${snapshots(0)}") - .collect()(0) - .getLong(0) == 3) - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF ${snapshots(1)}") - .collect()(0) - .getLong(0) == 5) - }, - preparation.test( - "timeTravel.timestampAsOf", - "TIMESTAMP AS OF the first commit's time reads that snapshot's 3 rows.") { table => - val firstCommitTimestamp = table.spark - .sql( - s"SELECT CAST(committed_at AS STRING) FROM ${table.name}.snapshots " + - "ORDER BY committed_at LIMIT 1") - .collect()(0) - .getString(0) - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"TIMESTAMP AS OF '$firstCommitTimestamp'") - .collect()(0) - .getLong(0) == 3) - }, - preparation.test( - "timeTravel.metadataTables", - "The snapshots and history metadata tables each report 2 rows, and the files and " + - "manifests metadata tables report at least 1 row.") { table => - def metadataRowCount(metadataTable: String): Long = - table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.$metadataTable") - .collect()(0) - .getLong(0) - - assert(metadataRowCount("snapshots") == 2) - assert(metadataRowCount("history") == 2) - assert( - metadataRowCount("files") >= 1 && - metadataRowCount("manifests") >= 1) - }, - preparation.test( - "timeTravel.incrementalRead", - "An incremental read spanning the two seed snapshots returns exactly the 2 rows added " + - "by the second snapshot.") { table => - val snapshots = snapshotIds(table.spark, table.name) - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", snapshots(0)) - .option("end-snapshot-id", snapshots(1)) - .load(table.name) - .count() - - assert(addedRowCount == 2) - }) + timeTravelVersionAsOfCase(preparation), + timeTravelTimestampAsOfCase(preparation), + timeTravelMetadataTablesCase(preparation), + timeTravelIncrementalReadCase(preparation)) + } + + /** rollback_to_snapshot to the first snapshot restores the 3 rows the seed commit wrote. */ + private def restoreRollbackToSnapshotCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("restore.rollbackToSnapshot") { table => + val firstSnapshotId = + snapshotIds(table.spark, table.name).head + + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $firstSnapshotId)") + + assert(table.rows.size == 3) + } + + /** set_current_snapshot to the first snapshot restores the 3 rows the seed commit wrote. */ + private def restoreSetCurrentSnapshotCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("restore.setCurrentSnapshot") { table => + val firstSnapshotId = + snapshotIds(table.spark, table.name).head + + table.spark.sql( + "CALL openhouse.system.set_current_snapshot(" + + s"'${catalogRelative(table.name)}', $firstSnapshotId)") + + assert(table.rows.size == 3) } + /** Restore back to the seed snapshot of the two-snapshot table, in parquet and in orc. */ val restoreRollbackCases: List[Plan.Case] = List("parquet", "orc").flatMap { format => - val preparation = TablePreparation( - format, - coreTwoSnapshots(format), - description = s"Five seed rows across two snapshots in a $format table.") + val preparation = twoSnapshotPreparation(format) List( - preparation.test( - "restore.rollbackToSnapshot", - "rollback_to_snapshot to the first snapshot restores the table to its 3-row state.") { table => - val firstSnapshotId = - snapshotIds(table.spark, table.name).head - - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $firstSnapshotId)") - - assert(table.rows.size == 3) - }, - preparation.test( - "restore.setCurrentSnapshot", - "set_current_snapshot to the first snapshot restores the table to its 3-row state.") { table => - val firstSnapshotId = - snapshotIds(table.spark, table.name).head - - table.spark.sql( - "CALL openhouse.system.set_current_snapshot(" + - s"'${catalogRelative(table.name)}', $firstSnapshotId)") - - assert(table.rows.size == 3) - }) + restoreRollbackToSnapshotCase(preparation), + restoreSetCurrentSnapshotCase(preparation)) + } + + /** + * expire_snapshots with retain_last=1 removes the seed snapshot and leaves all 5 current rows + * unchanged. + */ + private def maintenanceExpireSnapshotsCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("maintenance.expireSnapshots") { table => + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + assert( + table.rows.size == 5, + "expire_snapshots changed the current data") + assert( + table.snapshotCount < table.preparedSnapshotCount, + "expire_snapshots did not remove a snapshot: " + + s"${table.preparedSnapshotCount} -> ${table.snapshotCount}") + } + + /** rewrite_data_files compacts the data files and leaves all 5 rows unchanged. */ + private def maintenanceRewriteDataFilesCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("maintenance.rewriteDataFiles") { table => + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}')") + + assert(table.rows.size == 5, "compaction changed rows") + } + + /** remove_orphan_files leaves all 5 rows unchanged. */ + private def maintenanceRemoveOrphanFilesCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("maintenance.removeOrphanFiles") { table => + table.spark.sql( + "CALL openhouse.system.remove_orphan_files(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2020-01-01 00:00:00')") + + assert(table.rows.size == 5, "orphan removal changed rows") } + /** The maintenance procedures run over the two-snapshot table, in parquet and in orc. */ val maintenanceCases: List[Plan.Case] = List("parquet", "orc").flatMap { format => - val preparation = TablePreparation( - format, - coreTwoSnapshots(format), - description = s"Five seed rows across two snapshots in a $format table.") + val preparation = twoSnapshotPreparation(format) List( - preparation.test( - "maintenance.expireSnapshots", - "expire_snapshots with retain_last=1 removes an old snapshot and leaves the current 5 " + - "rows unchanged.") { table => - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - assert( - table.rows.size == 5, - "expire_snapshots changed the current data") - assert( - table.snapshotCount < table.preparedSnapshotCount, - "expire_snapshots did not remove a snapshot: " + - s"${table.preparedSnapshotCount} -> ${table.snapshotCount}") - }, - preparation.test( - "maintenance.rewriteDataFiles", - "rewrite_data_files compacts the table's data files while preserving all 5 rows.") { table => - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}')") - - assert(table.rows.size == 5, "compaction changed rows") - }, - preparation.test( - "maintenance.removeOrphanFiles", - "remove_orphan_files leaves all 5 rows unchanged.") { table => - table.spark.sql( - "CALL openhouse.system.remove_orphan_files(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2020-01-01 00:00:00')") - - assert(table.rows.size == 5, "orphan removal changed rows") - }) + maintenanceExpireSnapshotsCase(preparation), + maintenanceRewriteDataFilesCase(preparation), + maintenanceRemoveOrphanFilesCase(preparation)) } - // Control-plane (REST) operations with no SQL surface, driven through the embedded server's - // HTTP API. Lock enforcement: POST /lock is a real public endpoint; a subsequent Spark mutation - // is rejected server-side with LOCKED_TABLE_OPERATION, and DELETE /lock restores mutability. The - // embedded server runs the real TablesController and TablesServiceImpl, so this exercises the - // production REST path. + /** + * POSTing a table lock causes a following Spark UPDATE to be rejected server-side with + * LOCKED_TABLE_OPERATION, and DELETEing the lock lets a later UPDATE apply. The lock endpoint has + * no SQL surface, so the case drives it over HTTP against the embedded server, which runs the + * same TablesController and TablesServiceImpl as production. + */ def controlLockEnforcement(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_lock" @@ -205,13 +235,11 @@ trait MaintControlScenarios extends ScenarioKit { } finally spark.sql(s"DROP TABLE IF EXISTS $table") } + /** The control-plane cases, each driven over HTTP against the embedded server. */ val controlPlaneCases: List[Plan.Case] = List( Plan.Case( "control.lock.enforcement @ embedded", - controlLockEnforcement, - description = "POSTing a table lock causes a subsequent UPDATE to be rejected, and " + - "DELETEing the lock allows a following UPDATE to apply.")) - + controlLockEnforcement)) } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala index a81a42bd3..d5bba4048 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala @@ -15,75 +15,94 @@ trait NegativeDdlScenarios extends ScenarioKit { private val S = CoreTable.string0.columnName - val negativeCases: List[Plan.Case] = - preparedCoreFormats.flatMap { preparation => - List( - preparation.test( - "negative.nonExistentColumn", - "DELETE with a WHERE clause on a nonexistent column is rejected with an " + - "AnalysisException naming that column.") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"DELETE FROM ${table.name} WHERE no_such_column = 1")) - - assert(exception.getMessage.contains("no_such_column")) - }, - preparation.test( - "negative.nonDeterministicDelete", - "DELETE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an " + - "AnalysisException about determinism.") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"DELETE FROM ${table.name} WHERE rand() < 0.5")) - - assert( - exception.getMessage.toLowerCase.contains("deterministic")) - }, - preparation.test( - "negative.nonDeterministicUpdate", - "UPDATE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an " + - "AnalysisException about determinism.") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"UPDATE ${table.name} SET $S = 'x' WHERE rand() < 0.5")) - - assert( - exception.getMessage.toLowerCase.contains("deterministic")) - }, - preparation.test( - "negative.insertArity", - "INSERT INTO with too few values for the table's columns is rejected with an " + - "AnalysisException about the missing data columns.") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"INSERT INTO ${table.name} VALUES (CAST(1 AS BIGINT), 1)")) - - assert( - exception.getMessage.toLowerCase.contains( - "not enough data columns")) - }, - preparation.test( - "negative.mergeConflictingUpdates", - "A MERGE whose UPDATE SET assigns the same target column twice is rejected with an " + - "AnalysisException about multiple assignments.") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"""MERGE INTO ${table.name} target USING ( + /** + * DELETE with a WHERE clause on a nonexistent column is rejected with an AnalysisException + * naming that column. + */ + private def negativeNonExistentColumnCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("negative.nonExistentColumn") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"DELETE FROM ${table.name} WHERE no_such_column = 1")) + + assert(exception.getMessage.contains("no_such_column")) + } + + /** + * DELETE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an + * AnalysisException about determinism. + */ + private def negativeNonDeterministicDeleteCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("negative.nonDeterministicDelete") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"DELETE FROM ${table.name} WHERE rand() < 0.5")) + + assert( + exception.getMessage.toLowerCase.contains("deterministic")) + } + + /** + * UPDATE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an + * AnalysisException about determinism. + */ + private def negativeNonDeterministicUpdateCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("negative.nonDeterministicUpdate") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"UPDATE ${table.name} SET $S = 'x' WHERE rand() < 0.5")) + + assert( + exception.getMessage.toLowerCase.contains("deterministic")) + } + + /** + * INSERT INTO with too few values for the table's columns is rejected with an AnalysisException + * about the missing data columns. + */ + private def negativeInsertArityCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("negative.insertArity") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"INSERT INTO ${table.name} VALUES (CAST(1 AS BIGINT), 1)")) + + assert( + exception.getMessage.toLowerCase.contains( + "not enough data columns")) + } + + /** + * A MERGE whose UPDATE SET assigns the same target column twice is rejected with an + * AnalysisException about multiple assignments. + */ + private def negativeMergeConflictingUpdatesCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("negative.mergeConflictingUpdates") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"""MERGE INTO ${table.name} target USING ( SELECT * FROM VALUES (CAST(2 AS BIGINT)) AS source($L) ) source ON target.$L = source.$L WHEN MATCHED THEN UPDATE SET target.$S = 'a', target.$S = 'b'""")) - assert(exception.getMessage.contains("Multiple assignments")) - }, - preparation.test( - "negative.mergeCardinalityViolation", - "A MERGE whose source has two rows matching the same target row fails with a " + - "cardinality-violation error naming the multi-row match.") { table => - val exception = Check.intercept[Exception]( - table.spark.sql( - s"""MERGE INTO ${table.name} target USING ( + assert(exception.getMessage.contains("Multiple assignments")) + } + + /** + * A MERGE whose source has two rows matching the same target row fails with a + * cardinality-violation error naming the multi-row match. + */ + private def negativeMergeCardinalityViolationCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("negative.mergeCardinalityViolation") { table => + val exception = Check.intercept[Exception]( + table.spark.sql( + s"""MERGE INTO ${table.name} target USING ( SELECT * FROM VALUES (CAST(2 AS BIGINT), 'a'), (CAST(2 AS BIGINT), 'b') @@ -92,73 +111,171 @@ trait NegativeDdlScenarios extends ScenarioKit { ON target.$L = source.$L WHEN MATCHED THEN UPDATE SET target.$S = source.$S""")) - assert( - Exceptions.causeChain(exception).exists { cause => - Option(cause.getMessage).exists( - _.contains("matched a single row from the target table")) - }, - "expected a MERGE cardinality-violation message, got: " + - exception.getMessage) + assert( + Exceptions.causeChain(exception).exists { cause => + Option(cause.getMessage).exists( + _.contains("matched a single row from the target table")) }, - preparation.test( - "negative.partitionByNonExistent", - "CREATE TABLE PARTITIONED BY a nonexistent column is rejected with an " + - "AnalysisException naming that column, and no scratch table is left behind.") { table => - val scratchTable = table.name + "_x" - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"CREATE TABLE $scratchTable ($columnDefinitions) " + - s"USING $dataSource PARTITIONED BY (no_such_column) " + - s"TBLPROPERTIES ('write.format.default'='${preparation.label}')")) - - table.spark.sql(s"DROP TABLE IF EXISTS $scratchTable") - assert(exception.getMessage.contains("no_such_column")) - }) + "expected a MERGE cardinality-violation message, got: " + + exception.getMessage) + } + + /** + * CREATE TABLE PARTITIONED BY a nonexistent column is rejected with an AnalysisException naming + * that column, and no scratch table is left behind. + */ + private def negativePartitionByNonExistentCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("negative.partitionByNonExistent") { table => + val scratchTable = table.name + "_x" + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"CREATE TABLE $scratchTable ($columnDefinitions) " + + s"USING $dataSource PARTITIONED BY (no_such_column) " + + s"TBLPROPERTIES ('write.format.default'='${preparation.label}')")) + + table.spark.sql(s"DROP TABLE IF EXISTS $scratchTable") + assert(exception.getMessage.contains("no_such_column")) } + /** The rejected DML statements, on the preparedCoreFormats preparations. */ + val negativeCases: List[Plan.Case] = + preparedCoreFormats.flatMap { preparation => + List( + negativeNonExistentColumnCase(preparation), + negativeNonDeterministicDeleteCase(preparation), + negativeNonDeterministicUpdateCase(preparation), + negativeInsertArityCase(preparation), + negativeMergeConflictingUpdatesCase(preparation), + negativeMergeCardinalityViolationCase(preparation), + negativePartitionByNonExistentCase(preparation)) + } + + /** + * ALTER TABLE DROP COLUMN is rejected with a BadRequestException naming the column that would be + * dropped. + */ + private def ddlNegDropColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.neg.dropColumn") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} DROP COLUMN ${Core.int0.columnName}")) + + assert( + exception.getMessage.contains("not found in newSchema"), + s"unexpected message: ${exception.getMessage.take(160)}") + assert( + exception.getMessage.contains(Core.int0.columnName), + s"message should name the dropped column: ${exception.getMessage.take(160)}") + } + + /** + * ALTER TABLE ALTER COLUMN to a narrower type (bigint to int) is rejected with an + * AnalysisException about the unsupported column change. + */ + private def ddlNegNarrowTypeCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.neg.narrowType") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.long0.columnName} TYPE int")) + + assert( + exception.getMessage.contains("NOT_SUPPORTED_CHANGE_COLUMN"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + + /** + * ALTER TABLE ALTER COLUMN SET NOT NULL on a nullable column is rejected with an + * AnalysisException about the nullable-to-non-nullable change. + */ + private def ddlNegSetNotNullCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.neg.setNotNull") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.string0.columnName} SET NOT NULL")) + + assert( + exception.getMessage.contains("Cannot change nullable column to non-nullable"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + + /** The rejected schema changes, on the preparedCoreFormats preparations. */ val ddlNegativeCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => List( - preparation.test( - "ddl.neg.dropColumn", - "ALTER TABLE DROP COLUMN is rejected with a BadRequestException naming the column that " + - "would be dropped.") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} DROP COLUMN ${Core.int0.columnName}")) + ddlNegDropColumnCase(preparation), + ddlNegNarrowTypeCase(preparation), + ddlNegSetNotNullCase(preparation)) + } - assert( - exception.getMessage.contains("not found in newSchema"), - s"unexpected message: ${exception.getMessage.take(160)}") - assert( - exception.getMessage.contains(Core.int0.columnName), - s"message should name the dropped column: ${exception.getMessage.take(160)}") - }, - preparation.test( - "ddl.neg.narrowType", - "ALTER TABLE ALTER COLUMN to a narrower type (bigint to int) is rejected with an " + - "AnalysisException about the unsupported column change.") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.long0.columnName} TYPE int")) + /** SET TBLPROPERTIES adds a user property that reads back, and UNSET TBLPROPERTIES removes it. */ + private def ddlPropsUserRoundTripCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.props.userRoundTrip") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('my_key'='my_val')") + assert( + tableProps(table.spark, table.name).get("my_key").contains("my_val"), + "user prop not set") + + table.spark.sql(s"ALTER TABLE ${table.name} UNSET TBLPROPERTIES ('my_key')") + assert( + !tableProps(table.spark, table.name).contains("my_key"), + "user prop not removed") + } - assert( - exception.getMessage.contains("NOT_SUPPORTED_CHANGE_COLUMN"), - s"unexpected message: ${exception.getMessage.take(160)}") - }, - preparation.test( - "ddl.neg.setNotNull", - "ALTER TABLE ALTER COLUMN SET NOT NULL on a nullable column is rejected with an " + - "AnalysisException about the nullable-to-non-nullable change.") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.string0.columnName} SET NOT NULL")) + /** + * SET TBLPROPERTIES on the reserved openhouse.tableUUID property is rejected with a + * BadRequestException about the restriction. + */ + private def ddlPropsReservedOpenhouseCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.props.reservedOpenhouse") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + + "'openhouse.tableUUID'='deadbeef')")) - assert( - exception.getMessage.contains("Cannot change nullable column to non-nullable"), - s"unexpected message: ${exception.getMessage.take(160)}") - }) - } + assert( + exception.getMessage.toLowerCase.contains("restriction"), + s"msg: ${exception.getMessage.take(200)}") + } + + /** + * Even though format-version=1 was requested at creation, the table is forced to + * format-version=2 and remains writable. + */ + private def ddlPropsFormatVersionForcedCase( + formatVersionPreparation: TablePreparation[CoreTable.type]): Plan.Case = + formatVersionPreparation.test("ddl.props.formatVersionForced") { table => + val formatVersion = tableProps(table.spark, table.name).get("format-version") + + assert( + formatVersion.contains("2"), + s"expected forced format-version=2, got $formatVersion") + assert( + table.rows.size == 3, + "table not writable at the forced format-version") + } + /** + * The write.metadata.previous-versions-max property requested at creation is honored and reads + * back as 7. + */ + private def ddlPropsPreviousVersionsHonoredCase( + previousVersionsPreparation: TablePreparation[CoreTable.type]): Plan.Case = + previousVersionsPreparation.test("ddl.props.previousVersionsHonored") { table => + val previousVersions = + tableProps(table.spark, table.name).get("write.metadata.previous-versions-max") + + assert( + previousVersions.contains("7"), + s"expected previous-versions-max=7, got $previousVersions") + } + + /** + * The table-property cases. Two of them start from the preparedCoreFormats preparation for the + * file format, one from a table created with format-version=1 requested, and one from an unseeded + * table created with write.metadata.previous-versions-max=7. + */ val ddlPropertyCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => val format = preparation.label val formatVersionPreparation = TablePreparation( @@ -167,161 +284,245 @@ trait NegativeDdlScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + s"'write.format.default'='$format', 'format-version'='1')")() - .insert(3)(), - description = "Three seed rows in a table created with format-version=1 requested.") + .insert(3)()) val previousVersionsPreparation = TablePreparation( format, TableTest(Core).sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'write.metadata.previous-versions-max'='7')")(), - description = "An unseeded table created with write.metadata.previous-versions-max=7.") + s"'write.format.default'='$format', 'write.metadata.previous-versions-max'='7')")()) List( - preparation.test( - "ddl.props.userRoundTrip", - "SET TBLPROPERTIES adds a user property that reads back, and UNSET TBLPROPERTIES " + - "removes it.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('my_key'='my_val')") - assert( - tableProps(table.spark, table.name).get("my_key").contains("my_val"), - "user prop not set") + ddlPropsUserRoundTripCase(preparation), + ddlPropsReservedOpenhouseCase(preparation), + ddlPropsFormatVersionForcedCase(formatVersionPreparation), + ddlPropsPreviousVersionsHonoredCase(previousVersionsPreparation)) + } - table.spark.sql(s"ALTER TABLE ${table.name} UNSET TBLPROPERTIES ('my_key')") - assert( - !tableProps(table.spark, table.name).contains("my_key"), - "user prop not removed") - }, - preparation.test( - "ddl.props.reservedOpenhouse", - "SET TBLPROPERTIES on the reserved openhouse.tableUUID property is rejected with a " + - "BadRequestException about the restriction.") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + - "'openhouse.tableUUID'='deadbeef')")) + /** ALTER TABLE WRITE ORDERED BY a single column sets write.distribution-mode to range. */ + private def ddlSortOrderOrderedByCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.sortOrder.orderedBy") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} WRITE ORDERED BY ${Core.long0.columnName}") - assert( - exception.getMessage.toLowerCase.contains("restriction"), - s"msg: ${exception.getMessage.take(200)}") - }, - formatVersionPreparation.test( - "ddl.props.formatVersionForced", - "Even though format-version=1 was requested at creation, the table is forced to " + - "format-version=2 and remains writable.") { table => - val formatVersion = tableProps(table.spark, table.name).get("format-version") + val distributionMode = + tableProps(table.spark, table.name).get("write.distribution-mode") - assert( - formatVersion.contains("2"), - s"expected forced format-version=2, got $formatVersion") - assert( - table.rows.size == 3, - "table not writable at the forced format-version") - }, - previousVersionsPreparation.test( - "ddl.props.previousVersionsHonored", - "The write.metadata.previous-versions-max property requested at creation is honored " + - "and reads back as 7.") { table => - val previousVersions = - tableProps(table.spark, table.name).get("write.metadata.previous-versions-max") + assert( + distributionMode.contains("range"), + s"distribution-mode not range: $distributionMode") + } - assert( - previousVersions.contains("7"), - s"expected previous-versions-max=7, got $previousVersions") - }) - } + /** + * ALTER TABLE WRITE ORDERED BY multiple columns sets range distribution and the table remains + * writable, growing from 3 to 5 rows after a follow-up insert. + */ + private def ddlSortOrderOrderedByMultiCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.sortOrder.orderedByMulti") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} WRITE ORDERED BY " + + s"${Core.string0.columnName} DESC NULLS FIRST, ${Core.long0.columnName}") + + assert( + tableProps(table.spark, table.name).get("write.distribution-mode").contains("range"), + "multi-col ordered-by should set range") + + table.spark.sql( + s"INSERT INTO ${table.name} ${RowGenerator.valuesClause(Core, 2)}") + + assert(table.rows.size == 5, "multi-col ordered write path failed") + } + + /** + * ALTER TABLE RENAME TO moves the table to the new name with its 3 rows intact, and the old name + * stops resolving. A second rename puts the table back under its original name, which teardown + * drops. + */ + private def ddlRenameTableCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.renameTable") { table => + val renamedTable = s"${table.name}_ren" + + table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $renamedTable") + assert( + table.spark.sql(s"SELECT count(*) FROM $renamedTable").collect()(0).getLong(0) == 3, + "renamed table lost rows") + Check.intercept[Exception]( + table.spark.sql(s"SELECT 1 FROM ${table.name} LIMIT 1")) + table.spark.sql(s"ALTER TABLE $renamedTable RENAME TO ${table.name}") + } + + /** + * ALTER TABLE RENAME TO a name that already exists is rejected with an error naming the + * conflict. + */ + private def ddlRenameTableConflictCase( + preparation: TablePreparation[CoreTable.type], + format: String): Plan.Case = + preparation.test("ddl.renameTable.conflict") { table => + val conflictingTable = s"${table.name}_other" + + table.spark.sql(s"DROP TABLE IF EXISTS $conflictingTable") + table.spark.sql( + s"CREATE TABLE $conflictingTable ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')") + val exception = Check.intercept[WebClientResponseWithMessageException]( + table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $conflictingTable")) + + assert( + exception.getMessage.contains("already exists"), + s"msg: ${exception.getMessage.take(160)}") + table.spark.sql(s"DROP TABLE IF EXISTS $conflictingTable") + } - // Each preparation carries its format directly into the cases assembled below. + /** + * CREATE NAMESPACE is rejected with an UnsupportedOperationException, since this catalog does + * not support creating namespaces. + */ + private def ddlNsCreateRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.ns.createRejected") { table => + val exception = Check.intercept[UnsupportedOperationException]( + table.spark.sql("CREATE NAMESPACE openhouse.a_new_db")) + + assert( + exception.getMessage.contains("not supported"), + s"msg: ${exception.getMessage.take(160)}") + } + + /** + * DROP NAMESPACE is rejected with an UnsupportedOperationException, since this catalog does not + * support dropping namespaces. + */ + private def ddlNsDropRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.ns.dropRejected") { table => + val exception = Check.intercept[UnsupportedOperationException]( + table.spark.sql("DROP NAMESPACE openhouse.dbMatrix")) + + assert( + exception.getMessage.contains("not supported"), + s"msg: ${exception.getMessage.take(160)}") + } + + /** The remaining DDL cases, on the preparedCoreFormats preparations. */ val ddlMiscellaneousCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => val format = preparation.label List( - preparation.test( - "ddl.sortOrder.orderedBy", - "ALTER TABLE WRITE ORDERED BY a single column sets write.distribution-mode to range.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} WRITE ORDERED BY ${Core.long0.columnName}") - - val distributionMode = - tableProps(table.spark, table.name).get("write.distribution-mode") + ddlSortOrderOrderedByCase(preparation), + ddlSortOrderOrderedByMultiCase(preparation), + ddlRenameTableCase(preparation), + ddlRenameTableConflictCase(preparation, format), + ddlNsCreateRejectedCase(preparation), + ddlNsDropRejectedCase(preparation)) + } - assert( - distributionMode.contains("range"), - s"distribution-mode not range: $distributionMode") - }, - preparation.test( - "ddl.sortOrder.orderedByMulti", - "ALTER TABLE WRITE ORDERED BY multiple columns sets range distribution and the table " + - "remains writable, growing from 3 to 5 rows after a follow-up insert.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} WRITE ORDERED BY " + - s"${Core.string0.columnName} DESC NULLS FIRST, ${Core.long0.columnName}") + /** SET POLICY (SHARING=TRUE) records the sharing policy and the table remains queryable. */ + private def ddlPolicySharingCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.policy.sharing") { + table => + table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") + + val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + + assert( + policies.toLowerCase.contains("true") || + policies.toLowerCase.contains("sharing"), + s"sharing policy not stored: $policies") + assert( + table.rows.size == 3, + "table not queryable after SET POLICY (SHARING)") + } - assert( - tableProps(table.spark, table.name).get("write.distribution-mode").contains("range"), - "multi-col ordered-by should set range") + /** + * SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20) records the history policy and the table remains + * queryable. + */ + private def ddlPolicyHistoryCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.policy.history") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20)") + + val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + + assert( + policies.contains("20") || policies.toLowerCase.contains("history"), + s"history policy not stored: $policies") + assert( + table.rows.size == 3, + "table not queryable after SET POLICY (HISTORY)") + } - table.spark.sql( - s"INSERT INTO ${table.name} ${RowGenerator.valuesClause(Core, 2)}") + /** + * SET POLICY (REPLICATION) followed by UNSET POLICY (REPLICATION) leaves the table queryable + * with its 3 rows intact. + */ + private def ddlPolicyReplicationCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.policy.replication") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") + table.spark.sql( + s"ALTER TABLE ${table.name} UNSET POLICY (REPLICATION)") + + assert(table.rows.size == 3) + } - assert(table.rows.size == 5, "multi-col ordered write path failed") - }, - preparation.test( - "ddl.renameTable", - "ALTER TABLE RENAME TO moves the table to the new name with its 3 rows intact and the " + - "old name stops resolving; the test restores the original name afterward.") { table => - val renamedTable = s"${table.name}_ren" + /** + * SET POLICY (RETENTION = 30d ON COLUMN datepartition ...) records the retention policy and the + * table remains queryable. + */ + private def ddlPolicyRetentionCase( + retentionPreparation: TablePreparation[CoreTable.type]): Plan.Case = + retentionPreparation.test("ddl.policy.retention") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (" + + "RETENTION = 30d ON COLUMN datepartition WHERE pattern = 'yyyy-MM-dd-HH')") + + val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + + assert( + policies.toLowerCase.contains("retention") || policies.contains("30"), + s"retention policy not stored: $policies") + assert( + table.rows.size == 3, + "table not queryable after SET POLICY (RETENTION)") + } - table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $renamedTable") - assert( - table.spark.sql(s"SELECT count(*) FROM $renamedTable").collect()(0).getLong(0) == 3, - "renamed table lost rows") - Check.intercept[Exception]( - table.spark.sql(s"SELECT 1 FROM ${table.name} LIMIT 1")) - table.spark.sql(s"ALTER TABLE $renamedTable RENAME TO ${table.name}") - }, - preparation.test( - "ddl.renameTable.conflict", - "ALTER TABLE RENAME TO a name that already exists is rejected with an error naming the " + - "conflict.") { table => - val conflictingTable = s"${table.name}_other" - - table.spark.sql(s"DROP TABLE IF EXISTS $conflictingTable") + /** + * SET POLICY (HISTORY MAX_AGE=5D) exceeds the allowed range and is rejected with a + * BadRequestException stating the 1-to-3-day limit. + */ + private def ddlPolicyNegHistoryMaxAgeCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.policy.neg.historyMaxAge") { table => + val exception = Check.intercept[BadRequestException]( table.spark.sql( - s"CREATE TABLE $conflictingTable ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')") - val exception = Check.intercept[WebClientResponseWithMessageException]( - table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $conflictingTable")) + s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=5D)")) - assert( - exception.getMessage.contains("already exists"), - s"msg: ${exception.getMessage.take(160)}") - table.spark.sql(s"DROP TABLE IF EXISTS $conflictingTable") - }, - preparation.test( - "ddl.ns.createRejected", - "CREATE NAMESPACE is rejected with an UnsupportedOperationException, since this " + - "catalog does not support creating namespaces.") { table => - val exception = Check.intercept[UnsupportedOperationException]( - table.spark.sql("CREATE NAMESPACE openhouse.a_new_db")) + assert( + exception.getMessage.contains("max age must be between 1 to 3 days"), + s"msg: ${exception.getMessage.take(160)}") + } - assert( - exception.getMessage.contains("not supported"), - s"msg: ${exception.getMessage.take(160)}") - }, - preparation.test( - "ddl.ns.dropRejected", - "DROP NAMESPACE is rejected with an UnsupportedOperationException, since this catalog " + - "does not support dropping namespaces.") { table => - val exception = Check.intercept[UnsupportedOperationException]( - table.spark.sql("DROP NAMESPACE openhouse.dbMatrix")) + /** + * SET POLICY (HISTORY VERSIONS=200) exceeds the allowed range and is rejected with a + * BadRequestException stating the 2-to-100-version limit. + */ + private def ddlPolicyNegHistoryVersionsCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.policy.neg.historyVersions") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (HISTORY VERSIONS=200)")) - assert( - exception.getMessage.contains("not supported"), - s"msg: ${exception.getMessage.take(160)}") - }) - } + assert( + exception.getMessage.contains("must be between 2 to 100 versions"), + s"msg: ${exception.getMessage.take(160)}") + } + /** + * The table-policy cases. They start from the preparedCoreFormats preparation for the file + * format, except the retention case, which starts from three seed rows in a table partitioned by + * datepartition. + */ val ddlPolicyCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => val format = preparation.label val retentionPreparation = TablePreparation( @@ -331,96 +532,131 @@ trait NegativeDdlScenarios extends ScenarioKit { s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + "PARTITIONED BY (datepartition) " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = "Three seed rows in a table partitioned by datepartition.") + .insert(3)()) List( - preparation.test( - "ddl.policy.sharing", - "SET POLICY (SHARING=TRUE) records the sharing policy and the table remains queryable.") { - table => - table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") + ddlPolicySharingCase(preparation), + ddlPolicyHistoryCase(preparation), + ddlPolicyReplicationCase(preparation), + ddlPolicyRetentionCase(retentionPreparation), + ddlPolicyNegHistoryMaxAgeCase(preparation), + ddlPolicyNegHistoryVersionsCase(preparation)) + } - val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + /** + * ALTER TABLE MODIFY COLUMN SET TAG = (PII) tags a column without masking or changing the values + * that queries return. + */ + private def ddlColTagCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.colTag") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} MODIFY COLUMN " + + s"${Core.string0.columnName} SET TAG = (PII)") + + val values = table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getString(0)) + + assert( + values == Seq("row-1", "row-2", "row-3"), + s"SET TAG changed query results (should not mask): $values") + } - assert( - policies.toLowerCase.contains("true") || - policies.toLowerCase.contains("sharing"), - s"sharing policy not stored: $policies") - assert( - table.rows.size == 3, - "table not queryable after SET POLICY (SHARING)") - }, - preparation.test( - "ddl.policy.history", - "SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20) records the history policy and the table " + - "remains queryable.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20)") + /** + * GRANT SELECT on a table that is not marked shared is rejected with an IllegalArgumentException + * stating the table is not shared. + */ + private def ddlAclGrantUnsharedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.acl.grantUnshared") { table => + val exception = Check.intercept[IllegalArgumentException]( + table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC")) + + assert( + exception.getMessage.contains("is not a shared table"), + s"msg: ${exception.getMessage.take(160)}") + } - val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + /** + * On a shared table, GRANT SELECT TO PUBLIC makes SHOW GRANTS list SELECT for PUBLIC and the + * table stays queryable; REVOKE SELECT then removes that grant from SHOW GRANTS. + */ + private def ddlAclGrantSharedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation + .test("ddl.acl.grantShared") { table => + table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") + table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC") + val grantsAfterGrant = table.spark + .sql(s"SHOW GRANTS ON TABLE ${table.name}") + .collect() + .map(row => (row.getString(0), row.getString(1))) + .toSet assert( - policies.contains("20") || policies.toLowerCase.contains("history"), - s"history policy not stored: $policies") - assert( - table.rows.size == 3, - "table not queryable after SET POLICY (HISTORY)") - }, - preparation.test( - "ddl.policy.replication", - "SET POLICY (REPLICATION) followed by UNSET POLICY (REPLICATION) leaves the table " + - "queryable with its 3 rows intact.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") - table.spark.sql( - s"ALTER TABLE ${table.name} UNSET POLICY (REPLICATION)") - - assert(table.rows.size == 3) - }, - retentionPreparation.test( - "ddl.policy.retention", - "SET POLICY (RETENTION = 30d ON COLUMN datepartition ...) records the retention policy " + - "and the table remains queryable.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (" + - "RETENTION = 30d ON COLUMN datepartition WHERE pattern = 'yyyy-MM-dd-HH')") - - val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + grantsAfterGrant.contains(("SELECT", "PUBLIC")), + s"SHOW GRANTS did not include SELECT for PUBLIC: $grantsAfterGrant") + assert(table.rows.size == 3, "shared/granted table not queryable") + table.spark.sql(s"REVOKE SELECT ON TABLE ${table.name} FROM PUBLIC") + val grantsAfterRevoke = table.spark + .sql(s"SHOW GRANTS ON TABLE ${table.name}") + .collect() + .map(row => (row.getString(0), row.getString(1))) + .toSet assert( - policies.toLowerCase.contains("retention") || policies.contains("30"), - s"retention policy not stored: $policies") - assert( - table.rows.size == 3, - "table not queryable after SET POLICY (RETENTION)") - }, - preparation.test( - "ddl.policy.neg.historyMaxAge", - "SET POLICY (HISTORY MAX_AGE=5D) exceeds the allowed range and is rejected with a " + - "BadRequestException stating the 1-to-3-day limit.") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=5D)")) + !grantsAfterRevoke.contains(("SELECT", "PUBLIC")), + s"SHOW GRANTS retained SELECT for PUBLIC: $grantsAfterRevoke") + } + .copy(embeddedSkipReason = Some( + "The embedded test server has no OPA endpoint configured, so grantRole and " + + "listAclPolicies are no-ops that always report an empty ACL list. GRANT and REVOKE " + + "succeed without error, while SHOW GRANTS always returns an empty ACL list. The " + + "li-openhouse acceptance environment runs the assertions against its configured " + + "authorization service.")) + + /** + * The write.distribution-mode=none property requested at creation is honored and the table + * remains writable under it. + */ + private def ddlFeatureFlagDistributionModeCase( + distributionModePreparation: TablePreparation[CoreTable.type]): Plan.Case = + distributionModePreparation.test("ddl.featureFlag.distributionMode") { table => + val distributionMode = + tableProps(table.spark, table.name).get("write.distribution-mode") + + assert( + distributionMode.contains("none"), + s"distribution-mode not honored: $distributionMode") + assert( + table.rows.size == 3, + "table not writable under distribution-mode=none") + } - assert( - exception.getMessage.contains("max age must be between 1 to 3 days"), - s"msg: ${exception.getMessage.take(160)}") - }, - preparation.test( - "ddl.policy.neg.historyVersions", - "SET POLICY (HISTORY VERSIONS=200) exceeds the allowed range and is rejected with a " + - "BadRequestException stating the 2-to-100-version limit.") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (HISTORY VERSIONS=200)")) + /** + * ALTER TABLE SET TBLPROPERTIES ('openhouse.tableType'='REPLICA_TABLE') is rejected with a + * BadRequestException, since table type cannot be changed after creation. + */ + private def ddlReplTableTypeImmutableCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("ddl.repl.tableTypeImmutable") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + + "'openhouse.tableType'='REPLICA_TABLE')")) - assert( - exception.getMessage.contains("must be between 2 to 100 versions"), - s"msg: ${exception.getMessage.take(160)}") - }) - } + assert( + exception.getMessage.contains("restriction"), + s"msg: ${exception.getMessage.take(160)}") + } + /** + * The column-tag, ACL and feature-flag cases. They start from the preparedCoreFormats preparation + * for the file format, except the distribution-mode case, which starts from three seed rows in a + * table created with write.distribution-mode=none. + */ val ddlTagAclFeatureCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => val format = preparation.label val distributionModePreparation = TablePreparation( @@ -429,103 +665,14 @@ trait NegativeDdlScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + s"'write.format.default'='$format', 'write.distribution-mode'='none')")() - .insert(3)(), - description = "Three seed rows in a table created with write.distribution-mode=none.") + .insert(3)()) List( - preparation.test( - "ddl.colTag", - "ALTER TABLE MODIFY COLUMN SET TAG = (PII) tags a column without masking or changing " + - "the values that queries return.") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} MODIFY COLUMN " + - s"${Core.string0.columnName} SET TAG = (PII)") - - val values = table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - s"ORDER BY ${Core.long0.columnName}") - .collect() - .toSeq - .map(_.getString(0)) - - assert( - values == Seq("row-1", "row-2", "row-3"), - s"SET TAG changed query results (should not mask): $values") - }, - preparation.test( - "ddl.acl.grantUnshared", - "GRANT SELECT on a table that is not marked shared is rejected with an " + - "IllegalArgumentException stating the table is not shared.") { table => - val exception = Check.intercept[IllegalArgumentException]( - table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC")) - - assert( - exception.getMessage.contains("is not a shared table"), - s"msg: ${exception.getMessage.take(160)}") - }, - preparation - .test( - "ddl.acl.grantShared", - "On a shared table, GRANT SELECT TO PUBLIC makes SHOW GRANTS list SELECT for PUBLIC " + - "and the table stays queryable; REVOKE SELECT then removes that grant from SHOW " + - "GRANTS.") { table => - table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") - table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC") - - val grantsAfterGrant = table.spark - .sql(s"SHOW GRANTS ON TABLE ${table.name}") - .collect() - .map(row => (row.getString(0), row.getString(1))) - .toSet - assert( - grantsAfterGrant.contains(("SELECT", "PUBLIC")), - s"SHOW GRANTS did not include SELECT for PUBLIC: $grantsAfterGrant") - assert(table.rows.size == 3, "shared/granted table not queryable") - - table.spark.sql(s"REVOKE SELECT ON TABLE ${table.name} FROM PUBLIC") - val grantsAfterRevoke = table.spark - .sql(s"SHOW GRANTS ON TABLE ${table.name}") - .collect() - .map(row => (row.getString(0), row.getString(1))) - .toSet - assert( - !grantsAfterRevoke.contains(("SELECT", "PUBLIC")), - s"SHOW GRANTS retained SELECT for PUBLIC: $grantsAfterRevoke") - } - .copy(embeddedSkipReason = Some( - "The embedded test server has no OPA endpoint configured, so grantRole and " + - "listAclPolicies are no-ops that always report an empty ACL list. GRANT and REVOKE " + - "succeed without error, while SHOW GRANTS always returns an empty ACL list. The " + - "li-openhouse acceptance environment runs the assertions against its configured " + - "authorization service.")), - distributionModePreparation.test( - "ddl.featureFlag.distributionMode", - "The write.distribution-mode=none property requested at creation is honored and the " + - "table remains writable under it.") { table => - val distributionMode = - tableProps(table.spark, table.name).get("write.distribution-mode") - - assert( - distributionMode.contains("none"), - s"distribution-mode not honored: $distributionMode") - assert( - table.rows.size == 3, - "table not writable under distribution-mode=none") - }, - preparation.test( - "ddl.repl.tableTypeImmutable", - "ALTER TABLE SET TBLPROPERTIES ('openhouse.tableType'='REPLICA_TABLE') is rejected with " + - "a BadRequestException, since table type cannot be changed after creation.") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + - "'openhouse.tableType'='REPLICA_TABLE')")) - - assert( - exception.getMessage.contains("restriction"), - s"msg: ${exception.getMessage.take(160)}") - }) + ddlColTagCase(preparation), + ddlAclGrantUnsharedCase(preparation), + ddlAclGrantSharedCase(preparation), + ddlFeatureFlagDistributionModeCase(distributionModePreparation), + ddlReplTableTypeImmutableCase(preparation)) } } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala index 4470a7f70..6083cce15 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala @@ -14,101 +14,107 @@ trait NestedTypesScenarios extends ScenarioKit { import Rows._ // Nested and complex types (NestedTable). + + /** One unpartitioned nested-column table per file format. */ val nestedLayouts: List[Layout] = List("parquet", "orc", "avro").map(format => Layout(s"nested-unpartitioned/$format", table => s"CREATE TABLE $table (${NestedTable.columnDefinitions}) USING $dataSource TBLPROPERTIES ('write.format.default'='$format')")) + /** Creates the nested-column table under `layout`, then seeds `numberOfRows` rows. */ def createAndSeedNested(layout: Layout, numberOfRows: Int): TableTest[NestedTable.type] = TableTest(NestedTable).sql("create")(layout.create)().insert(numberOfRows)() - val nestedCases: List[Plan.Case] = - nestedLayouts - .map(layout => - TablePreparation( - layout.label, - createAndSeedNested(layout, 3), - description = s"Three seed rows with nested struct, array, map and doubly-nested " + - s"struct fields in an unpartitioned ${layout.label.split('/').last} table.")) - .flatMap { preparation => - List( - preparation.test( - "nested.roundtrip", - "Selecting the top-level id alongside struct, array, map and nested-struct fields " + - "reads back exactly the seeded values for all 3 rows.") { table => - val actual = table.spark - .sql( - s"SELECT id, s.x, s.y, arr, m['k'], nested.inner.z " + - s"FROM ${table.name} ORDER BY id") - .collect() - .toSeq - .map(row => - ( - row.getLong(0), - row.getInt(1), - row.getString(2), - row.getSeq[Int](3), - row.getInt(4), - row.getInt(5))) - val expected = (1 to 3).map { value => - ( - value.toLong, - value, - s"row-$value", - Seq(value, value + 1), - value, - value) - } - - assert(actual == expected) - }, - preparation.test( - "nested.projectField", - "Selecting only a nested struct field (s.x) returns just that field's values for " + - "all 3 rows, in id order.") { table => - val actual = table.spark - .sql(s"SELECT s.x FROM ${table.name} ORDER BY id") - .collect() - .toSeq - .map(_.getInt(0)) - - assert(actual == Seq(1, 2, 3)) - }, - preparation.test( - "nested.filterNestedField", - "Filtering WHERE s.x = 2 on a nested struct field returns only the matching row's " + - "id.") { table => - val actual = table.spark - .sql(s"SELECT id FROM ${table.name} WHERE s.x = 2 ORDER BY id") - .collect() - .toSeq - .map(_.getLong(0)) - - assert(actual == Seq(2L)) - }, - preparation.test( - "nested.updateStructField", - "UPDATE SET s.x = 99 WHERE id = 2 changes only that row's nested field, leaving " + - "other rows' nested fields untouched.") { table => - table.spark.sql( - s"UPDATE ${table.name} SET s.x = 99 WHERE id = 2") - - assert( - table.spark - .sql(s"SELECT s.x FROM ${table.name} WHERE id = 2") - .collect()(0) - .getInt(0) == 99) - assert( - table.spark - .sql(s"SELECT s.x FROM ${table.name} WHERE id = 1") - .collect()(0) - .getInt(0) == 1) - }, - preparation.test( - "nested.mergeInsert", - "MERGE WHEN NOT MATCHED THEN INSERT with a fully nested source row adds a 4th row " + - "whose nested struct field reads back as inserted.") { table => - table.spark.sql( - s"""MERGE INTO ${table.name} target USING ( + /** + * Selecting the top-level id alongside struct, array, map and nested-struct fields reads back + * exactly the seeded values for all 3 rows. + */ + private def nestedRoundtripCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.roundtrip") { table => + val actual = table.spark + .sql( + s"SELECT id, s.x, s.y, arr, m['k'], nested.inner.z " + + s"FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(row => + ( + row.getLong(0), + row.getInt(1), + row.getString(2), + row.getSeq[Int](3), + row.getInt(4), + row.getInt(5))) + val expected = (1 to 3).map { value => + ( + value.toLong, + value, + s"row-$value", + Seq(value, value + 1), + value, + value) + } + + assert(actual == expected) + } + + /** + * Selecting only a nested struct field (s.x) returns just that field's values for all 3 rows, in + * id order. + */ + private def nestedProjectFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.projectField") { table => + val actual = table.spark + .sql(s"SELECT s.x FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(_.getInt(0)) + + assert(actual == Seq(1, 2, 3)) + } + + /** Filtering WHERE s.x = 2 on a nested struct field returns only the matching row's id. */ + private def nestedFilterNestedFieldCase( + preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.filterNestedField") { table => + val actual = table.spark + .sql(s"SELECT id FROM ${table.name} WHERE s.x = 2 ORDER BY id") + .collect() + .toSeq + .map(_.getLong(0)) + + assert(actual == Seq(2L)) + } + + /** + * UPDATE SET s.x = 99 WHERE id = 2 changes only that row's nested field and leaves every other + * row unchanged. + */ + private def nestedUpdateStructFieldCase( + preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.updateStructField") { table => + table.spark.sql( + s"UPDATE ${table.name} SET s.x = 99 WHERE id = 2") + + assert( + table.spark + .sql(s"SELECT s.x FROM ${table.name} WHERE id = 2") + .collect()(0) + .getInt(0) == 99) + assert( + table.spark + .sql(s"SELECT s.x FROM ${table.name} WHERE id = 1") + .collect()(0) + .getInt(0) == 1) + } + + /** + * MERGE WHEN NOT MATCHED THEN INSERT with a fully nested source row adds a 4th row whose nested + * struct field reads back as inserted. + */ + private def nestedMergeInsertCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.mergeInsert") { table => + table.spark.sql( + s"""MERGE INTO ${table.name} target USING ( SELECT * FROM VALUES ( CAST(4 AS BIGINT), @@ -120,64 +126,94 @@ trait NestedTypesScenarios extends ScenarioKit { ) source ON target.id = source.id WHEN NOT MATCHED THEN INSERT *""") - val ids = table.spark - .sql(s"SELECT id FROM ${table.name} ORDER BY id") - .collect() - .toSeq - .map(_.getLong(0)) - - assert(ids == Seq(1L, 2L, 3L, 4L)) - assert( - table.spark - .sql(s"SELECT s.x FROM ${table.name} WHERE id = 4") - .collect()(0) - .getInt(0) == 4) - }, - preparation - .test( - "nested.deleteByNestedField", - "DELETE WHERE s.x = 2 filtering on a nested struct field removes only the matching " + - "row, leaving ids 1 and 3.") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE s.x = 2") - - val ids = table.spark - .sql(s"SELECT id FROM ${table.name} ORDER BY id") - .collect() - .toSeq - .map(_.getLong(0)) - - assert(ids == Seq(1L, 3L)) - } - .copy(knownBugReason = Some( - "DELETE on a nested struct field crashes in the Spark and Iceberg row-level " + - "rewrite.")), - preparation.test( - "nested.nullValues", - "Inserting a row with NULL struct, empty array and empty map reads back a null " + - "struct and an empty array for that row.") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES (" + - "CAST(4 AS BIGINT), " + - "CAST(NULL AS struct), " + - "CAST(array() AS array), " + - "CAST(map() AS map), " + - "CAST(NULL AS struct>))") - - val insertedRow = table.spark - .sql(s"SELECT id, s, arr FROM ${table.name} WHERE id = 4") - .collect()(0) - - assert(insertedRow.isNullAt(1)) - assert(insertedRow.getSeq[Int](2).isEmpty) - }) + val ids = table.spark + .sql(s"SELECT id FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(_.getLong(0)) + + assert(ids == Seq(1L, 2L, 3L, 4L)) + assert( + table.spark + .sql(s"SELECT s.x FROM ${table.name} WHERE id = 4") + .collect()(0) + .getInt(0) == 4) + } + + /** + * DELETE WHERE s.x = 2 filtering on a nested struct field removes only the matching row, leaving + * ids 1 and 3. + */ + private def nestedDeleteByNestedFieldCase( + preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation + .test("nested.deleteByNestedField") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE s.x = 2") + + val ids = table.spark + .sql(s"SELECT id FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(_.getLong(0)) + + assert(ids == Seq(1L, 3L)) + } + .copy(knownBugReason = Some( + "DELETE on a nested struct field crashes in the Spark and Iceberg row-level " + + "rewrite.")) + + /** + * Inserting a row with NULL struct, empty array and empty map reads back a null struct and an + * empty array for that row. + */ + private def nestedNullValuesCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.nullValues") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES (" + + "CAST(4 AS BIGINT), " + + "CAST(NULL AS struct), " + + "CAST(array() AS array), " + + "CAST(map() AS map), " + + "CAST(NULL AS struct>))") + + val insertedRow = table.spark + .sql(s"SELECT id, s, arr FROM ${table.name} WHERE id = 4") + .collect()(0) + + assert(insertedRow.isNullAt(1)) + assert(insertedRow.getSeq[Int](2).isEmpty) + } + + /** + * The nested-type cases. Each preparation holds three seed rows with struct, array, map and + * doubly-nested struct fields in one unpartitioned nested layout. + */ + val nestedCases: List[Plan.Case] = + nestedLayouts + .map(layout => + TablePreparation( + layout.label, + createAndSeedNested(layout, 3))) + .flatMap { preparation => + List( + nestedRoundtripCase(preparation), + nestedProjectFieldCase(preparation), + nestedFilterNestedFieldCase(preparation), + nestedUpdateStructFieldCase(preparation), + nestedMergeInsertCase(preparation), + nestedDeleteByNestedFieldCase(preparation), + nestedNullValuesCase(preparation)) } // Type-edge coverage (TypesTable). + + /** One unpartitioned scalar-type table per file format. */ val typesLayouts: List[Layout] = List("parquet", "orc", "avro").map(format => Layout(s"types-unpartitioned/$format", table => s"CREATE TABLE $table (${TypesTable.columnDefinitions}) USING $dataSource TBLPROPERTIES ('write.format.default'='$format')")) + /** Creates the scalar-type table under `layout`, then seeds `numberOfRows` rows. */ def createAndSeedTypes(layout: Layout, numberOfRows: Int): TableTest[TypesTable.type] = TableTest(TypesTable).sql("create")(layout.create)().insert(numberOfRows)() @@ -191,120 +227,214 @@ trait NestedTypesScenarios extends ScenarioKit { s"CAST(${id}.50 AS decimal(10,2)), '$str', CAST('bin-$id' AS binary), " + s"DATE '${timestamp.take(10)}', TIMESTAMP '$timestamp', TIMESTAMP_NTZ '$timestamp')" + /** + * Selecting id, n, x, dec and str for the first seeded row reads back the exact long, int, + * double, decimal and string values that were seeded. + */ + private def typesRoundtripCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + preparation.test("types.roundtrip") { table => + val row = table.spark + .sql( + s"SELECT id, n, x, dec, str FROM ${table.name} WHERE id = 1") + .collect()(0) + + assert( + row.getLong(0) == 1L && + row.getInt(1) == 1 && + row.getDouble(2) == 1.5) + assert( + row.getDecimal(3).compareTo( + new java.math.BigDecimal("1.50")) == 0) + assert(row.getString(4) == "row-1") + } + + /** + * Inserting a row with every non-key column NULL reads back as null for the int, double, string, + * timestamp and timestamp_ntz columns. + */ + private def typesNullsCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + preparation.test("types.nulls") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES (" + + "CAST(10 AS BIGINT), NULL, NULL, NULL, NULL, " + + "NULL, NULL, NULL, NULL)") + + val row = table.spark + .sql( + s"SELECT n, x, str, ts, tsntz FROM ${table.name} WHERE id = 10") + .collect()(0) + + assert((0 to 4).forall(row.isNullAt)) + } + + /** + * Inserting rows with double('NaN') and double('Infinity') reads back as NaN and positive + * infinity respectively. + */ + private def typesSpecialFloatsCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + preparation.test("types.specialFloats") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + s"${typesRow(11, "0", "double('NaN')", "CAST(0 AS decimal(10,2))", "'x'")}, " + + s"${typesRow(12, "0", "double('Infinity')", "CAST(0 AS decimal(10,2))", "'y'")}") + + assert( + table.spark + .sql(s"SELECT x FROM ${table.name} WHERE id = 11") + .collect()(0) + .getDouble(0) + .isNaN) + assert( + table.spark + .sql(s"SELECT x FROM ${table.name} WHERE id = 12") + .collect()(0) + .getDouble(0) + .isInfinite) + } + + /** + * Inserting a row at Long.MaxValue, Int.MaxValue and a max-precision decimal reads those + * boundary values back unchanged. + */ + private def typesBoundariesCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + preparation.test("types.boundaries") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + typesRow( + Long.MaxValue, + Int.MaxValue.toString, + "0.0", + "CAST(99999999.99 AS decimal(10,2))", + "'max'")) + + val row = table.spark + .sql( + s"SELECT id, n, dec FROM ${table.name} WHERE str = 'max'") + .collect()(0) + + assert( + row.getLong(0) == Long.MaxValue && + row.getInt(1) == Int.MaxValue) + assert( + row.getDecimal(2).compareTo( + new java.math.BigDecimal("99999999.99")) == 0) + } + + /** Inserting rows with a unicode string and an empty string reads each back unchanged. */ + private def typesUnicodeAndEmptyCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + preparation.test("types.unicodeAndEmpty") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + s"${typesRow(13, "0", "0.0", "CAST(0 AS decimal(10,2))", "'\u65e5\u672c\u8a9e \uD83C\uDF89'")}, " + + s"${typesRow(14, "0", "0.0", "CAST(0 AS decimal(10,2))", "''")}") + + assert( + table.spark + .sql(s"SELECT str FROM ${table.name} WHERE id = 13") + .collect()(0) + .getString(0) == "\u65e5\u672c\u8a9e \uD83C\uDF89") + assert( + table.spark + .sql(s"SELECT str FROM ${table.name} WHERE id = 14") + .collect()(0) + .getString(0) == "") + } + + /** + * The type-edge cases. Each preparation holds three seed rows covering the int, double, decimal, + * string, binary, date, timestamp and timestamp_ntz columns in one unpartitioned types layout. + */ val typesCases: List[Plan.Case] = typesLayouts .map(layout => TablePreparation( layout.label, - createAndSeedTypes(layout, 3), - description = "Three seed rows covering int, double, decimal, string, binary, date, " + - s"timestamp and timestamp_ntz columns in an unpartitioned ${layout.label.split('/').last} table.")) + createAndSeedTypes(layout, 3))) .flatMap { preparation => List( - preparation.test( - "types.roundtrip", - "Selecting id, n, x, dec and str for the first seeded row reads back the exact " + - "long, int, double, decimal and string values that were seeded.") { table => - val row = table.spark - .sql( - s"SELECT id, n, x, dec, str FROM ${table.name} WHERE id = 1") - .collect()(0) - - assert( - row.getLong(0) == 1L && - row.getInt(1) == 1 && - row.getDouble(2) == 1.5) - assert( - row.getDecimal(3).compareTo( - new java.math.BigDecimal("1.50")) == 0) - assert(row.getString(4) == "row-1") - }, - preparation.test( - "types.nulls", - "Inserting a row with every non-key column NULL reads back as null for the int, " + - "double, string, timestamp and timestamp_ntz columns.") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES (" + - "CAST(10 AS BIGINT), NULL, NULL, NULL, NULL, " + - "NULL, NULL, NULL, NULL)") - - val row = table.spark - .sql( - s"SELECT n, x, str, ts, tsntz FROM ${table.name} WHERE id = 10") - .collect()(0) - - assert((0 to 4).forall(row.isNullAt)) - }, - preparation.test( - "types.specialFloats", - "Inserting rows with double('NaN') and double('Infinity') reads back as NaN and " + - "positive infinity respectively.") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - s"${typesRow(11, "0", "double('NaN')", "CAST(0 AS decimal(10,2))", "'x'")}, " + - s"${typesRow(12, "0", "double('Infinity')", "CAST(0 AS decimal(10,2))", "'y'")}") - - assert( - table.spark - .sql(s"SELECT x FROM ${table.name} WHERE id = 11") - .collect()(0) - .getDouble(0) - .isNaN) - assert( - table.spark - .sql(s"SELECT x FROM ${table.name} WHERE id = 12") - .collect()(0) - .getDouble(0) - .isInfinite) - }, - preparation.test( - "types.boundaries", - "Inserting a row at Long.MaxValue, Int.MaxValue and a max-precision decimal reads " + - "those boundary values back unchanged.") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - typesRow( - Long.MaxValue, - Int.MaxValue.toString, - "0.0", - "CAST(99999999.99 AS decimal(10,2))", - "'max'")) - - val row = table.spark - .sql( - s"SELECT id, n, dec FROM ${table.name} WHERE str = 'max'") - .collect()(0) - - assert( - row.getLong(0) == Long.MaxValue && - row.getInt(1) == Int.MaxValue) - assert( - row.getDecimal(2).compareTo( - new java.math.BigDecimal("99999999.99")) == 0) - }, - preparation.test( - "types.unicodeAndEmpty", - "Inserting rows with a unicode string and an empty string reads each back " + - "unchanged.") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - s"${typesRow(13, "0", "0.0", "CAST(0 AS decimal(10,2))", "'\u65e5\u672c\u8a9e \uD83C\uDF89'")}, " + - s"${typesRow(14, "0", "0.0", "CAST(0 AS decimal(10,2))", "''")}") - - assert( - table.spark - .sql(s"SELECT str FROM ${table.name} WHERE id = 13") - .collect()(0) - .getString(0) == "\u65e5\u672c\u8a9e \uD83C\uDF89") - assert( - table.spark - .sql(s"SELECT str FROM ${table.name} WHERE id = 14") - .collect()(0) - .getString(0) == "") - }) + typesRoundtripCase(preparation), + typesNullsCase(preparation), + typesSpecialFloatsCase(preparation), + typesBoundariesCase(preparation), + typesUnicodeAndEmptyCase(preparation)) } // Partition transforms and evolution. + + /** + * One supported partition transform: a table PARTITIONED BY that transform reports a single + * partition field with the expected name in its partitions metadata table, and the seeded rows + * land in the expected number of distinct partitions. The transform, its partition field name, + * and that partition count are the parameters. + */ + private def supportedPartitionTransformCase( + format: String, + caseName: String, + transform: String, + partitionField: String, + expectedPartitionCount: Int): Plan.Case = + TablePreparation( + format, + TableTest(TypesTable) + .sql("create")(table => + s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + + s"USING $dataSource PARTITIONED BY ($transform) " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .sql("insertPartitionRows")(table => + s"INSERT INTO $table VALUES " + + partitionRow(1, "aa-1", "2023-12-31 23:00:00") + ", " + + partitionRow(2, "bb-2", "2024-01-01 00:00:00") + ", " + + partitionRow(3, "cc-3", "2024-02-01 01:00:00"))(view => + assert( + view.after.size == view.before.size + 3, + s"expected three partition test rows, got ${view.after.size}"))) + .test(caseName) { table => + val partitionTable = + table.spark.table(s"${table.name}.partitions") + val partitionFields = partitionTable.schema("partition").dataType + .asInstanceOf[org.apache.spark.sql.types.StructType] + .fieldNames + .toSeq + + assert( + partitionFields == Seq(partitionField), + s"expected partition field $partitionField, got ${partitionFields.mkString(", ")}") + assert( + partitionTable.count() == expectedPartitionCount, + s"expected $expectedPartitionCount partitions for $transform") + } + + /** + * One rejected partition transform: CREATE TABLE PARTITIONED BY that transform fails with a + * RuntimeException carrying the expected message, and no scratch table is left behind. The + * transform and the expected message are the parameters. + */ + private def rejectedPartitionTransformCase( + format: String, + caseName: String, + transform: String, + expectedMessage: String): Plan.Case = + TablePreparation( + format, + TableTest(TypesTable) + .sql("create")(table => + s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + + s"USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")()) + .test(caseName) { table => + val scratchTable = table.name + "_x" + val exception = Check.intercept[RuntimeException]( + table.spark.sql( + s"CREATE TABLE $scratchTable " + + s"(${TypesTable.columnDefinitions}) " + + s"USING $dataSource PARTITIONED BY ($transform) " + + s"TBLPROPERTIES ('write.format.default'='$format')")) + + table.spark.sql(s"DROP TABLE IF EXISTS $scratchTable") + assert(exception.getMessage.contains(expectedMessage)) + } + + /** The supported and the rejected partition transforms, in parquet and in orc. */ val partitionTransformCases: List[Plan.Case] = List("parquet", "orc").flatMap { format => val supported = List( @@ -317,41 +447,7 @@ trait NestedTypesScenarios extends ScenarioKit { ("partition.hours", "hours(ts)", "ts_hour", 3)) .map { case (caseName, transform, partitionField, expectedPartitionCount) => - TablePreparation( - format, - TableTest(TypesTable) - .sql("create")(table => - s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + - s"USING $dataSource PARTITIONED BY ($transform) " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .sql("insertPartitionRows")(table => - s"INSERT INTO $table VALUES " + - partitionRow(1, "aa-1", "2023-12-31 23:00:00") + ", " + - partitionRow(2, "bb-2", "2024-01-01 00:00:00") + ", " + - partitionRow(3, "cc-3", "2024-02-01 01:00:00"))(view => - assert( - view.after.size == view.before.size + 3, - s"expected three partition test rows, got ${view.after.size}")), - description = s"Three rows in a $format table partitioned by $transform.") - .test( - caseName, - s"With PARTITIONED BY ($transform), the partitions metadata table reports a " + - s"single partition field named $partitionField and $expectedPartitionCount " + - "distinct partitions for the seeded rows.") { table => - val partitionTable = - table.spark.table(s"${table.name}.partitions") - val partitionFields = partitionTable.schema("partition").dataType - .asInstanceOf[org.apache.spark.sql.types.StructType] - .fieldNames - .toSeq - - assert( - partitionFields == Seq(partitionField), - s"expected partition field $partitionField, got ${partitionFields.mkString(", ")}") - assert( - partitionTable.count() == expectedPartitionCount, - s"expected $expectedPartitionCount partitions for $transform") - } + supportedPartitionTransformCase(format, caseName, transform, partitionField, expectedPartitionCount) } val rejected = List( ("partition.void.rejected", "void(n)", "not supported"), @@ -361,80 +457,63 @@ trait NestedTypesScenarios extends ScenarioKit { "Unsupported column")) .map { case (caseName, transform, expectedMessage) => - TablePreparation( - format, - TableTest(TypesTable) - .sql("create")(table => - s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + - s"USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")(), - description = s"An unpartitioned, unseeded $format table.") - .test( - caseName, - s"CREATE TABLE PARTITIONED BY ($transform) is rejected with a RuntimeException " + - s"whose message contains '$expectedMessage', and no scratch table is left " + - "behind.") { table => - val scratchTable = table.name + "_x" - val exception = Check.intercept[RuntimeException]( - table.spark.sql( - s"CREATE TABLE $scratchTable " + - s"(${TypesTable.columnDefinitions}) " + - s"USING $dataSource PARTITIONED BY ($transform) " + - s"TBLPROPERTIES ('write.format.default'='$format')")) - - table.spark.sql(s"DROP TABLE IF EXISTS $scratchTable") - assert(exception.getMessage.contains(expectedMessage)) - } + rejectedPartitionTransformCase(format, caseName, transform, expectedMessage) } supported ++ rejected } - // Partition evolution is not supported: ALTER TABLE ADD or DROP PARTITION FIELD is rejected with - // a 400 response telling the caller to recreate the table. These cases capture that rejection. + /** + * On three seed rows in an unpartitioned table in the given file format, ALTER TABLE ADD + * PARTITION FIELD is rejected with an exception stating that evolution of table partitioning is + * unsupported, which leaves recreating the table as the way to change partitioning. + */ + private def partitionEvolutionAddRejectedCase(format: String): Plan.Case = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + .test("partition.evolutionAdd.rejected") { table => + val exception = Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} ADD PARTITION FIELD datepartition")) + + assert( + exception.getMessage.contains("Evolution of table partitioning")) + } + + /** + * On three seed rows in a table partitioned by datepartition in the given file format, ALTER + * TABLE DROP PARTITION FIELD is rejected with an exception stating that evolution of table + * partitioning is unsupported. + */ + private def partitionEvolutionDropRejectedCase(format: String): Plan.Case = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + "PARTITIONED BY (datepartition) " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(3)()) + .test("partition.evolutionDrop.rejected") { table => + val exception = Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} DROP PARTITION FIELD datepartition")) + + assert( + exception.getMessage.contains("Evolution of table partitioning")) + } + + /** The rejected partition-evolution statements, in parquet and in orc. */ val partitionEvolutionCases: List[Plan.Case] = List("parquet", "orc").flatMap { format => List( - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = s"Three seed rows in an unpartitioned $format table.") - .test( - "partition.evolutionAdd.rejected", - "ALTER TABLE ADD PARTITION FIELD is rejected with an exception stating partition " + - "evolution is not supported.") { table => - val exception = Check.intercept[Exception]( - table.spark.sql( - s"ALTER TABLE ${table.name} ADD PARTITION FIELD datepartition")) - - assert( - exception.getMessage.contains("Evolution of table partitioning")) - }, - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "PARTITIONED BY (datepartition) " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = s"Three seed rows in a $format table partitioned by datepartition.") - .test( - "partition.evolutionDrop.rejected", - "ALTER TABLE DROP PARTITION FIELD is rejected with an exception stating partition " + - "evolution is not supported.") { table => - val exception = Check.intercept[Exception]( - table.spark.sql( - s"ALTER TABLE ${table.name} DROP PARTITION FIELD datepartition")) - - assert( - exception.getMessage.contains("Evolution of table partitioning")) - }) + partitionEvolutionAddRejectedCase(format), + partitionEvolutionDropRejectedCase(format)) } - } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala index 5aea715df..32ad8acad 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala @@ -3,15 +3,11 @@ package harness /** Defines the ordered catalog of scenario-owned test cases. */ object Plan { final case class Case( - id: String, - run: Ctx => Unit, - description: String, - preparationDescription: String = "", - knownBugReason: Option[String] = None, - embeddedSkipReason: Option[String] = None - ) { - require(description.trim.nonEmpty, s"test case $id needs a description") - } + id: String, + run: Ctx => Unit, + knownBugReason: Option[String] = None, + embeddedSkipReason: Option[String] = None + ) /** The deterministic ordered case catalog. Reading it does not execute a case or start Spark. */ def caseIds: List[String] = cases.map(_.id) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala index 412a2dd58..a2ebf9ddf 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala @@ -43,168 +43,191 @@ trait ScenarioKit { protected val columnDefinitions = "foo_col_long bigint, foo_col_int int, foo_col_string string, foo_col_double double, foo_col_boolean boolean, datepartition string" - /** One starting table shape: the label that names it in a case id, a human description of the - * resulting table, and the CREATE statement that builds it. */ - final case class Layout(label: String, description: String, create: String => String) + /** One starting table shape: the label that names it in a case ID and the CREATE statement that + * builds it. */ + final case class Layout(label: String, create: String => String) - object Layout { - /** A layout whose label already reads as its description. */ - def apply(label: String, create: String => String): Layout = Layout(label, label, create) - } - - /** One partitioning choice: the label that names it in a case id, a human description, and the - * CREATE clause that applies it. */ - final case class Partitioning(label: String, description: String, clause: String) + /** One partitioning choice: the label that names it in a case ID and the CREATE clause that + * applies it. */ + final case class Partitioning(label: String, clause: String) - protected val unpartitioned = Partitioning("unpartitioned", "with no partitioning", "") + /** The empty partitioning clause: the table keeps all its rows in one unpartitioned file set. */ + protected val unpartitioned = Partitioning("unpartitioned", "") + /** Partitions the table by datepartition, so each distinct date value owns one partition. */ protected val partitionedByDate = - Partitioning("partitioned", "partitioned by datepartition", "PARTITIONED BY (datepartition)") + Partitioning("partitioned", "PARTITIONED BY (datepartition)") protected val partitionings: List[Partitioning] = List(unpartitioned, partitionedByDate) protected val fileFormats: List[String] = List("parquet", "orc", "avro") + /** One copy-on-write table in `format`, shaped by `partitioning`, labelled for its case IDs. */ private def coreLayout(partitioning: Partitioning, format: String): Layout = Layout( s"${partitioning.label}/$format", - s"a copy-on-write $format table ${partitioning.description}", table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource ${partitioning.clause} " + s"TBLPROPERTIES ('write.format.default'='$format')") + /** Every core layout: each file format crossed with each partitioning. */ val layouts: List[Layout] = for { format <- fileFormats partitioning <- partitionings } yield coreLayout(partitioning, format) + /** The core layouts partitioned by datepartition, one per file format. */ val partitionedLayouts: List[Layout] = fileFormats.map(format => coreLayout(partitionedByDate, format)) - // Parquet and ORC layouts for bespoke DDL cases that do not need the full format cross. + /** + * The Parquet and ORC core layouts, each crossed with both partitionings, for the bespoke DDL + * cases that do not need the full file-format cross. + */ val parquetAndOrcLayouts: List[Layout] = for { format <- List("parquet", "orc") partitioning <- partitionings } yield coreLayout(partitioning, format) - // Create the table under `layout`, then seed deterministic rows as a second visible step. + /** Creates the table under `layout`, then seeds `numberOfRows` deterministic rows. */ def createAndSeed(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = TableTest(Core).sql("create")(layout.create)().insert(numberOfRows)() + /** One preparation per core layout: three seed rows with keys 1, 2 and 3. */ val preparedCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation( layout.label, - createAndSeed(layout, 3), - description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}.")) + createAndSeed(layout, 3))) + /** + * One preparation per datepartition-partitioned core layout: three seed rows with keys 1, 2 and + * 3, one row per datepartition value. + */ val preparedPartitionedCoreTables: List[TablePreparation[CoreTable.type]] = partitionedLayouts.map(layout => TablePreparation( layout.label, - createAndSeed(layout, 3), - description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, " + - "one row per datepartition value.")) + createAndSeed(layout, 3))) + /** + * One preparation per core layout: three seed rows, then ALTER TABLE WRITE ORDERED BY the long + * key, so the table carries that write sort order. + */ val preparedOrderedCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation( layout.label, createAndSeedOrdered(layout, 3), - "prep.ordered:", - description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + - s"ALTER TABLE WRITE ORDERED BY ${Core.long0.columnName}, so the table carries that write sort order.")) + "prep.ordered:")) + /** + * One preparation per core layout: three seed rows, then ADD COLUMN prep_extra int, so the table + * carries one column beyond the seed row shape and the seeded rows read null for it. + */ val preparedEvolvedCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation( layout.label, createAndSeedEvolved(layout, 3), - "prep.evolved:", - description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}, then " + - "ADD COLUMN prep_extra int, so the table carries one column beyond the seed row shape " + - "and the seeded rows read null for it.")) + "prep.evolved:")) + /** + * One preparation per core layout: the table is created and left unseeded, so it holds no rows. + */ val preparedEmptyCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation( layout.label, - TableTest(Core).sql("create")(layout.create)(), - description = s"${layout.description.capitalize} that is created and left unseeded, so it holds no rows.")) + TableTest(Core).sql("create")(layout.create)())) + /** + * One preparation per Parquet and ORC unpartitioned layout: three seed rows with keys 1, 2 and 3. + */ val preparedCoreFormats: List[TablePreparation[CoreTable.type]] = List("parquet", "orc").map { format => val layout = coreLayout(unpartitioned, format) TablePreparation( format, - createAndSeed(layout, 3), - description = s"Three seed rows with keys 1, 2 and 3 in ${layout.description}.") + createAndSeed(layout, 3)) } - // A DDL step evolves the starting state, and the test case then runs on the evolved table. The - // ordered preparation adds a write sort order and leaves the column list intact, so every DML case - // runs on it. The evolved preparation adds a column, so it runs the cases that address columns by - // name: reads, deletes, and updates. + /** + * Creates and seeds the table under `layout`, then gives it a write sort order on the long key. + * The column list stays as seeded, so every DML case runs on the result. + */ def createAndSeedOrdered(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = createAndSeed(layout, numberOfRows).sql("prep.ordered")(t => s"ALTER TABLE $t WRITE ORDERED BY ${CoreTable.long0.columnName}")() + /** + * Creates and seeds the table under `layout`, then adds the prep_extra column. The column list + * grows past the seed row shape, so the cases that address columns by name run on the result: + * the reads, the deletes and the updates. + */ def createAndSeedEvolved(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = createAndSeed(layout, numberOfRows).sql("prep.evolved")(t => s"ALTER TABLE $t ADD COLUMN prep_extra int")() - // The same starting state with one more row appended, whose string column is null. A DELETE that - // selects rows by IS NULL is then written as one operation against a table that already holds a - // null string. + /** + * The same starting state with a fourth row whose key is 99 and whose string column is null, so + * exactly one row of the table reads null for that column. + */ protected def withNullStringRow( basePreparation: TablePreparation[CoreTable.type] ): TablePreparation[CoreTable.type] = basePreparation.copy( preparation = basePreparation.preparation.sql("prep.nullStringRow")(table => - s"INSERT INTO $table VALUES (CAST(99 AS BIGINT), 99, NULL, 99.5, false, '2024-01-01-00')")(), - description = s"${basePreparation.description} A fourth row with key 99 is then appended " + - s"whose ${Core.string0.columnName} is null, so exactly one row of the table reads null for " + - "that column.") + s"INSERT INTO $table VALUES (CAST(99 AS BIGINT), 99, NULL, 99.5, false, '2024-01-01-00')")()) + /** The core preparations, each carrying one row whose string column is null. */ val preparedNullStringCoreTables: List[TablePreparation[CoreTable.type]] = preparedCoreTables.map(withNullStringRow) + /** The write-ordered preparations, each carrying one row whose string column is null. */ val preparedNullStringOrderedCoreTables: List[TablePreparation[CoreTable.type]] = preparedOrderedCoreTables.map(withNullStringRow) - // This list validates that each preparation writes data files in its declared format. It runs on - // every preparation that leaves data files behind. Each feature layer owns the list for its - // preparations and builds it through this shared case body. + /** + * Every data file the preparation wrote carries the extension of the table's declared + * write.format.default, and listing the files leaves the table state unchanged. + */ + private def formatMaterializationCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("format.materialization") { table => + val before = table.state + val declaredFormat = table.spark + .sql(s"SHOW TBLPROPERTIES ${table.name} ('write.format.default')") + .collect()(0) + .getString(1) + val filePaths = table.spark + .sql(s"SELECT file_path FROM ${table.name}.files") + .collect() + .toSeq + .map(_.getString(0)) + val after = table.state + + assert( + filePaths.nonEmpty && filePaths.forall(_.toLowerCase.endsWith(s".$declaredFormat")), + s"data files are not all .$declaredFormat: $filePaths") + assert(after == before, "listing files leaves the rows and the snapshot count unchanged") + } + + /** + * The format-materialization case for each preparation given. It applies to any preparation that + * leaves data files behind, so each feature layer passes the list its own preparations produce. + */ def layoutFormatCasesFor( preparations: List[TablePreparation[CoreTable.type]] ): List[Plan.Case] = preparations.map { preparation => - preparation.test( - "format.materialization", - "Every data file the preparation wrote carries the extension of the table's declared " + - "write.format.default, and listing the files leaves the table state unchanged.") { table => - val before = table.state - val declaredFormat = table.spark - .sql(s"SHOW TBLPROPERTIES ${table.name} ('write.format.default')") - .collect()(0) - .getString(1) - val filePaths = table.spark - .sql(s"SELECT file_path FROM ${table.name}.files") - .collect() - .toSeq - .map(_.getString(0)) - val after = table.state - - assert( - filePaths.nonEmpty && filePaths.forall(_.toLowerCase.endsWith(s".$declaredFormat")), - s"data files are not all .$declaredFormat: $filePaths") - assert(after == before, "listing files leaves the rows and the snapshot count unchanged") - } + formatMaterializationCase(preparation) } + /** The standard preparations that leave data files behind: the core and write-ordered ones. */ val layoutFormatPreparations: List[TablePreparation[CoreTable.type]] = preparedCoreTables ++ preparedOrderedCoreTables + /** The format-materialization case on every standard preparation that writes data files. */ def layoutFormatCases: List[Plan.Case] = layoutFormatCasesFor(layoutFormatPreparations) private def waitForNextSnapshotTimestamp(spark: SparkSession, table: String): Unit = { @@ -229,6 +252,11 @@ trait ScenarioKit { } // Shared helpers used across domain traits. + + /** + * Creates a table in the given file format, seeds three rows as the first snapshot, then inserts + * rows 4 and 5 as a second snapshot committed at a later timestamp. + */ protected def coreTwoSnapshots(fmt: String): TableTest[CoreTable.type] = TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')")() @@ -237,6 +265,7 @@ trait ScenarioKit { .sql("insertMore")(table => s"INSERT INTO $table VALUES " + s"(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")() + /** The two-snapshot table in parquet. */ protected def coreTwoSnapshots: TableTest[CoreTable.type] = coreTwoSnapshots("parquet") // Snapshots in ancestry order (root first), following the parent_id chain. This is deterministic even diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala index 39abe1f75..b01392fda 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala @@ -12,8 +12,9 @@ import scala.util.control.NonFatal // The standard surface families. A surface case pins one edge of what the catalog exposes on a // plain copy-on-write table: a reader, a procedure, a metadata table, a concurrency outcome, a -// schema change, or a write property. The concurrency helpers below are feature neutral, so a -// feature layer reuses them through a self-type on this trait. The cases run on parquet and orc. +// schema change, or a write property. Each family builds the starting states it needs, so a family +// reads on its own. The concurrency helpers below are feature neutral, so a feature layer reuses +// them through a self-type on this trait. The cases run on parquet and orc. trait SurfaceScenarios extends ScenarioKit { import Rows._ @@ -68,8 +69,10 @@ trait SurfaceScenarios extends ScenarioKit { className.contains("WebClientResponse") } - // Each surface family builds the starting states it needs, so a family reads on its own. The - // seeded table is the plainest of them, so the feature layers build their cases on it too. + /** + * Three seed rows with keys 1, 2 and 3 in an unpartitioned table in the given file format. This + * is the plainest starting state here, so the feature layers build their cases on it too. + */ protected def surfaceBasePreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation( format, @@ -77,9 +80,12 @@ trait SurfaceScenarios extends ScenarioKit { .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)(), - description = s"Three seed rows with keys 1, 2 and 3 in an unpartitioned $format table.") + .insert(3)()) + /** + * Five rows across two snapshots, a three-row seed then a two-row insert, in an unpartitioned + * table in the given file format. + */ private def surfaceTwoSnapshotPreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation( format, @@ -91,19 +97,21 @@ trait SurfaceScenarios extends ScenarioKit { .sql("insertMore")(table => s"INSERT INTO $table VALUES " + "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + - "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")(), - description = s"Five rows across two snapshots (a 3-row seed then a 2-row insert) in an " + - s"unpartitioned $format table.") + "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) + /** An unseeded, empty unpartitioned table in the given file format. */ private def surfaceEmptyPreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation( format, TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")(), - description = s"An unseeded, empty unpartitioned $format table.") + s"TBLPROPERTIES ('write.format.default'='$format')")()) + /** + * Three seed rows in a table in the given file format, partitioned by datepartition and carrying + * write.distribution-mode=hash. + */ private def surfaceHashPreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation( format, @@ -114,10 +122,12 @@ trait SurfaceScenarios extends ScenarioKit { "TBLPROPERTIES (" + s"'write.format.default'='$format', " + "'write.distribution-mode'='hash')")() - .insert(3)(), - description = s"Three seed rows in a $format table partitioned by datepartition with " + - "write.distribution-mode=hash.") + .insert(3)()) + /** + * Three seed rows in an unpartitioned table in the given file format, carrying + * write.target-file-size-bytes=1048576. + */ private def surfaceTargetFileSizePreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation( format, @@ -127,572 +137,649 @@ trait SurfaceScenarios extends ScenarioKit { "TBLPROPERTIES (" + s"'write.format.default'='$format', " + "'write.target-file-size-bytes'='1048576')")() - .insert(3)(), - description = s"Three seed rows in an unpartitioned $format table with " + - "write.target-file-size-bytes=1048576.") - - // The structured-streaming reader and writer, and the changelog view. - def surfaceReaderCases(format: String): List[Plan.Case] = - List( - surfaceBasePreparation(format).test( - "surface.stream.read", - "A Spark structured streaming read of the table, run in AvailableNow batch mode, " + - "delivers all 3 seed rows to a memory sink within 120 seconds.") { table => - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-read").toString - val sink = s"memsink_${System.nanoTime}" - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("memory") - .queryName(sink) - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .start() + .insert(3)()) + + /** + * A Spark structured streaming read of the table, run in AvailableNow batch mode, delivers all 3 + * seed rows to a memory sink within 120 seconds. + */ + private def surfaceStreamReadCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.stream.read") { table => + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-read").toString + val sink = s"memsink_${System.nanoTime}" + val query = table.spark.readStream + .table(table.name) + .writeStream + .format("memory") + .queryName(sink) + .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .start() + + assert( + query.awaitTermination(120000), + "streaming read did not finish in 120 seconds") + assert( + countOf(table.spark, s"SELECT count(*) FROM $sink") == "3", + "streaming read should deliver the three seed rows") + } - assert( - query.awaitTermination(120000), - "streaming read did not finish in 120 seconds") - assert( - countOf(table.spark, s"SELECT count(*) FROM $sink") == "3", - "streaming read should deliver the three seed rows") - }, - surfaceBasePreparation(format).test( - "surface.stream.write", - "A Spark structured streaming append of two rows through the iceberg write-stream " + - "format lands both rows, growing the table from 3 to 5 rows.") { table => - import table.spark.implicits._ - implicit val sqlContext: org.apache.spark.sql.SQLContext = - table.spark.sqlContext - val memoryStream = - org.apache.spark.sql.execution.streaming.MemoryStream[Long] - memoryStream.addData(100L, 101L) - val rows = memoryStream.toDF().selectExpr( - s"value AS ${Core.long0.columnName}", - s"CAST(value AS INT) AS ${Core.int0.columnName}", - s"concat('row-', value) AS ${Core.string0.columnName}", - s"CAST(value AS DOUBLE) AS ${Core.double0.columnName}", - s"true AS ${Core.boolean0.columnName}", - s"'2024-01-01-00' AS ${Core.datePartition.columnName}") - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-write").toString - val query = rows.writeStream - .format("iceberg") - .outputMode("append") - .option("checkpointLocation", checkpoint) - .toTable(table.name) - - query.processAllAvailable() - query.stop() - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "5", - "streaming write should append two rows") - }, - surfaceTwoSnapshotPreparation(format).test( - "surface.cdc.changelogView", - "create_changelog_view over an append-only history reports 5 changes, all of change " + - "type INSERT.") { table => - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}')") - .collect()(0) - .getString(0) - val changeCount = table.spark - .sql(s"SELECT count(*) FROM $view") - .collect()(0) - .getLong(0) - val changeTypes = table.spark - .sql(s"SELECT DISTINCT _change_type FROM $view") - .collect() - .map(_.getString(0)) - .toSet + /** + * A Spark structured streaming append of two rows through the iceberg write-stream format lands + * both rows, growing the table from 3 to 5 rows. + */ + private def surfaceStreamWriteCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.stream.write") { table => + import table.spark.implicits._ + implicit val sqlContext: org.apache.spark.sql.SQLContext = + table.spark.sqlContext + val memoryStream = + org.apache.spark.sql.execution.streaming.MemoryStream[Long] + memoryStream.addData(100L, 101L) + val rows = memoryStream.toDF().selectExpr( + s"value AS ${Core.long0.columnName}", + s"CAST(value AS INT) AS ${Core.int0.columnName}", + s"concat('row-', value) AS ${Core.string0.columnName}", + s"CAST(value AS DOUBLE) AS ${Core.double0.columnName}", + s"true AS ${Core.boolean0.columnName}", + s"'2024-01-01-00' AS ${Core.datePartition.columnName}") + val checkpoint = + java.nio.file.Files.createTempDirectory("ck-write").toString + val query = rows.writeStream + .format("iceberg") + .outputMode("append") + .option("checkpointLocation", checkpoint) + .toTable(table.name) + + query.processAllAvailable() + query.stop() + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "5", + "streaming write should append two rows") + } - assert( - changeCount == 5, - s"append-only changelog should contain 5 changes, got $changeCount") - assert( - changeTypes == Set("INSERT"), - s"append-only changelog should contain only INSERT: $changeTypes") - }) + /** + * create_changelog_view over an append-only history reports 5 changes, all of change type + * INSERT. + */ + private def surfaceCdcChangelogViewCase(format: String): Plan.Case = + surfaceTwoSnapshotPreparation(format).test("surface.cdc.changelogView") { table => + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}')") + .collect()(0) + .getString(0) + val changeCount = table.spark + .sql(s"SELECT count(*) FROM $view") + .collect()(0) + .getLong(0) + val changeTypes = table.spark + .sql(s"SELECT DISTINCT _change_type FROM $view") + .collect() + .map(_.getString(0)) + .toSet + + assert( + changeCount == 5, + s"append-only changelog should contain 5 changes, got $changeCount") + assert( + changeTypes == Set("INSERT"), + s"append-only changelog should contain only INSERT: $changeTypes") + } - // The rewrite procedure that compacts the manifest set. - def surfaceRewriteProcedureCases(format: String): List[Plan.Case] = + /** The structured-streaming reader and writer, and the changelog view. */ + def surfaceReaderCases(format: String): List[Plan.Case] = List( - surfaceEmptyPreparation(format).test( - "surface.proc.rewriteManifests", - "After 5 single-row inserts fragment the manifest list, rewrite_manifests compacts it " + - "to fewer manifests while preserving all 5 rows.") { table => - (1 to 5).foreach(index => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - coreRow(index, s"r$index"))) - val manifestCountBefore = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.manifests") - .collect()(0) - .getLong(0) + surfaceStreamReadCase(format), + surfaceStreamWriteCase(format), + surfaceCdcChangelogViewCase(format)) + + /** + * After 5 single-row inserts fragment the manifest list, rewrite_manifests compacts it to fewer + * manifests while preserving all 5 rows. + */ + private def surfaceProcRewriteManifestsCase(format: String): Plan.Case = + surfaceEmptyPreparation(format).test("surface.proc.rewriteManifests") { table => + (1 to 5).foreach(index => table.spark.sql( - "CALL openhouse.system.rewrite_manifests(" + - s"table => '${catalogRelative(table.name)}', " + - "use_caching => false)") - val manifestCountAfter = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.manifests") - .collect()(0) - .getLong(0) - - println( - "DIAG surface.proc.rewriteManifests: " + - s"manifests before=$manifestCountBefore after=$manifestCountAfter") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "5", - "rewrite_manifests should preserve the five rows") - assert( - manifestCountBefore >= 2 && - manifestCountAfter < manifestCountBefore, - "rewrite_manifests should compact the manifest set") - }) + s"INSERT INTO ${table.name} VALUES " + + coreRow(index, s"r$index"))) + val manifestCountBefore = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.manifests") + .collect()(0) + .getLong(0) + table.spark.sql( + "CALL openhouse.system.rewrite_manifests(" + + s"table => '${catalogRelative(table.name)}', " + + "use_caching => false)") + val manifestCountAfter = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.manifests") + .collect()(0) + .getLong(0) + + println( + "DIAG surface.proc.rewriteManifests: " + + s"manifests before=$manifestCountBefore after=$manifestCountAfter") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "5", + "rewrite_manifests should preserve the five rows") + assert( + manifestCountBefore >= 2 && + manifestCountAfter < manifestCountBefore, + "rewrite_manifests should compact the manifest set") + } - // The procedures that read snapshot ancestry and remove orphan files. - def surfaceSnapshotProcedureCases(format: String): List[Plan.Case] = + /** + * The rewrite procedure that compacts the manifest set. The case starts from an unseeded table + * in the given file format and fragments the manifest list itself. + */ + def surfaceRewriteProcedureCases(format: String): List[Plan.Case] = List( - surfaceTwoSnapshotPreparation(format).test( - "surface.proc.ancestorsOf", - "ancestors_of lists both snapshots of the table's two-snapshot history.") { table => - val ancestorCount = table.spark - .sql( - "CALL openhouse.system.ancestors_of(" + - s"table => '${catalogRelative(table.name)}')") - .collect() - .length - - assert( - ancestorCount == 2, - s"ancestors_of should list two snapshots, got $ancestorCount") - }, - surfaceBasePreparation(format).test( - "surface.proc.removeOrphanReal", - "remove_orphan_files deletes a planted, backdated stray file next to a real data file " + - "while the table's 3 live rows remain intact.") { table => - val dataFile = table.spark - .sql(s"SELECT file_path FROM ${table.name}.files LIMIT 1") - .collect()(0) - .getString(0) - .stripPrefix("file:") - val orphanFile = java.nio.file.Paths - .get(dataFile) - .getParent - .resolve("zz_orphan_plant.parquet") - java.nio.file.Files.write( - orphanFile, - "not-a-real-parquet".getBytes) - java.nio.file.Files.setLastModifiedTime( - orphanFile, - java.nio.file.attribute.FileTime.fromMillis(1546300800000L)) + surfaceProcRewriteManifestsCase(format)) + + /** ancestors_of lists both snapshots of the table's two-snapshot history. */ + private def surfaceProcAncestorsOfCase(format: String): Plan.Case = + surfaceTwoSnapshotPreparation(format).test("surface.proc.ancestorsOf") { table => + val ancestorCount = table.spark + .sql( + "CALL openhouse.system.ancestors_of(" + + s"table => '${catalogRelative(table.name)}')") + .collect() + .length + + assert( + ancestorCount == 2, + s"ancestors_of should list two snapshots, got $ancestorCount") + } - table.spark.sql( - "CALL openhouse.system.remove_orphan_files(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2020-01-01 00:00:00')") - assert( - java.nio.file.Files.notExists(orphanFile), - "remove_orphan_files should delete the planted orphan") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "remove_orphan_files should preserve live data") - }) + /** + * remove_orphan_files deletes a planted, backdated stray file next to a real data file while the + * table's 3 live rows remain intact. + */ + private def surfaceProcRemoveOrphanRealCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.proc.removeOrphanReal") { table => + val dataFile = table.spark + .sql(s"SELECT file_path FROM ${table.name}.files LIMIT 1") + .collect()(0) + .getString(0) + .stripPrefix("file:") + val orphanFile = java.nio.file.Paths + .get(dataFile) + .getParent + .resolve("zz_orphan_plant.parquet") + java.nio.file.Files.write( + orphanFile, + "not-a-real-parquet".getBytes) + java.nio.file.Files.setLastModifiedTime( + orphanFile, + java.nio.file.attribute.FileTime.fromMillis(1546300800000L)) + + table.spark.sql( + "CALL openhouse.system.remove_orphan_files(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2020-01-01 00:00:00')") + assert( + java.nio.file.Files.notExists(orphanFile), + "remove_orphan_files should delete the planted orphan") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "remove_orphan_files should preserve live data") + } - // The hidden metadata columns and the Iceberg metadata tables. - def surfaceMetadataCases(format: String): List[Plan.Case] = + /** + * The procedures that read snapshot ancestry and remove orphan files. Ancestry runs on a + * two-snapshot table and orphan removal on a seeded table, each in the given file format. + */ + def surfaceSnapshotProcedureCases(format: String): List[Plan.Case] = List( - surfaceBasePreparation(format).test( - "surface.meta.hiddenColumns", - "Selecting the hidden metadata columns _file, _pos, _spec_id and _partition returns " + - "one row per seed row, each with a populated file path and a non-negative position.") { - table => - val rows = table.spark + surfaceProcAncestorsOfCase(format), + surfaceProcRemoveOrphanRealCase(format)) + + /** + * Selecting the hidden metadata columns _file, _pos, _spec_id and _partition returns one row per + * seed row, each with a populated file path and a non-negative position. + */ + private def surfaceMetaHiddenColumnsCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.meta.hiddenColumns") { + table => + val rows = table.spark + .sql( + s"SELECT _file, _pos, _spec_id, _partition FROM ${table.name}") + .collect() + .toSeq + + assert( + rows.size == 3, + s"hidden metadata columns should return 3 rows, got ${rows.size}") + assert( + rows.forall(row => + Option(row.getString(0)).exists(_.nonEmpty)), + "_file should be populated for every row") + assert( + rows.forall(_.getLong(1) >= 0), + "_pos should be non-negative for every row") + } + + /** + * Every Iceberg metadata table (entries, files, manifests, snapshots, history, refs, partitions, + * and their all_* variants) is queryable without error, and the snapshots metadata table reports + * the table's 2 snapshots. + */ + private def surfaceMetaTableSweepCase(format: String): Plan.Case = + surfaceTwoSnapshotPreparation(format).test("surface.meta.tableSweep") { table => + val metadataTables = Seq( + "entries", + "files", + "manifests", + "snapshots", + "history", + "refs", + "partitions", + "metadata_log_entries", + "data_files", + "all_data_files", + "all_manifests", + "all_entries", + "all_files") + metadataTables.foreach { metadataTable => + table.spark .sql( - s"SELECT _file, _pos, _spec_id, _partition FROM ${table.name}") + s"SELECT count(*) FROM ${table.name}.`$metadataTable`") .collect() - .toSeq - - assert( - rows.size == 3, - s"hidden metadata columns should return 3 rows, got ${rows.size}") - assert( - rows.forall(row => - Option(row.getString(0)).exists(_.nonEmpty)), - "_file should be populated for every row") - assert( - rows.forall(_.getLong(1) >= 0), - "_pos should be non-negative for every row") - }, - surfaceTwoSnapshotPreparation(format).test( - "surface.meta.tableSweep", - "Every Iceberg metadata table (entries, files, manifests, snapshots, history, refs, " + - "partitions, and their all_* variants) is queryable without error, and the snapshots " + - "metadata table reports the table's 2 snapshots.") { table => - val metadataTables = Seq( - "entries", - "files", - "manifests", - "snapshots", - "history", - "refs", - "partitions", - "metadata_log_entries", - "data_files", - "all_data_files", - "all_manifests", - "all_entries", - "all_files") - metadataTables.foreach { metadataTable => - table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.`$metadataTable`") - .collect() - } - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots") == "2", - "snapshot metadata should contain two snapshots") - }) - - // Two writers racing on one table. Every outcome is either a commit or a typed commit conflict. - def surfaceConcurrencyCases(format: String): List[Plan.Case] = - List( - surfaceBasePreparation(format).test( - "surface.conc.appendAppend", - "Two threads concurrently insert 3 rows each; every insert either commits or fails " + - "with a typed commit-conflict exception, and the final row count matches 3 plus the " + - "number of inserts that actually committed.") { table => - val failureCount = - new java.util.concurrent.atomic.AtomicInteger(0) - def writer(base: Int): () => Unit = () => - (0 until 3).foreach { offset => - val value = base + offset - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - s"(CAST($value AS BIGINT), $value, 'row-c', 1.5, " + - "true, '2024-01-09-01')") - } catch { - case exception: Throwable => - assert( - isTypedCommitConflict(exception), - "concurrent append failed with an untyped error: " + - s"${exception.getClass.getName}") - failureCount.incrementAndGet() - } - } - val threadErrors = - runConcurrently(Seq(writer(100), writer(200))) - val expectedRowCount = 3 + 6 - failureCount.get - val actualRowCount = countOf( + } + assert( + countOf( table.spark, - s"SELECT count(*) FROM ${table.name}") + s"SELECT count(*) FROM ${table.name}.snapshots") == "2", + "snapshot metadata should contain two snapshots") + } - assert( - threadErrors.isEmpty, - s"writer thread failed outside the insert loop: $threadErrors") - assert( - actualRowCount == expectedRowCount.toString, - s"expected $expectedRowCount rows, got $actualRowCount") - println( - s"DIAG conc.appendAppend: ${failureCount.get}/6 inserts " + - "hit a typed commit conflict") - }, - surfaceBasePreparation(format).test( - "surface.conc.updateUpdate", - "Two threads concurrently UPDATE the same row to different values; the row count stays " + - "at 3, and the final value is one of the two competing updates or the original seed " + - "value, with any failure being a typed commit conflict.") { table => - val column = Core.string0.columnName - def updater(value: String): () => Unit = () => + /** The hidden metadata columns and the Iceberg metadata tables. */ + def surfaceMetadataCases(format: String): List[Plan.Case] = + List( + surfaceMetaHiddenColumnsCase(format), + surfaceMetaTableSweepCase(format)) + + /** + * Two threads concurrently insert 3 rows each; every insert either commits or fails with a typed + * commit-conflict exception, and the final row count matches 3 plus the number of inserts that + * actually committed. + */ + private def surfaceConcAppendAppendCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.conc.appendAppend") { table => + val failureCount = + new java.util.concurrent.atomic.AtomicInteger(0) + def writer(base: Int): () => Unit = () => + (0 until 3).foreach { offset => + val value = base + offset try { table.spark.sql( - s"UPDATE ${table.name} SET $column = '$value' " + - s"WHERE ${Core.long0.columnName} = 2") + s"INSERT INTO ${table.name} VALUES " + + s"(CAST($value AS BIGINT), $value, 'row-c', 1.5, " + + "true, '2024-01-09-01')") } catch { case exception: Throwable => assert( isTypedCommitConflict(exception), - "concurrent update failed with an untyped error: " + + "concurrent append failed with an untyped error: " + s"${exception.getClass.getName}") + failureCount.incrementAndGet() } - val threadErrors = - runConcurrently(Seq(updater("AAA"), updater("BBB"))) - val finalValue = table.spark - .sql( - s"SELECT $column FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 2") - .collect()(0) - .getString(0) - - assert( - threadErrors.isEmpty, - s"updater thread failed with a non-conflict error: $threadErrors") - assert( - finalValue == "AAA" || - finalValue == "BBB" || - finalValue == "row-2", - s"concurrent updates produced a torn value: $finalValue") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "concurrent updates should not change row count") - }) - - // Schema changes that Iceberg allows and the ones the catalog rejects. - def surfaceSchemaCases(format: String): List[Plan.Case] = - List( - surfaceBasePreparation(format).test( - "surface.schema.relaxNotNull", - "On a side table, dropping NOT NULL from a column allows a subsequent insert of a null " + - "value for that column.") { table => - val sideTable = s"${table.name}_nn" - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - try { - table.spark.sql( - s"CREATE TABLE $sideTable " + - s"(id BIGINT, req INT NOT NULL) USING $dataSource") - table.spark.sql( - s"ALTER TABLE $sideTable ALTER COLUMN req DROP NOT NULL") - table.spark.sql( - s"INSERT INTO $sideTable VALUES (CAST(1 AS BIGINT), NULL)") - assert( - table.spark - .sql(s"SELECT count(*) FROM $sideTable WHERE req IS NULL") - .collect()(0) - .getLong(0) == 1, - "relaxing NOT NULL should allow a null write") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") } - }, - surfaceBasePreparation(format).test( - "surface.schema.decimalWiden", - "On a side table, widening a decimal column's precision preserves the original row and " + - "accepts a new row whose value only fits the wider precision.") { table => - val sideTable = s"${table.name}_dec" - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + val threadErrors = + runConcurrently(Seq(writer(100), writer(200))) + val expectedRowCount = 3 + 6 - failureCount.get + val actualRowCount = countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") + + assert( + threadErrors.isEmpty, + s"writer thread failed outside the insert loop: $threadErrors") + assert( + actualRowCount == expectedRowCount.toString, + s"expected $expectedRowCount rows, got $actualRowCount") + println( + s"DIAG conc.appendAppend: ${failureCount.get}/6 inserts " + + "hit a typed commit conflict") + } + + /** + * Two threads concurrently UPDATE the same row to different values; the row count stays at 3, + * and the final value is one of the two competing updates or the original seed value, with any + * failure being a typed commit conflict. + */ + private def surfaceConcUpdateUpdateCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.conc.updateUpdate") { table => + val column = Core.string0.columnName + def updater(value: String): () => Unit = () => try { table.spark.sql( - s"CREATE TABLE $sideTable " + - s"(id BIGINT, dec DECIMAL(10,2)) USING $dataSource") - table.spark.sql( - s"INSERT INTO $sideTable VALUES " + - "(CAST(1 AS BIGINT), CAST(12345678.99 AS DECIMAL(10,2)))") - table.spark.sql( - s"ALTER TABLE $sideTable ALTER COLUMN dec TYPE DECIMAL(12,2)") - table.spark.sql( - s"INSERT INTO $sideTable VALUES " + - "(CAST(2 AS BIGINT), CAST(1234567890.99 AS DECIMAL(12,2)))") - assert( - table.spark - .sql(s"SELECT count(*) FROM $sideTable") - .collect()(0) - .getLong(0) == 2, - "decimal widening should preserve old and new values") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + s"UPDATE ${table.name} SET $column = '$value' " + + s"WHERE ${Core.long0.columnName} = 2") + } catch { + case exception: Throwable => + assert( + isTypedCommitConflict(exception), + "concurrent update failed with an untyped error: " + + s"${exception.getClass.getName}") } - }, - surfaceBasePreparation(format).test( - "surface.schema.nestedAddField", - "On a side table, ADD COLUMN of a new nested struct field null-fills it for the " + - "existing row and accepts a new row that sets the field.") { table => - val sideTable = s"${table.name}_nst" - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - try { - table.spark.sql( - s"CREATE TABLE $sideTable " + - s"(id BIGINT, s STRUCT) USING $dataSource") - table.spark.sql( - s"INSERT INTO $sideTable VALUES " + - "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") - table.spark.sql( - s"ALTER TABLE $sideTable ADD COLUMN s.w INT") - assert( - table.spark - .sql(s"SELECT count(*) FROM $sideTable WHERE s.w IS NULL") - .collect()(0) - .getLong(0) == 1, - "new nested field should null-fill the existing row") + val threadErrors = + runConcurrently(Seq(updater("AAA"), updater("BBB"))) + val finalValue = table.spark + .sql( + s"SELECT $column FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 2") + .collect()(0) + .getString(0) + + assert( + threadErrors.isEmpty, + s"updater thread failed with a non-conflict error: $threadErrors") + assert( + finalValue == "AAA" || + finalValue == "BBB" || + finalValue == "row-2", + s"concurrent updates produced a torn value: $finalValue") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "concurrent updates should not change row count") + } - table.spark.sql( - s"INSERT INTO $sideTable VALUES " + - "(CAST(2 AS BIGINT), " + - "named_struct('x', 2, 'y', 'b', 'w', 9))") - assert( - table.spark - .sql(s"SELECT count(*) FROM $sideTable WHERE s.w = 9") - .collect()(0) - .getLong(0) == 1, - "new nested field should be writable") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - } - }, - surfaceBasePreparation(format).test( - "surface.schema.nestedDropField", - "On a side table, ALTER TABLE DROP COLUMN of a nested struct field is rejected with an " + - "exception, and the field remains readable afterward.") { table => - val sideTable = s"${table.name}_nsd" + /** + * Two writers racing on one table. Every outcome is either a commit or a typed commit conflict. + */ + def surfaceConcurrencyCases(format: String): List[Plan.Case] = + List( + surfaceConcAppendAppendCase(format), + surfaceConcUpdateUpdateCase(format)) + + /** + * On a side table, dropping NOT NULL from a column allows a subsequent insert of a null value + * for that column. + */ + private def surfaceSchemaRelaxNotNullCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.schema.relaxNotNull") { table => + val sideTable = s"${table.name}_nn" + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + try { + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, req INT NOT NULL) USING $dataSource") + table.spark.sql( + s"ALTER TABLE $sideTable ALTER COLUMN req DROP NOT NULL") + table.spark.sql( + s"INSERT INTO $sideTable VALUES (CAST(1 AS BIGINT), NULL)") + assert( + table.spark + .sql(s"SELECT count(*) FROM $sideTable WHERE req IS NULL") + .collect()(0) + .getLong(0) == 1, + "relaxing NOT NULL should allow a null write") + } finally { table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - try { - table.spark.sql( - s"CREATE TABLE $sideTable " + - s"(id BIGINT, s STRUCT) USING $dataSource") - table.spark.sql( - s"INSERT INTO $sideTable VALUES " + - "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") - Check.intercept[Exception]( - table.spark.sql( - s"ALTER TABLE $sideTable DROP COLUMN s.x")) - - assert( - table.spark - .sql(s"SELECT s.x FROM $sideTable") - .collect()(0) - .getInt(0) == 1, - "rejected nested drop should leave the field readable") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - } - }, - surfaceBasePreparation(format).test( - "surface.schema.reorderExisting", - "ALTER TABLE ALTER COLUMN ... FIRST moves that column to the front of the schema while " + - "preserving all 3 rows.") { table => + } + } + + /** + * On a side table, widening a decimal column's precision preserves the original row and accepts + * a new row whose value only fits the wider precision. + */ + private def surfaceSchemaDecimalWidenCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.schema.decimalWiden") { table => + val sideTable = s"${table.name}_dec" + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + try { table.spark.sql( - s"ALTER TABLE ${table.name} " + - s"ALTER COLUMN ${Core.string0.columnName} FIRST") - val columns = table.spark - .sql(s"SELECT * FROM ${table.name} LIMIT 1") - .columns - .toSeq + s"CREATE TABLE $sideTable " + + s"(id BIGINT, dec DECIMAL(10,2)) USING $dataSource") + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(1 AS BIGINT), CAST(12345678.99 AS DECIMAL(10,2)))") + table.spark.sql( + s"ALTER TABLE $sideTable ALTER COLUMN dec TYPE DECIMAL(12,2)") + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(2 AS BIGINT), CAST(1234567890.99 AS DECIMAL(12,2)))") + assert( + table.spark + .sql(s"SELECT count(*) FROM $sideTable") + .collect()(0) + .getLong(0) == 2, + "decimal widening should preserve old and new values") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + } + } + /** + * On a side table, ADD COLUMN of a new nested struct field null-fills it for the existing row + * and accepts a new row that sets the field. + */ + private def surfaceSchemaNestedAddFieldCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.schema.nestedAddField") { table => + val sideTable = s"${table.name}_nst" + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + try { + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, s STRUCT) USING $dataSource") + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") + table.spark.sql( + s"ALTER TABLE $sideTable ADD COLUMN s.w INT") assert( - columns.head == Core.string0.columnName, - s"FIRST should move the column to the front: $columns") + table.spark + .sql(s"SELECT count(*) FROM $sideTable WHERE s.w IS NULL") + .collect()(0) + .getLong(0) == 1, + "new nested field should null-fill the existing row") + + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(2 AS BIGINT), " + + "named_struct('x', 2, 'y', 'b', 'w', 9))") assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "column reorder should preserve the rows") - }) + table.spark + .sql(s"SELECT count(*) FROM $sideTable WHERE s.w = 9") + .collect()(0) + .getLong(0) == 1, + "new nested field should be writable") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + } + } - // The write-planning properties: distribution mode and target file size. - def surfaceWriteCases(format: String): List[Plan.Case] = - List( - surfaceHashPreparation(format).test( - "surface.write.distributionHash", - "The write.distribution-mode=hash property requested at creation is retained and the " + - "table holds its 3 seed rows.") { table => - val properties = tableProps(table.spark, table.name) - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) + /** + * On a side table, ALTER TABLE DROP COLUMN of a nested struct field is rejected with an + * exception, and the field remains readable afterward. + */ + private def surfaceSchemaNestedDropFieldCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.schema.nestedDropField") { table => + val sideTable = s"${table.name}_nsd" + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + try { + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, s STRUCT) USING $dataSource") + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") + Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE $sideTable DROP COLUMN s.x")) assert( - properties.get("write.distribution-mode").contains("hash"), - "hash distribution mode should be retained") - assert( - rowCount == 3, - s"hash-distributed seed should contain 3 rows, got $rowCount") - }, - surfaceTargetFileSizePreparation(format).test( - "surface.write.targetFileSize", - "The write.target-file-size-bytes=1048576 property requested at creation is retained " + - "and the table holds its 3 seed rows.") { table => - val properties = tableProps(table.spark, table.name) - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) + table.spark + .sql(s"SELECT s.x FROM $sideTable") + .collect()(0) + .getInt(0) == 1, + "rejected nested drop should leave the field readable") + } finally { + table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") + } + } - assert( - properties - .get("write.target-file-size-bytes") - .contains("1048576"), - "target file size should be retained") - assert( - rowCount == 3, - s"custom target-size seed should contain 3 rows, got $rowCount") - }) + /** + * ALTER TABLE ALTER COLUMN ... FIRST moves that column to the front of the schema while + * preserving all 3 rows. + */ + private def surfaceSchemaReorderExistingCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.schema.reorderExisting") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} " + + s"ALTER COLUMN ${Core.string0.columnName} FIRST") + val columns = table.spark + .sql(s"SELECT * FROM ${table.name} LIMIT 1") + .columns + .toSeq + + assert( + columns.head == Core.string0.columnName, + s"FIRST should move the column to the front: $columns") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "column reorder should preserve the rows") + } - // Pins on the surfaces the catalog rejects: the import procedures, views and ANALYZE TABLE. - def surfacePinCases(format: String): List[Plan.Case] = + /** The schema changes Iceberg allows and the ones the catalog rejects. */ + def surfaceSchemaCases(format: String): List[Plan.Case] = List( - surfaceBasePreparation(format).test( - "surface.pin.importProcs", - "register_table onto a new name makes the source table's snapshot readable there " + - "(3 rows) without affecting the source, and dropping the registered table leaves " + - "the source untouched; the system.snapshot and system.add_files procedures are " + - "each confirmed to reject their unsupported inputs with an exception.") { table => - val registeredTable = s"${table.name}_registered" - val metadataFile = table.spark - .sql( - s"SELECT file FROM ${table.name}.metadata_log_entries " + - "ORDER BY timestamp DESC LIMIT 1") - .collect()(0) - .getString(0) + surfaceSchemaRelaxNotNullCase(format), + surfaceSchemaDecimalWidenCase(format), + surfaceSchemaNestedAddFieldCase(format), + surfaceSchemaNestedDropFieldCase(format), + surfaceSchemaReorderExistingCase(format)) + + /** + * The write.distribution-mode=hash property requested at creation is retained and the table + * holds its 3 seed rows. + */ + private def surfaceWriteDistributionHashCase(format: String): Plan.Case = + surfaceHashPreparation(format).test("surface.write.distributionHash") { table => + val properties = tableProps(table.spark, table.name) + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + properties.get("write.distribution-mode").contains("hash"), + "hash distribution mode should be retained") + assert( + rowCount == 3, + s"hash-distributed seed should contain 3 rows, got $rowCount") + } - try { - table.spark.sql( - "CALL openhouse.system.register_table(" + - s"table => '${catalogRelative(registeredTable)}', " + - s"metadata_file => '$metadataFile')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $registeredTable") == "3", - "register_table should make all source rows readable") - } finally { - try { - table.spark.sql( - s"DROP TABLE IF EXISTS $registeredTable") - } catch { - case NonFatal(_) => () - } - } + /** + * The write.target-file-size-bytes=1048576 property requested at creation is retained and the + * table holds its 3 seed rows. + */ + private def surfaceWriteTargetFileSizeCase(format: String): Plan.Case = + surfaceTargetFileSizePreparation(format).test("surface.write.targetFileSize") { table => + val properties = tableProps(table.spark, table.name) + val rowCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}") + .collect()(0) + .getLong(0) + + assert( + properties + .get("write.target-file-size-bytes") + .contains("1048576"), + "target file size should be retained") + assert( + rowCount == 3, + s"custom target-size seed should contain 3 rows, got $rowCount") + } + + /** The write-planning properties: distribution mode and target file size. */ + def surfaceWriteCases(format: String): List[Plan.Case] = + List( + surfaceWriteDistributionHashCase(format), + surfaceWriteTargetFileSizeCase(format)) + + /** + * register_table onto a new name makes the source table's snapshot readable there (3 rows) and + * leaves the source unchanged, and dropping the registered table leaves the source unchanged. + * The system.snapshot and system.add_files procedures each reject their unsupported inputs with + * an exception. + */ + private def surfacePinImportProcsCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.pin.importProcs") { table => + val registeredTable = s"${table.name}_registered" + val metadataFile = table.spark + .sql( + s"SELECT file FROM ${table.name}.metadata_log_entries " + + "ORDER BY timestamp DESC LIMIT 1") + .collect()(0) + .getString(0) + + try { + table.spark.sql( + "CALL openhouse.system.register_table(" + + s"table => '${catalogRelative(registeredTable)}', " + + s"metadata_file => '$metadataFile')") assert( countOf( table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "dropping the registered table should not remove source rows") - - Check.intercept[Exception]( + s"SELECT count(*) FROM $registeredTable") == "3", + "register_table should make all source rows readable") + } finally { + try { table.spark.sql( - "CALL openhouse.system.snapshot(" + - s"source_table => '${catalogRelative(table.name)}', " + - "table => 'dbMatrix.zz_snap')")) + s"DROP TABLE IF EXISTS $registeredTable") + } catch { + case NonFatal(_) => () + } + } + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name}") == "3", + "dropping the registered table should not remove source rows") - Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.add_files(" + - s"table => '${catalogRelative(table.name)}', " + - "source_table => '`parquet`.`/tmp/zz_nope_dir`')")) - }, - surfaceBasePreparation(format).test( - "surface.pin.viewsAnalyze", - "CREATE VIEW and ANALYZE TABLE COMPUTE STATISTICS are each rejected with an " + - "exception.") { table => - Check.intercept[Exception]( - table.spark.sql( - "CREATE VIEW openhouse.dbMatrix.zz_v1 AS SELECT 1 AS one")) + Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.snapshot(" + + s"source_table => '${catalogRelative(table.name)}', " + + "table => 'dbMatrix.zz_snap')")) - Check.intercept[Exception]( - table.spark.sql( - s"ANALYZE TABLE ${table.name} COMPUTE STATISTICS")) - }) + Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.add_files(" + + s"table => '${catalogRelative(table.name)}', " + + "source_table => '`parquet`.`/tmp/zz_nope_dir`')")) + } + + /** CREATE VIEW and ANALYZE TABLE COMPUTE STATISTICS are each rejected with an exception. */ + private def surfacePinViewsAnalyzeCase(format: String): Plan.Case = + surfaceBasePreparation(format).test("surface.pin.viewsAnalyze") { table => + Check.intercept[Exception]( + table.spark.sql( + "CREATE VIEW openhouse.dbMatrix.zz_v1 AS SELECT 1 AS one")) + + Check.intercept[Exception]( + table.spark.sql( + s"ANALYZE TABLE ${table.name} COMPUTE STATISTICS")) + } + + /** Pins on the surfaces the catalog rejects: the import procedures, views and ANALYZE TABLE. */ + def surfacePinCases(format: String): List[Plan.Case] = + List( + surfacePinImportProcsCase(format), + surfacePinViewsAnalyzeCase(format)) } diff --git a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala index 9622c9d4b..9870804d5 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala @@ -23,9 +23,6 @@ final class CaseCatalogTest { assertTrue( duplicateCaseIds.isEmpty, s"case IDs must be unique; duplicates=${duplicateCaseIds.mkString(", ")}") - assertTrue( - cases.forall(_.description.trim.nonEmpty), - "every catalog case must describe the behavior it verifies") assertEquals( expectedCaseCount, caseIds.size, diff --git a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala index c32dec2a4..d8bcf4abc 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala @@ -143,12 +143,59 @@ final class DmlCaseCatalogTest { } @Test - def theNullStringPreparationDescribesTheRowItAppends(): Unit = { - Scenarios.preparedNullStringCoreTables.foreach { preparation => - assertTrue( - preparation.description.contains("null"), - s"${preparation.label} does not describe the null-string row it appends") + def theNullStringPreparationsExtendTheCorePreparations(): Unit = { + assertEquals( + Scenarios.preparedCoreTables.map(preparation => (preparation.casePrefix, preparation.label)), + Scenarios.preparedNullStringCoreTables.map(preparation => + (preparation.casePrefix, preparation.label))) + assertEquals( + Scenarios.preparedCoreTables.map(_.preparation.steps.size + 1), + Scenarios.preparedNullStringCoreTables.map(_.preparation.steps.size)) + assertEquals( + List("prep.nullStringRow"), + Scenarios.preparedNullStringCoreTables.head.preparation.steps.map(_.label).toList.takeRight(1)) + } + + @Test + def everyDmlCaseIdNamesItsOperationAndItsPreparation(): Unit = { + val describedBuckets = List( + Scenarios.coreDmlCases, + Scenarios.orderedDmlCases, + Scenarios.evolvedDmlCases, + Scenarios.partitionedDmlCases, + Scenarios.layoutFormatCases).flatten + val caseIds = describedBuckets.map(_.id) + + caseIds.foreach { caseId => + assertEquals( + 2, + caseId.split(" @ ").length, + s"$caseId must be an operation name, then ' @ ', then a preparation label") } + assertEquals(caseIds.distinct.size, caseIds.size, "DML case IDs must be unique") + } + + @Test + def eachLayoutListCrossesItsFormatsWithItsPartitionings(): Unit = { + assertEquals( + List( + "unpartitioned/parquet", + "partitioned/parquet", + "unpartitioned/orc", + "partitioned/orc", + "unpartitioned/avro", + "partitioned/avro"), + Scenarios.layouts.map(_.label)) + assertEquals( + List("partitioned/parquet", "partitioned/orc", "partitioned/avro"), + Scenarios.partitionedLayouts.map(_.label)) + assertEquals( + List( + "unpartitioned/parquet", + "partitioned/parquet", + "unpartitioned/orc", + "partitioned/orc"), + Scenarios.parquetAndOrcLayouts.map(_.label)) } @Test @@ -182,45 +229,6 @@ final class DmlCaseCatalogTest { } } - @Test - def everyDmlCaseCarriesItsOwnDescriptionAndItsPreparationDescription(): Unit = { - val describedBuckets = List( - Scenarios.coreDmlCases, - Scenarios.orderedDmlCases, - Scenarios.evolvedDmlCases, - Scenarios.partitionedDmlCases, - Scenarios.layoutFormatCases).flatten - - describedBuckets.foreach { testCase => - assertTrue( - testCase.description.trim.nonEmpty, - s"${testCase.id} has no description of the operation it runs") - assertTrue( - testCase.preparationDescription.trim.nonEmpty, - s"${testCase.id} has no description of the state it starts from") - assertTrue( - testCase.description != testCase.id, - s"${testCase.id} repeats its id; the description must explain the operation") - } - } - - @Test - def everyLayoutDescribesTheTableItCreates(): Unit = { - val describedLayouts = - Scenarios.layouts ++ - Scenarios.partitionedLayouts ++ - Scenarios.parquetAndOrcLayouts - - describedLayouts.foreach { layout => - assertTrue( - layout.description.trim.nonEmpty, - s"layout ${layout.label} has no description") - assertTrue( - layout.description != layout.label, - s"layout ${layout.label} repeats its label; the description must explain the table") - } - } - private def caseIds( preparations: List[TablePreparation[CoreTable.type]], testCases: List[DmlTestCase[CoreTable.type]] diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala index 77bc12b9f..0a6ed45d1 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala @@ -1,49 +1,76 @@ package harness -import org.junit.jupiter.api.Assertions.assertEquals +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} import org.junit.jupiter.api.Test +/** + * Pins how a preparation turns a test body into a catalog case: the ID it builds, the post-test + * hook every case from that preparation runs, and the known-bug reason a DML test case carries into + * its cases. Building a case runs no SQL, so these assertions need no Spark session. + */ final class TablePreparationTest { + private val emptyPreparation = TableTest(CoreTable) + @Test def formatsCaseIdFromPrefixNameAndLabel(): Unit = { - val preparation = TablePreparation( - "partitioned/orc", - TableTest(CoreTable), - "prep.evolved:", - description = "Three rows in an evolved ORC table.") + val preparation = TablePreparation("partitioned/orc", emptyPreparation, "prep.evolved:") - val testCase = preparation.test( - "delete.byPredicate", - "DELETE removes the rows selected by its predicate.")(_ => ()) + val testCase = preparation.test("delete.byPredicate")(_ => ()) + + assertEquals("prep.evolved:delete.byPredicate @ partitioned/orc", testCase.id) + } + + @Test + def formatsCaseIdWithoutAPrefixWhenThePreparationDeclaresNone(): Unit = { + val preparation = TablePreparation("unpartitioned/parquet", emptyPreparation) assertEquals( - "prep.evolved:delete.byPredicate @ partitioned/orc", - testCase.id) - assertEquals( - "Three rows in an evolved ORC table.", - testCase.preparationDescription) - assertEquals( - "DELETE removes the rows selected by its predicate.", - testCase.description) + "insert.into @ unpartitioned/parquet", + preparation.test("insert.into")(_ => ()).id) } @Test - def runsDescribedDmlCaseOnPreparation(): Unit = { - val preparation = TablePreparation( + def buildsACaseWithoutRunningItsBodyOrItsPostTestHook(): Unit = { + val calls = scala.collection.mutable.ListBuffer.empty[String] + val preparation = TablePreparation[CoreTable.type]( "unpartitioned/parquet", - TableTest(CoreTable), - description = "Three rows in an unpartitioned Parquet table.") + emptyPreparation, + afterTest = _ => calls += "afterTest") + + preparation.test("insert.into")(_ => calls += "body") + + assertTrue(calls.isEmpty, s"building a case ran $calls") + } + + @Test + def runsADmlTestCaseUnderTheIdOfThePreparationItIsGiven(): Unit = { + val calls = scala.collection.mutable.ListBuffer.empty[String] + val preparation = TablePreparation("unpartitioned/parquet", emptyPreparation) + val dmlTestCase = DmlTestCase( + "insert.into", + (_: PreparedTable[CoreTable.type]) => calls += "insert.into") + + val testCase = dmlTestCase.runOn(preparation) + + assertEquals("insert.into @ unpartitioned/parquet", testCase.id) + assertEquals(None, testCase.knownBugReason) + assertTrue(calls.isEmpty, s"runOn ran the operation: $calls") + } + + @Test + def carriesTheKnownBugReasonOfADmlTestCaseIntoItsCase(): Unit = { + val preparation = TablePreparation("partitioned/orc", emptyPreparation, "prep.ordered:") val dmlTestCase = DmlTestCase( - "insert.append", - "INSERT appends one row and commits one snapshot.", - (_: PreparedTable[CoreTable.type]) => ()) + "delete.byPartitionPredicate", + (_: PreparedTable[CoreTable.type]) => (), + knownBugReason = Some("the rewrite crashes on a write-ordered table")) val testCase = dmlTestCase.runOn(preparation) + assertEquals("prep.ordered:delete.byPartitionPredicate @ partitioned/orc", testCase.id) + assertEquals(Some("the rewrite crashes on a write-ordered table"), testCase.knownBugReason) assertEquals( - "insert.append @ unpartitioned/parquet", - testCase.id) - assertEquals(dmlTestCase.description, testCase.description) - assertEquals(preparation.description, testCase.preparationDescription) + Some("bug: the rewrite crashes on a write-ordered table"), + Plan.bugReason(testCase)) } } diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala new file mode 100644 index 000000000..61a5a3442 --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala @@ -0,0 +1,23 @@ +package harness + +import org.junit.jupiter.api.Assertions.{assertNotEquals, assertTrue} +import org.junit.jupiter.api.Test + +/** + * Pins the table-name generator that gives every case its own table: each call mints a fresh UUID, + * so two names differ even when the counter is reset between them, and every name stays inside the + * namespace the caller asked for. + */ +final class TableTestTest { + @Test + def generatedTableNamesAreDistinctAndNamespaceScoped(): Unit = { + TableTest.seedCounter(0) + val firstTable = TableTest.nextQualifiedTableName("test_namespace") + TableTest.seedCounter(0) + val secondTable = TableTest.nextQualifiedTableName("test_namespace") + + assertTrue(firstTable.startsWith("test_namespace.t_")) + assertTrue(secondTable.startsWith("test_namespace.t_")) + assertNotEquals(firstTable, secondTable) + } +} From efd7af2eb05983a18959cc7b9a62500ec44331d2 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Thu, 27 Aug 2026 20:01:36 -0700 Subject: [PATCH 09/24] test(delta-harness): cover ownership cleanup Extract the owned-table cleanup state machine behind a package-private boundary so its failure paths can be tested without starting Spark. Pin conflict preservation, successful cleanup, and suppression of cleanup failure behind the primary test failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../scala/harness/openhouse/Framework.scala | 15 +++-- .../test/scala/harness/TableTestTest.scala | 57 +++++++++++++++++-- 2 files changed, 64 insertions(+), 8 deletions(-) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala index 7515b87ac..813c7ab4a 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala @@ -317,11 +317,19 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste // suppressed exception. private def withTable(ctx: Ctx)(use: (String, () => Unit) => Unit): Unit = { val table = TableTest.nextQualifiedTableName(ctx.namespace) - var tableCreated = false + OwnedTableLifecycle.withOwnership( + ctx.spark.sql(s"DROP TABLE IF EXISTS $table"))( + markTableCreated => use(table, markTableCreated)) + } + +} +private[harness] object OwnedTableLifecycle { + def withOwnership(dropOwnedTable: => Unit)(use: (() => Unit) => Unit): Unit = { + var tableCreated = false var testFailure: Option[Throwable] = None try { - use(table, () => tableCreated = true) + use(() => tableCreated = true) } catch { case failure: Throwable => testFailure = Some(failure) @@ -329,7 +337,7 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste } finally { if (tableCreated) { try { - ctx.spark.sql(s"DROP TABLE IF EXISTS $table") + dropOwnedTable } catch { case cleanupFailure: Throwable => testFailure match { @@ -340,7 +348,6 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste } } } - } object TableTest { diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala index 61a5a3442..40148f3c0 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala @@ -1,12 +1,19 @@ package harness -import org.junit.jupiter.api.Assertions.{assertNotEquals, assertTrue} +import org.junit.jupiter.api.Assertions.{ + assertEquals, + assertFalse, + assertNotEquals, + assertSame, + assertThrows, + assertTrue +} import org.junit.jupiter.api.Test /** - * Pins the table-name generator that gives every case its own table: each call mints a fresh UUID, - * so two names differ even when the counter is reset between them, and every name stays inside the - * namespace the caller asked for. + * Pins fresh table identity and ownership cleanup: generated names stay namespace-scoped and + * unique across counter resets, cleanup starts after the ownership mark, and a cleanup failure is + * suppressed behind the primary test failure. */ final class TableTestTest { @Test @@ -20,4 +27,46 @@ final class TableTestTest { assertTrue(secondTable.startsWith("test_namespace.t_")) assertNotEquals(firstTable, secondTable) } + + @Test + def failureBeforeOwnershipSkipsCleanup(): Unit = { + val createFailure = new Exception("table already exists") + var cleanupCalled = false + + val thrown = assertThrows( + classOf[Exception], + () => + OwnedTableLifecycle.withOwnership(cleanupCalled = true)(_ => + throw createFailure)) + + assertSame(createFailure, thrown) + assertFalse(cleanupCalled, "a failed create must leave the conflicting table intact") + } + + @Test + def successfulOwnershipRunsCleanupOnce(): Unit = { + var cleanupCount = 0 + + OwnedTableLifecycle.withOwnership(cleanupCount += 1)(markTableCreated => + markTableCreated()) + + assertEquals(1, cleanupCount) + } + + @Test + def cleanupFailureIsSuppressedOnThePrimaryFailure(): Unit = { + val testFailure = new Exception("test failed") + val cleanupFailure = new Exception("cleanup failed") + + val thrown = assertThrows( + classOf[Exception], + () => + OwnedTableLifecycle.withOwnership(throw cleanupFailure) { markTableCreated => + markTableCreated() + throw testFailure + }) + + assertSame(testFailure, thrown) + assertEquals(List(cleanupFailure), thrown.getSuppressed.toList) + } } From 808fcc97414f69108e8a7a8da018524dff82981a Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Thu, 27 Aug 2026 20:07:10 -0700 Subject: [PATCH 10/24] test(delta-harness): cover cleanup failure Pin the remaining ownership outcome: when the test body succeeds and cleanup fails, the cleanup failure must surface to the runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../src/test/scala/harness/TableTestTest.scala | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala index 40148f3c0..30987963c 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala @@ -53,6 +53,19 @@ final class TableTestTest { assertEquals(1, cleanupCount) } + @Test + def cleanupFailureIsPrimaryAfterTheBodySucceeds(): Unit = { + val cleanupFailure = new Exception("cleanup failed") + + val thrown = assertThrows( + classOf[Exception], + () => + OwnedTableLifecycle.withOwnership(throw cleanupFailure)( + markTableCreated => markTableCreated())) + + assertSame(cleanupFailure, thrown) + } + @Test def cleanupFailureIsSuppressedOnThePrimaryFailure(): Unit = { val testFailure = new Exception("test failed") From 6a5fbff155cdf1d1ba371a9f50e7ee278e598b35 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Tue, 1 Sep 2026 12:39:00 -0700 Subject: [PATCH 11/24] refactor(delta-harness): clarify test intent Reflow harness documentation to the repository's 120-column target and explain the DML operation and preparation matrix at its source. Name the reusable date column independently from partitioning so layouts, not column identifiers, express partition choices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/openhouse/DmlScenarios.scala | 360 ++++++++---------- .../main/scala/harness/openhouse/Env.scala | 12 +- .../harness/openhouse/ForkScenarios.scala | 123 +++--- .../scala/harness/openhouse/Framework.scala | 96 +++-- .../HazardReaderWriterScenarios.scala | 73 ++-- .../ImplementationPinScenarios.scala | 16 +- .../openhouse/InteractionScenarios.scala | 33 +- .../openhouse/MaintControlScenarios.scala | 25 +- .../openhouse/NegativeDdlScenarios.scala | 134 +++---- .../openhouse/NestedTypesScenarios.scala | 82 ++-- .../main/scala/harness/openhouse/Plan.scala | 10 +- .../scala/harness/openhouse/ScenarioKit.scala | 101 +++-- .../harness/openhouse/SurfaceScenarios.scala | 114 +++--- .../scala/harness/DmlCaseCatalogTest.scala | 6 +- .../scala/harness/TablePreparationTest.scala | 6 +- .../test/scala/harness/TableTestTest.scala | 5 +- 16 files changed, 529 insertions(+), 667 deletions(-) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala index cd0ee3d0d..8251d9f38 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala @@ -3,24 +3,32 @@ package harness import org.apache.spark.sql.Row import org.apache.spark.sql.functions.lit -// The DML tests are written as two independent lists. A TablePreparation describes a starting -// table state (layout, evolution, restored table). A DmlTestCase describes one operation and asserts -// the rows and the snapshot delta that operation causes. A bucket of cases is the cross of a -// preparation list with the test-case list it is compatible with, so every case reads as "this -// operation, on this starting state". The named test-case lists below are the shared vocabulary a -// feature layer reuses: it crosses them with its own preparations through a self-type on this -// trait. +/** + * Defines 54 reusable DML operations over the six CoreTable columns: bigint, int, string, double, boolean, and a + * string-encoded date. The operation catalog contains 2 reads, 14 deletes, 13 updates, 16 merges, 6 inserts or + * overwrites, 1 null-string delete, and 2 partition-scoped overwrites. Each operation covers a distinct SQL or + * DataFrame form or a distinct observable state change within its family. + * + * ScenarioKit supplies the starting-state axes. Six core layouts cross three file formats with partitioned and + * unpartitioned tables. Three date-partitioned layouts receive partition-scoped writes. Six write-ordered layouts + * exercise the same catalog under sort order. Six evolved layouts receive the 29 operations that address columns by + * name. Null-string variants isolate the one operation that requires a null value. + * + * The final section defines 13 bespoke DDL follow-up cases. Six consume a table after a DDL state transition, and seven + * verify schema creation or evolution. These cases stay outside the DML cross-product because the DDL transition is + * part of the behavior under test. + */ trait DmlScenarios extends ScenarioKit { import Rows._ // --- the DML test cases --- - // Each case captures the table state, runs one operation, captures the state again, and asserts - // the row change and the snapshot delta that operation caused. Deltas are relative, so a case - // holds on any preparation regardless of how many snapshots the preparation itself committed. + // Each case captures the table state, runs one operation, captures the state again, and asserts the row change and + // the snapshot delta that operation caused. Deltas are relative, so a case holds on any preparation regardless of how + // many snapshots the preparation itself committed. /** - * SELECT of foo_col_string alone returns that column for every prepared row in key order and - * leaves the table state unchanged. + * SELECT of foo_col_string alone returns that column for every prepared row in key order and leaves the table state + * unchanged. */ private val readProjection: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -43,8 +51,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * SELECT with a foo_col_long >= 2 predicate returns exactly the prepared rows whose key is 2 or - * greater and leaves the table state unchanged. + * SELECT with a foo_col_long >= 2 predicate returns exactly the prepared rows whose key is 2 or greater and leaves + * the table state unchanged. */ private val readFilter: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -67,16 +75,16 @@ trait DmlScenarios extends ScenarioKit { }) /** - * The reads. They select columns by name and write nothing, so they run on any preparation that - * starts from the three seed rows, including one whose column list has grown past that shape. + * The reads. They select columns by name and write nothing, so they run on any preparation that starts from the three + * seed rows, including one whose column list has grown past that shape. */ val readTestCases: List[DmlTestCase[CoreTable.type]] = List( readProjection, readFilter) /** - * DELETE WHERE foo_col_string IS NULL removes exactly the prepared row whose string is null, - * leaves every other row unchanged, and commits one snapshot. + * DELETE WHERE foo_col_string IS NULL removes exactly the prepared row whose string is null, leaves every other row + * unchanged, and commits one snapshot. */ private val deleteByNullCondition: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -97,15 +105,15 @@ trait DmlScenarios extends ScenarioKit { }) /** - * The DELETE that selects a null string. It applies to a preparation that already holds a row - * whose string column is null, and it removes exactly that row. + * The DELETE that selects a null string. It applies to a preparation that already holds a row whose string column is + * null, and it removes exactly that row. */ val nullStringRowTestCases: List[DmlTestCase[CoreTable.type]] = List( deleteByNullCondition) /** - * DELETE WHERE datepartition = '2024-01-01-00' removes the rows in that partition value, keeps - * the rest, and commits one snapshot. + * DELETE WHERE foo_col_date = '2024-01-01-00' removes the rows with that date, keeps the rest, and commits one + * snapshot. */ private val deleteByPartitionPredicate: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -115,11 +123,11 @@ trait DmlScenarios extends ScenarioKit { table.spark.sql( s"DELETE FROM ${table.name} WHERE " + - s"${Core.datePartition.columnName} = '2024-01-01-00'") + s"${Core.date0.columnName} = '2024-01-01-00'") val after = table.state assert( - after.rows == before.rows.filterNot(_.get(Core.datePartition) == "2024-01-01-00"), + after.rows == before.rows.filterNot(_.get(Core.date0) == "2024-01-01-00"), s"rows after the DELETE: ${after.rows}") assert( after.snapshotCount == before.snapshotCount + 1, @@ -127,8 +135,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * DELETE WHERE foo_col_long < 2 removes the rows below key 2, leaves every other row unchanged, - * and commits one snapshot. + * DELETE WHERE foo_col_long < 2 removes the rows below key 2, leaves every other row unchanged, and commits one + * snapshot. */ private val deleteByPredicate: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -149,8 +157,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * DELETE WHERE foo_col_long IN (1, 3) removes keys 1 and 3, leaves every other row exactly as - * prepared, and commits one snapshot. + * DELETE WHERE foo_col_long IN (1, 3) removes keys 1 and 3, leaves every other row exactly as prepared, and commits + * one snapshot. */ private val deleteByInList: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -171,8 +179,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * DELETE WHERE foo_col_long IN (subquery yielding 2) removes key 2, leaves every other row - * unchanged, and commits one snapshot. + * DELETE WHERE foo_col_long IN (subquery yielding 2) removes key 2, leaves every other row unchanged, and commits one + * snapshot. */ private val deleteByInSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -194,8 +202,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * DELETE WHERE foo_col_long NOT IN (subquery yielding 2) removes every key other than 2, leaves - * the row for key 2 unchanged, and commits one snapshot. + * DELETE WHERE foo_col_long NOT IN (subquery yielding 2) removes every key other than 2, leaves the row for key 2 + * unchanged, and commits one snapshot. */ private val deleteByNotInSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -217,8 +225,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * DELETE WHERE EXISTS (correlated subquery matching foo_col_long = 2) removes key 2, leaves - * every other row unchanged, and commits one snapshot. + * DELETE WHERE EXISTS (correlated subquery matching foo_col_long = 2) removes key 2, leaves every other row + * unchanged, and commits one snapshot. */ private val deleteByExistsSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -241,8 +249,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * DELETE WHERE NOT EXISTS (correlated subquery matching foo_col_long = 2) removes every key - * other than 2, leaves the row for key 2 unchanged, and commits one snapshot. + * DELETE WHERE NOT EXISTS (correlated subquery matching foo_col_long = 2) removes every key other than 2, leaves the + * row for key 2 unchanged, and commits one snapshot. */ private val deleteByNotExistsSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -265,8 +273,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * DELETE WHERE foo_col_long = (scalar subquery yielding 2) removes key 2, leaves every other row - * unchanged, and commits one snapshot. + * DELETE WHERE foo_col_long = (scalar subquery yielding 2) removes key 2, leaves every other row unchanged, and + * commits one snapshot. */ private val deleteByScalarSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -303,10 +311,7 @@ trait DmlScenarios extends ScenarioKit { "an unconditional DELETE commits one snapshot") }) - /** - * DELETE WHERE foo_col_long = 999 matches no row, leaves every row unchanged, and still commits - * one snapshot. - */ + /** DELETE WHERE foo_col_long = 999 matches no row, leaves every row unchanged, and still commits one snapshot. */ private val deleteNone: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.none", @@ -324,8 +329,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * DELETE FROM
AS x WHERE x.foo_col_long < 2 resolves the alias, removes the rows below - * key 2, and commits one snapshot. + * DELETE FROM
AS x WHERE x.foo_col_long < 2 resolves the alias, removes the rows below key 2, and commits one + * snapshot. */ private val deleteWithAlias: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -345,9 +350,7 @@ trait DmlScenarios extends ScenarioKit { "DELETE through an alias commits one snapshot") }) - /** - * DELETE WHERE false is optimized away: the rows stay as they are and no snapshot is committed. - */ + /** DELETE WHERE false is optimized away: the rows stay as they are and no snapshot is committed. */ private val deleteWhereFalseNoSnapshot: DmlTestCase[CoreTable.type] = DmlTestCase( "delete.whereFalse.noSnapshot", @@ -380,8 +383,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * DELETE against a snapshot-pinned identifier is rejected with an IllegalArgumentException - * naming that snapshot, and the rows and the snapshot count stay unchanged. + * DELETE against a snapshot-pinned identifier is rejected with an IllegalArgumentException naming that snapshot, and + * the rows and the snapshot count stay unchanged. */ private val deleteAtSnapshotRejected: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -409,9 +412,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * The DELETE operations. They select rows by column name and write no new row, so they run on - * any preparation that starts from the three seed rows, including one whose column list has - * grown past that shape. + * The DELETE operations. They select rows by column name and write no new row, so they run on any preparation that + * starts from the three seed rows, including one whose column list has grown past that shape. */ private val deleteTestCases: List[DmlTestCase[CoreTable.type]] = List( deleteByPredicate, @@ -430,8 +432,8 @@ trait DmlScenarios extends ScenarioKit { deleteAtSnapshotRejected) /** - * UPDATE SET foo_col_string = 'X' WHERE foo_col_long = 2 rewrites that column for key 2 only, - * leaves every other row unchanged, and commits one snapshot. + * UPDATE SET foo_col_string = 'X' WHERE foo_col_long = 2 rewrites that column for key 2 only, leaves every other row + * unchanged, and commits one snapshot. */ private val updateByPredicate: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -454,8 +456,7 @@ trait DmlScenarios extends ScenarioKit { }) /** - * UPDATE SET foo_col_string = 'Z' without a WHERE clause rewrites that column for every row and - * commits one snapshot. + * UPDATE SET foo_col_string = 'Z' without a WHERE clause rewrites that column for every row and commits one snapshot. */ private val updateWithoutCondition: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -475,10 +476,7 @@ trait DmlScenarios extends ScenarioKit { "an unconditional UPDATE commits one snapshot") }) - /** - * UPDATE ... WHERE foo_col_long = 99 matches no row, leaves every row unchanged, and still - * commits one snapshot. - */ + /** UPDATE ... WHERE foo_col_long = 99 matches no row, leaves every row unchanged, and still commits one snapshot. */ private val updateNoMatch: DmlTestCase[CoreTable.type] = DmlTestCase( "update.noMatch", @@ -498,10 +496,7 @@ trait DmlScenarios extends ScenarioKit { "a no-match UPDATE still commits one snapshot") }) - /** - * UPDATE ... WHERE foo_col_long IN (subquery yielding 2) rewrites key 2 only and commits one - * snapshot. - */ + /** UPDATE ... WHERE foo_col_long IN (subquery yielding 2) rewrites key 2 only and commits one snapshot. */ private val updateByInSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "update.byInSubquery", @@ -524,8 +519,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * UPDATE ... WHERE foo_col_long NOT IN (subquery yielding 2) rewrites every key other than 2 and - * commits one snapshot. + * UPDATE ... WHERE foo_col_long NOT IN (subquery yielding 2) rewrites every key other than 2 and commits one + * snapshot. */ private val updateByNotInSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -549,8 +544,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * UPDATE ... WHERE EXISTS (correlated subquery matching foo_col_long = 2) rewrites key 2 only - * and commits one snapshot. + * UPDATE ... WHERE EXISTS (correlated subquery matching foo_col_long = 2) rewrites key 2 only and commits one + * snapshot. */ private val updateByExistsSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -574,8 +569,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * UPDATE ... WHERE NOT EXISTS (correlated subquery matching foo_col_long = 2) rewrites every key - * other than 2 and commits one snapshot. + * UPDATE ... WHERE NOT EXISTS (correlated subquery matching foo_col_long = 2) rewrites every key other than 2 and + * commits one snapshot. */ private val updateByNotExistsSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -598,10 +593,7 @@ trait DmlScenarios extends ScenarioKit { "UPDATE by a NOT EXISTS subquery commits one snapshot") }) - /** - * UPDATE ... WHERE foo_col_long = (scalar subquery yielding 2) rewrites key 2 only and commits - * one snapshot. - */ + /** UPDATE ... WHERE foo_col_long = (scalar subquery yielding 2) rewrites key 2 only and commits one snapshot. */ private val updateByScalarSubquery: DmlTestCase[CoreTable.type] = DmlTestCase( "update.byScalarSubquery", @@ -624,8 +616,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * UPDATE
AS x SET x.foo_col_string ... WHERE x.foo_col_long = 2 resolves the alias on - * both sides, rewrites key 2 only, and commits one snapshot. + * UPDATE
AS x SET x.foo_col_string ... WHERE x.foo_col_long = 2 resolves the alias on both sides, rewrites + * key 2 only, and commits one snapshot. */ private val updateWithAlias: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -648,8 +640,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * UPDATE SET foo_col_string = 'X', foo_col_int = 99 WHERE foo_col_long = 2 rewrites both columns - * of key 2 in one statement and commits one snapshot. + * UPDATE SET foo_col_string = 'X', foo_col_int = 99 WHERE foo_col_long = 2 rewrites both columns of key 2 in one + * statement and commits one snapshot. */ private val updateMultipleColumns: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -674,8 +666,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * UPDATE SET foo_col_long = foo_col_long + 10 WHERE foo_col_long = 2 moves key 2 to key 12, - * leaves every other row unchanged, and commits one snapshot. + * UPDATE SET foo_col_long = foo_col_long + 10 WHERE foo_col_long = 2 moves key 2 to key 12, leaves every other row + * unchanged, and commits one snapshot. */ private val updateByExpression: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -699,8 +691,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * UPDATE SET datepartition = '2099-12-31-23' WHERE foo_col_long = 2 moves key 2 to another - * partition value, leaves every other row unchanged, and commits one snapshot. + * UPDATE SET foo_col_date = '2099-12-31-23' WHERE foo_col_long = 2 moves key 2 to another date partition value, + * leaves every other row unchanged, and commits one snapshot. */ private val updateMovePartition: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -710,14 +702,14 @@ trait DmlScenarios extends ScenarioKit { table.spark.sql( s"UPDATE ${table.name} SET " + - s"${Core.datePartition.columnName} = '2099-12-31-23' " + + s"${Core.date0.columnName} = '2099-12-31-23' " + s"WHERE ${Core.long0.columnName} = 2") val after = table.state assert( after.rows == before.rows.map(row => if (row.get(Core.long0) == 2L) { - withColumnValue(row, Core.datePartition, "2099-12-31-23") + withColumnValue(row, Core.date0, "2099-12-31-23") } else row), s"rows after the UPDATE: ${after.rows}") assert( @@ -726,8 +718,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * UPDATE SET foo_col_string = NULL WHERE foo_col_long = 2 stores a null in that column for key 2 - * only and commits one snapshot. + * UPDATE SET foo_col_string = NULL WHERE foo_col_long = 2 stores a null in that column for key 2 only and commits one + * snapshot. */ private val updateNullAssignment: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -750,8 +742,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * The UPDATE operations. They assign columns by name, so they run on any preparation that starts - * from the three seed rows, including one whose column list has grown past that shape. + * The UPDATE operations. They assign columns by name, so they run on any preparation that starts from the three seed + * rows, including one whose column list has grown past that shape. */ private val updateTestCases: List[DmlTestCase[CoreTable.type]] = List( updateByPredicate, @@ -769,9 +761,8 @@ trait DmlScenarios extends ScenarioKit { updateNullAssignment) /** - * MERGE with only a WHEN NOT MATCHED THEN INSERT * clause appends the two source rows (keys 4 - * and 5) with every source column value, leaves the prepared rows unchanged, and commits one - * snapshot. + * MERGE with only a WHEN NOT MATCHED THEN INSERT * clause appends the two source rows (keys 4 and 5) with every + * source column value, leaves the prepared rows unchanged, and commits one snapshot. */ private val mergeInsertNotMatched: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -800,8 +791,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE with only a WHEN MATCHED THEN UPDATE clause rewrites the matched key 2, leaves the - * unmatched rows unchanged, and commits one snapshot. + * MERGE with only a WHEN MATCHED THEN UPDATE clause rewrites the matched key 2, leaves the unmatched rows unchanged, + * and commits one snapshot. */ private val mergeUpdateMatched: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -828,8 +819,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE with only a WHEN MATCHED THEN DELETE clause removes the matched keys 1 and 3, keeps the - * unmatched rows, and commits one snapshot. + * MERGE with only a WHEN MATCHED THEN DELETE clause removes the matched keys 1 and 3, keeps the unmatched rows, and + * commits one snapshot. */ private val mergeDeleteMatched: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -854,8 +845,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE with both an UPDATE clause and an INSERT clause rewrites the matched key 2 and appends - * the unmatched key 7 in a single statement, and commits one snapshot. + * MERGE with both an UPDATE clause and an INSERT clause rewrites the matched key 2 and appends the unmatched key 7 in + * a single statement, and commits one snapshot. */ private val mergeUpsert: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -887,8 +878,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE with a WHEN NOT MATCHED BY SOURCE THEN DELETE clause removes every row the source does - * not carry, keeps the matched key 2, and commits one snapshot. + * MERGE with a WHEN NOT MATCHED BY SOURCE THEN DELETE clause removes every row the source does not carry, keeps the + * matched key 2, and commits one snapshot. */ private val mergeDeleteNotMatchedBySource: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -913,9 +904,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE with a WHEN MATCHED AND THEN UPDATE clause rewrites only the matched row - * that also satisfies the condition (key 2), leaves matched key 3 unchanged, and commits one - * snapshot. + * MERGE with a WHEN MATCHED AND THEN UPDATE clause rewrites only the matched row that also satisfies the + * condition (key 2), leaves matched key 3 unchanged, and commits one snapshot. */ private val mergeConditionalUpdate: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -943,8 +933,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE with two MATCHED clauses applies the first matching clause per row: key 2 is updated by - * the conditional clause and key 3 falls through to the DELETE clause, in one snapshot. + * MERGE with two MATCHED clauses applies the first matching clause per row: key 2 is updated by the conditional + * clause and key 3 falls through to the DELETE clause, in one snapshot. */ private val mergeMultipleMatchedClauses: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -975,8 +965,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE with a WHEN NOT MATCHED AND THEN INSERT clause appends only the source row - * that satisfies the condition (key 4), skips key 5, and commits one snapshot. + * MERGE with a WHEN NOT MATCHED AND THEN INSERT clause appends only the source row that satisfies the + * condition (key 4), skips key 5, and commits one snapshot. */ private val mergeConditionalInsert: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1003,8 +993,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE carrying UPDATE, INSERT, and NOT MATCHED BY SOURCE DELETE clauses updates key 2, inserts - * key 4, deletes the rows the source omits, and commits one snapshot. + * MERGE carrying UPDATE, INSERT, and NOT MATCHED BY SOURCE DELETE clauses updates key 2, inserts key 4, deletes the + * rows the source omits, and commits one snapshot. */ private val mergeAllClauses: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1038,8 +1028,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE with WHEN MATCHED THEN UPDATE SET * copies every source column onto the matched key 2, - * leaves the unmatched rows unchanged, and commits one snapshot. + * MERGE with WHEN MATCHED THEN UPDATE SET * copies every source column onto the matched key 2, leaves the unmatched + * rows unchanged, and commits one snapshot. */ private val mergeUpdateStar: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1068,8 +1058,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE whose INSERT clause names a column subset appends key 7 with the named values, leaves - * the unnamed columns null, and commits one snapshot. + * MERGE whose INSERT clause names a column subset appends key 7 with the named values, leaves the unnamed columns + * null, and commits one snapshot. */ private val mergeInsertExplicitColumns: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1096,8 +1086,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE whose source is a common table expression appends the key 8 that CTE yields, with null - * in every column the CTE does not supply, and commits one snapshot. + * MERGE whose source is a common table expression appends the key 8 that CTE yields, with null in every column the + * CTE does not supply, and commits one snapshot. */ private val mergeSourceCTE: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1125,8 +1115,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE whose source is a UNION ALL appends both keys the set operation yields (8 and 9), with - * null in every column the source does not supply, and commits one snapshot. + * MERGE whose source is a UNION ALL appends both keys the set operation yields (8 and 9), with null in every column + * the source does not supply, and commits one snapshot. */ private val mergeSourceSetOp: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1155,8 +1145,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * After the table is emptied, MERGE with a NOT MATCHED INSERT clause inserts both source rows - * (keys 4 and 5) into the empty target and commits one snapshot. + * After the table is emptied, MERGE with a NOT MATCHED INSERT clause inserts both source rows (keys 4 and 5) into the + * empty target and commits one snapshot. */ private val mergeIntoEmptyTarget: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1188,8 +1178,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE whose source carries a null join key matches no target row on that key: only the - * non-null key 2 is updated, no row is added or removed, and one snapshot is committed. + * MERGE whose source carries a null join key matches no target row on that key: only the non-null key 2 is updated, + * no row is added or removed, and one snapshot is committed. */ private val mergeNullJoinKey: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1217,9 +1207,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * MERGE with INSERT * resolves the source columns by name: key 7 lands with every source value - * in its named column when the source lists its columns in another order, and one snapshot is - * committed. + * MERGE with INSERT * resolves the source columns by name: key 7 lands with every source value in its named column + * when the source lists its columns in another order, and one snapshot is committed. */ private val mergeResolveByName: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1237,7 +1226,7 @@ trait DmlScenarios extends ScenarioKit { ${Core.int0.columnName}, ${Core.double0.columnName}, ${Core.boolean0.columnName}, - datepartition) + ${Core.date0.columnName}) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED THEN INSERT *""") val after = table.state @@ -1251,8 +1240,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * The MERGE operations. Their INSERT clauses write a whole seed-shaped row, so they run on a - * preparation whose column list is still the seed shape. + * The MERGE operations. Their INSERT clauses write a whole seed-shaped row, so they run on a preparation whose column + * list is still the seed shape. */ private val mergeTestCases: List[DmlTestCase[CoreTable.type]] = List( mergeInsertNotMatched, @@ -1273,8 +1262,8 @@ trait DmlScenarios extends ScenarioKit { mergeResolveByName) /** - * INSERT INTO ... VALUES appends the two literal rows (keys 4 and 5), leaves the prepared rows - * unchanged, and commits one snapshot. + * INSERT INTO ... VALUES appends the two literal rows (keys 4 and 5), leaves the prepared rows unchanged, and commits + * one snapshot. */ private val insertInto: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1299,8 +1288,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * INSERT INTO naming a subset of the columns is rejected by the engine with a message naming the - * omitted data, and the rows and the snapshot count stay unchanged. + * INSERT INTO naming a subset of the columns is rejected by the engine with a message naming the omitted data, and + * the rows and the snapshot count stay unchanged. */ private val insertExplicitColumns: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1326,8 +1315,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * INSERT INTO ... SELECT appends the row the SELECT produces (key 6), leaves the prepared rows - * unchanged, and commits one snapshot. + * INSERT INTO ... SELECT appends the row the SELECT produces (key 6), leaves the prepared rows unchanged, and commits + * one snapshot. */ private val insertIntoSelect: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1350,8 +1339,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * The DataFrame writeTo(...).append() path appends the frame's row (key 6), keeps the prepared - * rows, and commits one snapshot. + * The DataFrame writeTo(...).append() path appends the frame's row (key 6), keeps the prepared rows, and commits one + * snapshot. */ private val appendDataFrame: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1377,8 +1366,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * INSERT OVERWRITE ... VALUES replaces the table contents with the two literal rows (keys 1 and - * 2) and commits one snapshot. + * INSERT OVERWRITE ... VALUES replaces the table contents with the two literal rows (keys 1 and 2) and commits one + * snapshot. */ private val insertOverwrite: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1403,8 +1392,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * The DataFrame writeTo(...).overwrite(lit(true)) path replaces every row with the frame's row - * (key 8) and commits one snapshot. + * The DataFrame writeTo(...).overwrite(lit(true)) path replaces every row with the frame's row (key 8) and commits + * one snapshot. */ private val overwriteDataFrame: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1430,8 +1419,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * The appends and the overwrites. They write whole seed-shaped rows, so they run on a - * preparation whose column list is still the seed shape. + * The appends and the overwrites. They write whole seed-shaped rows, so they run on a preparation whose column list + * is still the seed shape. */ private val insertAndOverwriteTestCases: List[DmlTestCase[CoreTable.type]] = List( insertInto, @@ -1442,9 +1431,8 @@ trait DmlScenarios extends ScenarioKit { overwriteDataFrame) /** - * Under partitionOverwriteMode=dynamic, INSERT OVERWRITE with one row replaces only that row's - * partition (2024-01-01-00), leaves the rows of every other partition unchanged, and commits one - * snapshot. + * Under partitionOverwriteMode=dynamic, INSERT OVERWRITE with one row replaces only that row's partition + * (2024-01-01-00), leaves the rows of every other partition unchanged, and commits one snapshot. */ private val insertDynamicOverwrite: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1464,7 +1452,7 @@ trait DmlScenarios extends ScenarioKit { assert( after.rows == inKeyOrder( - before.rows.filterNot(_.get(Core.datePartition) == "2024-01-01-00") :+ + before.rows.filterNot(_.get(Core.date0) == "2024-01-01-00") :+ Row(10L, 10, "p", 10.5, true, "2024-01-01-00")), s"rows after the dynamic overwrite: ${after.rows}") assert( @@ -1473,9 +1461,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * The DataFrame writeTo(...).overwritePartitions() path replaces only the partitions the frame - * carries (2024-01-01-00), leaves the rows of every other partition unchanged, and commits one - * snapshot. + * The DataFrame writeTo(...).overwritePartitions() path replaces only the partitions the frame carries + * (2024-01-01-00), leaves the rows of every other partition unchanged, and commits one snapshot. */ private val overwritePartitions: DmlTestCase[CoreTable.type] = DmlTestCase( @@ -1494,7 +1481,7 @@ trait DmlScenarios extends ScenarioKit { assert( after.rows == inKeyOrder( - before.rows.filterNot(_.get(Core.datePartition) == "2024-01-01-00") :+ + before.rows.filterNot(_.get(Core.date0) == "2024-01-01-00") :+ Row(10L, 10, "p", 10.5, true, "2024-01-01-00")), s"rows after the partition overwrite: ${after.rows}") assert( @@ -1503,8 +1490,8 @@ trait DmlScenarios extends ScenarioKit { }) /** - * The partition-scoped writes. They replace whole partitions, so they apply to a preparation - * that partitions the table, and they cross with the partitioned preparations alone. + * The partition-scoped writes. They replace whole partitions, so they apply to a preparation that partitions the + * table, and they cross with the partitioned preparations alone. */ val partitionedTableTestCases: List[DmlTestCase[CoreTable.type]] = List( insertDynamicOverwrite, @@ -1526,15 +1513,15 @@ trait DmlScenarios extends ScenarioKit { deleteTestCases ++ updateTestCases ++ mergeTestCases /** - * The cases that address columns by name and never write a whole seed-shaped row, so they run on - * a preparation whose column list has grown beyond the seed rows. + * The cases that address columns by name and never write a whole seed-shaped row, so they run on a preparation whose + * column list has grown beyond the seed rows. */ val testCasesCompatibleWithAnAddedColumn: List[DmlTestCase[CoreTable.type]] = readTestCases ++ deleteTestCases ++ updateTestCases /** - * Every DML case, with the partition-predicate DELETE marked as a known bug: the Spark and - * Iceberg rewrite crashes on it when the table carries a write order. + * Every DML case, with the partition-predicate DELETE marked as a known bug: the Spark and Iceberg rewrite crashes on + * it when the table carries a write order. */ val orderedDmlTestCases: List[DmlTestCase[CoreTable.type]] = allDmlTestCases.map { @@ -1549,8 +1536,8 @@ trait DmlScenarios extends ScenarioKit { // --- standard preparations crossed with the cases they are compatible with --- /** - * Every DML case on the core preparations, plus the null-string DELETE on the same preparations - * extended with a null-string row. + * Every DML case on the core preparations, plus the null-string DELETE on the same preparations extended with a + * null-string row. */ val coreDmlCases: List[Plan.Case] = preparedCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) ++ @@ -1562,10 +1549,7 @@ trait DmlScenarios extends ScenarioKit { preparedPartitionedCoreTables.flatMap(preparation => partitionedTableTestCases.map(_.runOn(preparation))) - /** - * Every DML case on the write-ordered preparations, plus the null-string DELETE on their - * null-string form. - */ + /** Every DML case on the write-ordered preparations, plus the null-string DELETE on their null-string form. */ val orderedDmlCases: List[Plan.Case] = preparedOrderedCoreTables.flatMap(preparation => orderedDmlTestCases.map(_.runOn(preparation))) ++ preparedNullStringOrderedCoreTables.flatMap(preparation => @@ -1579,11 +1563,11 @@ trait DmlScenarios extends ScenarioKit { // --- DDL consumers: a DDL evolves the table, then operations are run against it --- /** - * One preparation per Parquet and ORC layout and per DDL: three seed rows with keys 1, 2 and 3, - * then one of ADD COLUMN cc int, which the seed rows read as null; foo_col_int widened from int - * to bigint; WRITE ORDERED BY foo_col_long, which gives the table that write sort order; or - * write.distribution-mode set to range, which range distributes later writes. Plan walks this - * list so every consumer family lands on one preparation before the next preparation starts. + * One preparation per Parquet and ORC layout and per DDL: three seed rows with keys 1, 2 and 3, then one of ADD + * COLUMN cc int, which the seed rows read as null; foo_col_int widened from int to bigint; WRITE ORDERED BY + * foo_col_long, which gives the table that write sort order; or write.distribution-mode set to range, which range + * distributes later writes. Plan walks this list so every consumer family lands on one preparation before the next + * preparation starts. */ val ddlConsumerPreparations: List[TablePreparation[CoreTable.type]] = parquetAndOrcLayouts.flatMap { layout => @@ -1643,10 +1627,7 @@ trait DmlScenarios extends ScenarioKit { "mutation failed after DDL") } - /** - * The seed snapshot from before the DDL is still readable through VERSION AS OF and returns its - * three rows. - */ + /** The seed snapshot from before the DDL is still readable through VERSION AS OF and returns its three rows. */ private def timeTravelCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("timeTravel") { table => val seedSnapshotId = @@ -1663,8 +1644,8 @@ trait DmlScenarios extends ScenarioKit { } /** - * rollback_to_snapshot back to the seed snapshot undoes an INSERT made after the DDL and returns - * the table to its three seed rows. + * rollback_to_snapshot back to the seed snapshot undoes an INSERT made after the DDL and returns the table to its + * three seed rows. */ private def restoreCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("restore") { table => @@ -1686,10 +1667,7 @@ trait DmlScenarios extends ScenarioKit { "restore across DDL failed") } - /** - * expire_snapshots retaining only the newest snapshot leaves the table readable with its four - * current rows. - */ + /** expire_snapshots retaining only the newest snapshot leaves the table readable with its four current rows. */ private def expireCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("expire") { table => table.spark.sql( @@ -1719,10 +1697,7 @@ trait DmlScenarios extends ScenarioKit { restoreCase(preparation), expireCase(preparation)) - /** - * rewrite_data_files compacts the files written across the DDL and preserves the four current - * rows. - */ + /** rewrite_data_files compacts the files written across the DDL and preserves the four current rows. */ private def compactCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("compact") { table => table.spark.sql( @@ -1750,8 +1725,8 @@ trait DmlScenarios extends ScenarioKit { // --- DDL that changes the schema of a seeded table --- /** - * The created table's schema is exactly CoreTable's columns, in declaration order and with their - * declared types, and the table holds no rows. + * The created table's schema is exactly CoreTable's columns, in declaration order and with their declared types, and + * the table holds no rows. */ private def createSchemaCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("create.schema") { table => @@ -1772,10 +1747,7 @@ trait DmlScenarios extends ScenarioKit { createSchemaCase(preparation) } - /** - * ADD COLUMN adds the column to the schema, the existing rows read null for it, and the row - * count is unchanged. - */ + /** ADD COLUMN adds the column to the schema, the existing rows read null for it, and the row count is unchanged. */ private def ddlAddColumnSingleCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.addColumn.single") { table => table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_int int") @@ -1793,10 +1765,7 @@ trait DmlScenarios extends ScenarioKit { assert(table.rows.size == table.preparedRows.size, "ADD COLUMN changed the row count") } - /** - * ADD COLUMNS with two columns in one statement adds both to the schema and leaves the row count - * unchanged. - */ + /** ADD COLUMNS with two columns in one statement adds both to the schema and leaves the row count unchanged. */ private def ddlAddColumnMultipleCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.addColumn.multiple") { table => table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMNS (added_a int, added_b string)") @@ -1826,10 +1795,7 @@ trait DmlScenarios extends ScenarioKit { s"comment not stored: ${addedColumn.getComment()}") } - /** - * ADD COLUMN ... AFTER foo_col_long places the added column directly after that column in the - * schema. - */ + /** ADD COLUMN ... AFTER foo_col_long places the added column directly after that column in the schema. */ private def ddlAddColumnPositionCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.addColumn.position") { table => table.spark.sql( @@ -1843,8 +1809,8 @@ trait DmlScenarios extends ScenarioKit { } /** - * ALTER COLUMN foo_col_int TYPE bigint widens the column in the schema and the already-written - * values read back unchanged. + * ALTER COLUMN foo_col_int TYPE bigint widens the column in the schema and the already-written values read back + * unchanged. */ private def ddlAlterColumnTypeWidenCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -1869,8 +1835,8 @@ trait DmlScenarios extends ScenarioKit { } /** - * RENAME COLUMN renames the column in the schema: the new name is present, the old name is gone, - * and the row count is unchanged. + * RENAME COLUMN renames the column in the schema: the new name is present, the old name is gone, and the row count is + * unchanged. */ private def ddlRenameColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala index 92734dce3..47aa1dd80 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala @@ -50,8 +50,8 @@ object OpenHouseEnv { .config(s"spark.sql.catalog.$name.auth-token", token) def start(): (OpenHouseLocalServer, SparkSession, String, String) = { - // The embedded server uses Hibernate to create its H2 schema. Hibernate owns initialization for - // this process, so classpath SQL initialization stays disabled. + // The embedded server uses Hibernate to create its H2 schema. Hibernate owns initialization for this process, so + // classpath SQL initialization stays disabled. System.setProperty("spring.sql.init.mode", "never") System.setProperty("spring.jpa.hibernate.ddl-auto", "create-drop") @@ -98,8 +98,8 @@ object Main { spark.sparkContext.setLogLevel("ERROR") val ctx = Ctx(spark, "openhouse.dbMatrix", restUri, restToken) - // Each command-line argument is an include substring. A case runs when its ID contains every - // provided substring. An empty argument list selects the full catalog. + // Each command-line argument is an include substring. A case runs when its ID contains every provided substring. + // An empty argument list selects the full catalog. val filters = args.toList val cases = Plan.cases.filter(testCase => filters.forall(testCase.id.contains)) @@ -112,8 +112,8 @@ object Main { } println(s"\n=== delta-harness :: localized cases @ OpenHouse catalog ($header) ===\n") - // Each case owns a fresh table. Worker tasks use separate Spark sessions over the shared - // Spark context, and results are printed in catalog order. + // Each case owns a fresh table. Worker tasks use separate Spark sessions over the shared Spark context, and + // results are printed in catalog order. val parallelism = sys.env.get("HARNESS_PARALLELISM").map(_.toInt) .getOrElse(math.max(1, Runtime.getRuntime.availableProcessors())) println(s"parallelism: $parallelism worker sessions\n") diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala index 8295977cc..dd443536f 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala @@ -10,21 +10,18 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -// The fork cases. OpenHouse compiles and runs against LinkedIn's fork of Apache Iceberg, the -// com.linkedin.iceberg artifacts this module depends on, and a case here pins a behavior the -// Iceberg library decides rather than one the catalog exposes: the column-default path, the write -// distribution default for a partitioned write, the output-file replication key, the read split -// size, and the compaction plan. These behaviors have no catalog SQL surface of their own, so a -// case reaches them through the Iceberg API or a Spark configuration and asserts the result a -// caller can observe. +// The fork cases pin behavior decided by LinkedIn's fork of Apache Iceberg, the com.linkedin.iceberg artifacts this +// module depends on: the column-default path, the write distribution default for a partitioned write, the output-file +// replication key, the read split size, and the compaction plan. These behaviors have no catalog SQL surface of their +// own, so a case reaches them through the Iceberg API or a Spark configuration and asserts the result a caller can +// observe. trait ForkScenarios extends ScenarioKit { import Rows._ /** - * ALTER TABLE ADD COLUMN c int DEFAULT 5 parses, and the default value stops at the parser: the - * committed schema records no default for c, pre-existing rows read null for it, and an INSERT - * that omits c is rejected with INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA. The file format is - * the parameter. + * ALTER TABLE ADD COLUMN c int DEFAULT 5 parses, and the default value stops at the parser: the committed schema + * records no default for c, pre-existing rows read null for it, and an INSERT that omits c is rejected with + * INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA. The file format is the parameter. */ private def forkColDefaultAddColumn(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark @@ -60,13 +57,12 @@ trait ForkScenarios extends ScenarioKit { } /** - * A NestedField built with an initial default serializes initial-default into the schema JSON, - * and that value survives a fromJson then toJson round trip. SchemaParser.toJson takes no - * format-version parameter, so the key serializes the same at every format version. On an - * artifact whose NestedField exposes no builder, the column-default API is absent entirely, down - * to the initialDefault and writeDefault accessors, and the case pins that absence. Reflection - * reaches the builder because some Iceberg release jars leave it out, which a direct reference - * would fail to compile against. + * A NestedField built with an initial default serializes initial-default into the schema JSON, and that value + * survives a fromJson then toJson round trip. SchemaParser.toJson takes no format-version parameter, so the key + * serializes the same at every format version. On an artifact whose NestedField exposes no builder, the + * column-default API is absent entirely, down to the initialDefault and writeDefault accessors, and the case pins + * that absence. Reflection reaches the builder because some Iceberg release jars leave it out, which a direct + * reference would fail to compile against. */ private def forkColDefaultApiSerialization(ctx: Ctx): Unit = { val nestedFieldCls = Class.forName("org.apache.iceberg.types.Types$NestedField") @@ -104,8 +100,8 @@ trait ForkScenarios extends ScenarioKit { // (a) The default is serialized into the schema JSON. assert(json.contains("initial-default"), s"expected SchemaParser to serialize 'initial-default' into the schema JSON, got: $json") - // (b) toJson takes no format-version argument, so the key serializes the same regardless of format version. - // (c) The value round-trips through fromJson then toJson. + // (b) toJson takes no format-version argument, so the key serializes the same regardless of format version. (c) The + // value round-trips through fromJson then toJson. val reparsed = org.apache.iceberg.SchemaParser.fromJson(json) val json2 = org.apache.iceberg.SchemaParser.toJson(reparsed) assert(json2.contains("initial-default"), @@ -114,8 +110,8 @@ trait ForkScenarios extends ScenarioKit { } /** - * Reflectively builds an optional int NestedField carrying the given initial default. Returns - * None when the builder API is absent, so a caller can assert that absence directly. + * Reflectively builds an optional int NestedField carrying the given initial default. Returns None when the builder + * API is absent, so a caller can assert that absence directly. */ private def buildDefaultedIntField(id: Int, name: String, dflt: Int): Option[org.apache.iceberg.types.Types.NestedField] = { val nfCls = Class.forName("org.apache.iceberg.types.Types$NestedField") @@ -133,11 +129,10 @@ trait ForkScenarios extends ScenarioKit { } /** - * A column default added after data files exist persists into the committed schema. The schema - * evolution goes through the low-level TableMetadata API because the public UpdateSchema surface - * has no set-default operation. The documented read contract covers schema persistence only. The - * case prints the OSS Spark read result for pre-existing rows as diagnostic output, while its - * assertions stop at the persisted schema. + * A column default added after data files exist persists into the committed schema. The schema evolution goes through + * the low-level TableMetadata API because the public UpdateSchema surface has no set-default operation. The + * documented read contract covers schema persistence only. The case prints the OSS Spark read result for pre-existing + * rows as diagnostic output, while its assertions stop at the persisted schema. */ private def forkColDefaultReadApplyProbe(ctx: Ctx): Unit = { val spark = ctx.spark @@ -180,8 +175,8 @@ trait ForkScenarios extends ScenarioKit { assert(persisted.contains("initial-default"), s"expected initial-default to persist into the committed schema, got: $persisted") - // Recorded for reference only: the read path's treatment of the defaulted column over old files is - // not part of this connector's documented contract. + // Recorded for reference only: the read path's treatment of the defaulted column over old files is not part of this + // connector's documented contract. spark.sql(s"REFRESH TABLE $t") val vals = spark.sql(s"SELECT c FROM $t ORDER BY id").collect() .map(r => if (r.isNullAt(0)) "NULL" else r.getInt(0).toString) @@ -191,12 +186,11 @@ trait ForkScenarios extends ScenarioKit { } /** - * A partitioned write defaults write.distribution-mode to NONE, so every input task writes every - * partition it holds and one append produces up to (input tasks times partitions) data files. - * Under an explicit HASH distribution the writer shuffles rows so one task owns each partition, - * clustering the append to roughly one file per partition. Appending the same multi-task - * DataFrame into a 4-partition table under each mode therefore yields at least as many files - * under the default as under HASH. The file format is the parameter. + * A partitioned write defaults write.distribution-mode to NONE, so every input task writes every partition it holds + * and one append produces up to (input tasks times partitions) data files. Under an explicit HASH distribution the + * writer shuffles rows so one task owns each partition, clustering the append to roughly one file per partition. + * Appending the same multi-task DataFrame into a 4-partition table under each mode therefore yields at least as many + * files under the default as under HASH. The file format is the parameter. */ private def forkPartitionDistDefault(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark @@ -233,12 +227,11 @@ trait ForkScenarios extends ScenarioKit { } /** - * OutputFileFactory exposes FILE_REPLICATION_FACTOR as "file-replication-factor", and a factory - * built with a replication factor stamps that key into the property map of the output files it - * creates. Writes made through the table afterward still return the correct rows. It is not a - * settable table property; it is the key HDFS reads to set block replication on an output file - * when a replication factor is supplied to the factory, and the delete-file write path is the one - * path that supplies one. Reflection reaches the builder and getProperties because some Iceberg + * OutputFileFactory exposes FILE_REPLICATION_FACTOR as "file-replication-factor", and a factory built with a + * replication factor stamps that key into the property map of the output files it creates. Writes made through the + * table afterward still return the correct rows. It is not a settable table property; it is the key HDFS reads to set + * block replication on an output file when a replication factor is supplied to the factory, and the delete-file write + * path is the one path that supplies one. Reflection reaches the builder and getProperties because some Iceberg * artifacts leave them out of the public compiled API. */ private def forkFileReplicationFactor(ctx: Ctx): Unit = { @@ -285,20 +278,18 @@ trait ForkScenarios extends ScenarioKit { } /** - * spark.sql.iceberg.split-size decides how the read path combines data files into read tasks. - * Over several small files, a large split size combines them into fewer read tasks and a tiny - * split size splits them into more, visible through rdd.getNumPartitions, and both reads return - * the same rows. The planner shows the same effect directly: a split size above the whole table - * plans one task group, and a split size below one file plans one group per file. The file format - * is the parameter. + * spark.sql.iceberg.split-size decides how the read path combines data files into read tasks. Over several small + * files, a large split size combines them into fewer read tasks and a tiny split size splits them into more, visible + * through rdd.getNumPartitions, and both reads return the same rows. The planner shows the same effect directly: a + * split size above the whole table plans one task group, and a split size below one file plans one group per file. + * The file format is the parameter. */ private def forkSplitSize(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark val table = s"${ctx.namespace}.t_splitsize_$fmt" spark.sql(s"DROP TABLE IF EXISTS $table") - // distribution=none plus several separate inserts produces several distinct data files. An - // open-file-cost of 1 sets each file's planning weight to its byte length, making split-size the - // knob that governs task-group count. + // distribution=none plus several separate inserts produces several distinct data files. An open-file-cost of 1 sets + // each file's planning weight to its byte length, making split-size the knob that governs task-group count. spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + s"TBLPROPERTIES ('write.format.default'='$fmt', 'write.distribution-mode'='none', 'read.split.open-file-cost'='1')") val numberOfFiles = 6 @@ -314,8 +305,8 @@ trait ForkScenarios extends ScenarioKit { def rddParts(): Int = spark.sql(s"SELECT * FROM $table").rdd.getNumPartitions val expected = (0 until numberOfFiles).map(_.toLong) try { - // (a) Set spark.sql.iceberg.split-size directly and read the multi-file table under a large and a - // tiny split size; the row set must be invariant either way. + // (a) Set spark.sql.iceberg.split-size directly and read the multi-file table under a large and a tiny split + // size; the row set must be invariant either way. spark.conf.set(key, (512L * 1024 * 1024).toString) val bigRows = keys(); val bigRdd = rddParts() spark.conf.set(key, "1") @@ -325,9 +316,9 @@ trait ForkScenarios extends ScenarioKit { assert(smallRdd >= bigRdd, s"[$fmt] a smaller split-size must not decrease the read RDD partition count: small=$smallRdd big=$bigRdd") - // (b) The same knob checked directly at the planner: with open-file-cost=1, each file's planning - // weight is its byte length, so a split-size below one file combines nothing (one task group - // per file) while a split-size above the whole table combines everything into one group. + // (b) The same knob checked directly at the planner: with open-file-cost=1, each file's planning weight is its + // byte length, so a split-size below one file combines nothing (one task group per file) while a split-size above + // the whole table combines everything into one group. val ice = org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, table) val szKey = org.apache.iceberg.TableProperties.SPLIT_SIZE // "read.split.target-size" def planGroups(splitBytes: Long): Int = { @@ -350,10 +341,9 @@ trait ForkScenarios extends ScenarioKit { } /** - * rewrite_data_files packs data files into rewrite groups weighted by file length. Compacting a - * table whose data files are unevenly sized preserves the row count and every row's value, which - * is the observable result of that packing; the weighting itself is a planner decision that no - * SQL surface exposes. The file format is the parameter. + * rewrite_data_files packs data files into rewrite groups weighted by file length. Compacting a table whose data + * files are unevenly sized preserves the row count and every row's value, which is the observable result of that + * packing; the weighting itself is a planner decision that no SQL surface exposes. The file format is the parameter. */ private def forkBinPackByLength(fmt: String)(ctx: Ctx): Unit = { val spark = ctx.spark @@ -383,11 +373,11 @@ trait ForkScenarios extends ScenarioKit { } /** - * file_sequence_number is exposed on the live data-file entries of the entries metadata table and - * increases monotonically across commits, and rewrite_data_files with rewrite-all preserves the - * row count and the row set. A budgeted rewrite spends its budget in file-sequence-number order, - * so that column is the observable half of the ordering decision. Sequence numbers order commits - * the same way in every file format, so parquet alone covers this behavior. + * file_sequence_number is exposed on the live data-file entries of the entries metadata table and increases + * monotonically across commits, and rewrite_data_files with rewrite-all preserves the row count and the row set. A + * budgeted rewrite spends its budget in file-sequence-number order, so that column is the observable half of the + * ordering decision. Sequence numbers order commits the same way in every file format, so parquet alone covers this + * behavior. */ private def forkCompactionOrder(ctx: Ctx): Unit = { val spark = ctx.spark @@ -446,9 +436,8 @@ trait ForkScenarios extends ScenarioKit { forkPartitionDistDefault("orc"))) /** - * The output-file, split-size and compaction fork cases. They are the second of two fork - * contribution lists: one more fork entry sits between the two in the catalog, supplied by the - * layer that owns it, and Plan keeps that order. + * The output-file, split-size and compaction fork cases. They are the second of two fork contribution lists: one more + * fork entry sits between the two in the catalog, supplied by the layer that owns it, and Plan keeps that order. */ val forkFileAndCompactionCases: List[Plan.Case] = List( diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala index 813c7ab4a..ddb735703 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala @@ -9,14 +9,13 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -// The harness defines typed, reusable table preparations and localized Plan.Case bodies. -// Each case gets a fresh table, executes its preparation, runs its action and assertions, -// and drops the table during teardown. +// The harness defines typed, reusable table preparations and localized Plan.Case bodies. Each case gets a fresh table, +// executes its preparation, runs its action and assertions, and drops the table during teardown. final case class Ctx(spark: SparkSession, namespace: String, restUri: String = "", restToken: String = "") -// Minimal REST client to the embedded OpenHouse server (control-plane ops with no SQL surface: -// lock/unlock). Uses JDK 17's java.net.http; auth is the same Bearer token the Spark catalog uses. +// Minimal REST client to the embedded OpenHouse server (control-plane ops with no SQL surface: lock/unlock). Uses JDK +// 17's java.net.http; auth is the same Bearer token the Spark catalog uses. object Rest { import java.net.http.{HttpClient, HttpRequest, HttpResponse} import java.net.URI @@ -75,8 +74,8 @@ object Exceptions { def root(throwable: Throwable): Throwable = causeChain(throwable).last /** - * Retries errors positively identified as transient. Other failures remain terminal so data, - * permission, and assertion failures surface on their first attempt. + * Retries errors positively identified as transient. Other failures remain terminal so data, permission, and + * assertion failures surface on their first attempt. */ def isTransient(throwable: Throwable): Boolean = causeChain(throwable).exists { case _: java.net.SocketTimeoutException => true @@ -86,12 +85,10 @@ object Exceptions { } } -// Tests assert with plain `assert`; a failed assertion throws AssertionError, which is NonFatal -// and so is caught at the Runner edge and reported as a (terminal) failure. +// Tests assert with plain `assert`; a failed assertion throws AssertionError, which is NonFatal and so is caught at the +// Runner edge and reported as a (terminal) failure. object Check { - /** - * Requires `operation` to throw `E` and returns the exception for message assertions. - */ + /** Requires `operation` to throw `E` and returns the exception for message assertions. */ def intercept[E <: Throwable: ClassTag](operation: => Unit): E = { val expected = classTag[E].runtimeClass val caught: Option[Throwable] = @@ -116,10 +113,9 @@ object Check { } } -// `Column[T]` carries the Scala type the column reads back as, so typed -// row access (`row.get(CoreTable.long0): Long`) is compiler-checked. `literalAt(rowIndex)` is a -// pure function of the row index, so generated data is reproducible. Value generation lives on -// the column, which keeps RowGenerator a plain iteration with no knowledge of types. +// `Column[T]` carries the Scala type the column reads back as, so typed row access (`row.get(CoreTable.long0): Long`) +// is compiler-checked. `literalAt(rowIndex)` is a pure function of the row index, so generated data is reproducible. +// Value generation lives on the column, which keeps RowGenerator a plain iteration with no knowledge of types. final case class Column[T](columnName: String, sqlType: String, literalAt: Int => String) sealed trait Schema { @@ -134,27 +130,29 @@ object Rows { } } -// A representative core table with one column per common data type and a string date partition. -// Tests reference columns through these handles, so a column rename propagates to every caller. +// A representative core table with one column per common data type and a string-encoded date. Tests reference columns +// through these handles, so a column rename propagates to every caller. object CoreTable extends Schema { - val long0: Column[Long] = Column("foo_col_long", "bigint", rowIndex => rowIndex.toString) - val int0: Column[Int] = Column("foo_col_int", "int", rowIndex => rowIndex.toString) - val string0: Column[String] = Column("foo_col_string", "string", rowIndex => s"'row-$rowIndex'") - val double0: Column[Double] = Column("foo_col_double", "double", rowIndex => s"$rowIndex.5") - val boolean0: Column[Boolean] = Column("foo_col_boolean", "boolean", rowIndex => if (rowIndex % 2 == 0) "true" else "false") - val datePartition: Column[String] = Column("datepartition", "string", rowIndex => s"'${CoreTable.datePartitionLiteral(rowIndex)}'") - def tableColumns: Seq[Column[_]] = Seq(long0, int0, string0, double0, boolean0, datePartition) - - private val DatePartitionFormat = DateTimeFormatter.ofPattern("yyyy-MM-dd-HH") - private val DatePartitionEpoch = LocalDateTime.of(2024, 1, 1, 0, 0) - - /** Deterministic YYYY-MM-DD-HH partition value (one hour per row), formatted via java.time. */ - def datePartitionLiteral(rowIndex: Int): String = - DatePartitionEpoch.plusHours((rowIndex - 1).toLong).format(DatePartitionFormat) + val long0: Column[Long] = Column("foo_col_long", "bigint", rowIndex => rowIndex.toString) + val int0: Column[Int] = Column("foo_col_int", "int", rowIndex => rowIndex.toString) + val string0: Column[String] = Column("foo_col_string", "string", rowIndex => s"'row-$rowIndex'") + val double0: Column[Double] = Column("foo_col_double", "double", rowIndex => s"$rowIndex.5") + val boolean0: Column[Boolean] = + Column("foo_col_boolean", "boolean", rowIndex => if (rowIndex % 2 == 0) "true" else "false") + val date0: Column[String] = + Column("foo_col_date", "string", rowIndex => s"'${CoreTable.dateLiteral(rowIndex)}'") + def tableColumns: Seq[Column[_]] = Seq(long0, int0, string0, double0, boolean0, date0) + + private val DateFormat = DateTimeFormatter.ofPattern("yyyy-MM-dd-HH") + private val DateEpoch = LocalDateTime.of(2024, 1, 1, 0, 0) + + /** Deterministic YYYY-MM-DD-HH date value (one hour per row), formatted via java.time. */ + def dateLiteral(rowIndex: Int): String = + DateEpoch.plusHours((rowIndex - 1).toLong).format(DateFormat) } -// A schema exercising complex/nested types: a struct, an array, a map, and a struct-in-struct. -// Struct/array read back as Row/Seq; map as a Map. `id` is first so it is the ordering key. +// A schema exercising complex/nested types: a struct, an array, a map, and a struct-in-struct. Struct/array read back +// as Row/Seq; map as a Map. `id` is first so it is the ordering key. object NestedTable extends Schema { val id: Column[Long] = Column("id", "bigint", rowIndex => rowIndex.toString) val s: Column[Row] = Column("s", "struct", rowIndex => s"named_struct('x', $rowIndex, 'y', 'row-$rowIndex')") @@ -167,8 +165,8 @@ object NestedTable extends Schema { "id bigint, s struct, arr array, m map, nested struct>" } -// A schema for type-edge coverage: the common scalar types, exercised with nulls, special float -// values, boundary values, and unicode/empty strings. +// A schema for type-edge coverage: the common scalar types, exercised with nulls, special float values, boundary +// values, and unicode/empty strings. object TypesTable extends Schema { val id: Column[Long] = Column("id", "bigint", rowIndex => rowIndex.toString) val n: Column[Int] = Column("n", "int", rowIndex => rowIndex.toString) @@ -212,9 +210,9 @@ object RowGenerator { } /** - * What a step's validation thunk sees: the live table, its rows before and after the step, and - * the table's snapshot (commit) count before and after, so a test can assert the delta in both - * data and commits (e.g. "a no-match UPDATE still commits exactly one snapshot"). + * What a step's validation thunk sees: the live table, its rows before and after the step, and the table's snapshot + * (commit) count before and after, so a test can assert the delta in both data and commits (e.g. "a no-match UPDATE + * still commits exactly one snapshot"). */ final case class StepView[S <: Schema]( spark: SparkSession, @@ -262,8 +260,8 @@ final case class Step[S <: Schema]( final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Step[S]]) { private def add(step: Step[S]): TableTest[S] = new TableTest(schema, steps :+ step) - // The default validator asserts the seed actually appended `numberOfRows` rows. This defends the - // localized assertions from a vacuous pass on an empty or short baseline. + // The default validator asserts the seed actually appended `numberOfRows` rows. This defends the localized assertions + // from a vacuous pass on an empty or short baseline. def insert(numberOfRows: Int)( validate: StepView[S] => Unit = view => assert( view.after.size == view.before.size + numberOfRows, @@ -283,8 +281,8 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste step(label)((spark, table) => spark.sql(statement(table)))(validate) /** - * Execute these steps as a reusable preparation, then hand the prepared table to one localized - * test body. The fresh-table lifecycle covers both the preparation and the test body. + * Execute these steps as a reusable preparation, then hand the prepared table to one localized test body. The + * fresh-table lifecycle covers both the preparation and the test body. */ def prepare(ctx: Ctx)(use: PreparedTable[S] => Unit): Unit = withTable(ctx) { (table, markTableCreated) => @@ -311,10 +309,9 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste use(PreparedTable(ctx.spark, table, schema, preparedRows, preparedSnapshotCount)) } - // Gives the preparation a unique table name and drops that table after the test. Cleanup starts - // only after the first preparation step creates the table, so a name conflict preserves the - // pre-existing table. A test failure stays primary, and a cleanup failure is attached to it as a - // suppressed exception. + // Gives the preparation a unique table name and drops that table after the test. Cleanup starts only after the first + // preparation step creates the table, so a name conflict preserves the pre-existing table. A test failure stays + // primary, and a cleanup failure is attached to it as a suppressed exception. private def withTable(ctx: Ctx)(use: (String, () => Unit) => Unit): Unit = { val table = TableTest.nextQualifiedTableName(ctx.namespace) OwnedTableLifecycle.withOwnership( @@ -368,9 +365,8 @@ final case class TablePreparation[S <: Schema]( afterTest: PreparedTable[S] => Unit = (_: PreparedTable[S]) => () ) { /** - * Build the case that runs `body` against one freshly prepared table. The case ID combines the - * preparation's prefix and label with `caseName`, so one test body yields a separate case on - * every preparation it runs on. + * Build the case that runs `body` against one freshly prepared table. The case ID combines the preparation's prefix + * and label with `caseName`, so one test body yields a separate case on every preparation it runs on. */ def test(caseName: String)(body: PreparedTable[S] => Unit): Plan.Case = Plan.Case( diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala index 35d61e487..82cd9e392 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala @@ -10,12 +10,11 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -// The copy-on-write reader, writer and hazard families. The reader and writer cases pin the -// changelog view, the incremental read and the structured-streaming reader and writer against a -// plain copy-on-write table. The hazard cases pin what happens when two operations that can -// interfere are run against the same table. Plan crosses every family here with the parquet and -// orc file formats. `cowCreate` states the standard copy-on-write table shape, so a feature layer -// reaches it through a self-type on this trait. +// The copy-on-write reader, writer and hazard families. The reader and writer cases pin the changelog view, the +// incremental read and the structured-streaming reader and writer against a plain copy-on-write table. The hazard cases +// pin what happens when two operations that can interfere are run against the same table. Plan crosses every family +// here with the parquet and orc file formats. `cowCreate` states the standard copy-on-write table shape, so a feature +// layer reaches it through a self-type on this trait. trait HazardReaderWriterScenarios extends ScenarioKit { import Rows._ @@ -24,8 +23,8 @@ trait HazardReaderWriterScenarios extends ScenarioKit { s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')" /** - * Three seed rows in a copy-on-write table in the given file format. Each family builds its own - * table from this recipe, so a family reads on its own. + * Three seed rows in a copy-on-write table in the given file format. Each family builds its own table from this + * recipe, so a family reads on its own. */ private def cowPreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation( @@ -66,10 +65,7 @@ trait HazardReaderWriterScenarios extends ScenarioKit { List( readerWriterChangelogAppendCase(format)) - /** - * A changelog view over an INSERT OVERWRITE that drops one row reports exactly that row as a - * DELETE. - */ + /** A changelog view over an INSERT OVERWRITE that drops one row reports exactly that row as a DELETE. */ private def readerWriterChangelogOverwriteCase(format: String): Plan.Case = cowPreparation(format).test("readerWriter.changelog.overwrite") { table => val seedSnapshotId = snapshotIds(table.spark, table.name).head @@ -132,10 +128,7 @@ trait HazardReaderWriterScenarios extends ScenarioKit { List( readerWriterChangelogDeleteCase(format)) - /** - * A changelog view over an UPDATE reports the old row as a DELETE and the new value as an - * INSERT. - */ + /** A changelog view over an UPDATE reports the old row as a DELETE and the new value as an INSERT. */ private def readerWriterChangelogUpdateCase(format: String): Plan.Case = cowPreparation(format).test("readerWriter.changelog.update") { table => val seedSnapshotId = snapshotIds(table.spark, table.name).head @@ -166,10 +159,7 @@ trait HazardReaderWriterScenarios extends ScenarioKit { List( readerWriterChangelogUpdateCase(format)) - /** - * A changelog view over a MERGE that updates one row and inserts another reports one DELETE and - * two INSERTs. - */ + /** A changelog view over a MERGE that updates one row and inserts another reports one DELETE and two INSERTs. */ private def readerWriterChangelogMergeCase(format: String): Plan.Case = cowPreparation(format).test("readerWriter.changelog.merge") { table => val seedSnapshotId = snapshotIds(table.spark, table.name).head @@ -182,7 +172,7 @@ trait HazardReaderWriterScenarios extends ScenarioKit { "WHEN NOT MATCHED THEN INSERT " + s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + s"${Core.string0.columnName}, ${Core.double0.columnName}, " + - s"${Core.boolean0.columnName}, ${Core.datePartition.columnName}) " + + s"${Core.boolean0.columnName}, ${Core.date0.columnName}) " + "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") val view = table.spark .sql( @@ -293,8 +283,8 @@ trait HazardReaderWriterScenarios extends ScenarioKit { } /** - * A streaming read of the table delivers the seed rows on first run and the newly inserted row - * after restart, into a destination table. + * A streaming read of the table delivers the seed rows on first run and the newly inserted row after restart, into a + * destination table. */ private def readerWriterStreamAppendCase(format: String): Plan.Case = cowPreparation(format).test("readerWriter.stream.append") { table => @@ -334,8 +324,8 @@ trait HazardReaderWriterScenarios extends ScenarioKit { } /** - * An append-only stream restarted after a DELETE snapshot was written fails, with an error - * mentioning delete or overwrite. + * An append-only stream restarted after a DELETE snapshot was written fails, with an error mentioning delete or + * overwrite. */ private def readerWriterStreamDeleteRejectedCase(format: String): Plan.Case = cowPreparation(format).test("readerWriter.stream.deleteRejected") { table => @@ -379,8 +369,8 @@ trait HazardReaderWriterScenarios extends ScenarioKit { } /** - * The incremental reads between two snapshots and the structured-streaming reader and writer, on - * three seed rows in the given file format. + * The incremental reads between two snapshots and the structured-streaming reader and writer, on three seed rows in + * the given file format. */ def readerWriterIncrementalAndStreamCases(format: String): List[Plan.Case] = List( @@ -392,8 +382,8 @@ trait HazardReaderWriterScenarios extends ScenarioKit { readerWriterStreamDeleteRejectedCase(format)) /** - * A streaming read that resumes after its earliest offset snapshot has been expired fails, with - * an error naming the expired or missing snapshot. + * A streaming read that resumes after its earliest offset snapshot has been expired fails, with an error naming the + * expired or missing snapshot. */ private def hazardStreamExpiredCheckpointCase( format: String, @@ -458,10 +448,10 @@ trait HazardReaderWriterScenarios extends ScenarioKit { } /** - * After expire_snapshots removes a changelog start point, create_changelog_view over that start - * point either throws or reports fewer changes than the table's history holds, and any message it - * throws leaves expiration unnamed. The case covers three start points: an expired snapshot ID, a - * timestamp older than the whole history, and a timestamp inside the expired range. + * After expire_snapshots removes a changelog start point, create_changelog_view over that start point either throws + * or reports fewer changes than the table's history holds, and any message it throws leaves expiration unnamed. The + * case covers three start points: an expired snapshot ID, a timestamp older than the whole history, and a timestamp + * inside the expired range. */ private def hazardCdcExpiredRangeCase( basePreparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -548,8 +538,8 @@ trait HazardReaderWriterScenarios extends ScenarioKit { } /** - * The hazards a reader or a consumer meets when maintenance lands underneath it. Every case - * starts from three seed rows in a copy-on-write table in the given file format. + * The hazards a reader or a consumer meets when maintenance lands underneath it. Every case starts from three seed + * rows in a copy-on-write table in the given file format. */ def hazardReaderCases(format: String): List[Plan.Case] = { val basePreparation = TablePreparation( @@ -564,8 +554,7 @@ trait HazardReaderWriterScenarios extends ScenarioKit { } /** - * An explicit-column INSERT that worked before ADD COLUMN is rejected afterward, with an error - * naming the new column. + * An explicit-column INSERT that worked before ADD COLUMN is rejected afterward, with an error naming the new column. */ private def hazardAddColumnBreaksWritersCase( basePreparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -594,8 +583,8 @@ trait HazardReaderWriterScenarios extends ScenarioKit { } /** - * The hazard an explicit-column writer meets after a column is added. The case starts from three - * seed rows in a copy-on-write table in the given file format. + * The hazard an explicit-column writer meets after a column is added. The case starts from three seed rows in a + * copy-on-write table in the given file format. */ def hazardWriterCases(format: String): List[Plan.Case] = { val basePreparation = TablePreparation( @@ -609,9 +598,9 @@ trait HazardReaderWriterScenarios extends ScenarioKit { } /** - * While a table is REST-locked, an expire_snapshots call is rejected and snapshots keep - * accumulating. After the lock is deleted, expire_snapshots succeeds and the snapshot count - * drops, so the lock blocks every maintenance commit while it is held. + * While a table is REST-locked, an expire_snapshots call is rejected and snapshots keep accumulating. After the lock + * is deleted, expire_snapshots succeeds and the snapshot count drops, so the lock blocks every maintenance commit + * while it is held. */ def hazardLockStarvesMaintenance(ctx: Ctx): Unit = { val spark = ctx.spark diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala index b1b325f02..c40993df4 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala @@ -10,18 +10,18 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -// Pins on the physical form of what the OSS build writes. A case here fixes an implementation -// detail of the shipped write path, so a change to that detail shows up as a failing case. The -// behavior a case pins is an artifact of how OSS is wired, not a documented product feature. +// Pins on the physical form of what the OSS build writes. A case here fixes an implementation detail of the shipped +// write path, so a change to that detail shows up as a failing case. The behavior a case pins is an artifact of how OSS +// is wired, not a documented product feature. trait ImplementationPinScenarios extends ScenarioKit { import Rows._ /** - * A data file's Parquet footer magic bytes are the plaintext PAR1 marker, confirming OSS writes - * table data in plaintext. OpenHouse delegates table-data encryption to an external KMS plugin - * and the OSS build wires no KeyManagementClient into the catalog, so tables use the default - * PlaintextEncryptionManager. A Parquet footer reads PAR1 for plaintext and PARE under modular - * encryption regardless of compression, so that magic value settles which path wrote the file. + * A data file's Parquet footer magic bytes are the plaintext PAR1 marker, confirming OSS writes table data in + * plaintext. OpenHouse delegates table-data encryption to an external KMS plugin and the OSS build wires no + * KeyManagementClient into the catalog, so tables use the default PlaintextEncryptionManager. A Parquet footer reads + * PAR1 for plaintext and PARE under modular encryption regardless of compression, so that magic value settles which + * path wrote the file. */ private def surfacePinDataPlaintextCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala index f4aa03212..1653ca9ca 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala @@ -10,15 +10,15 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -// The standard interaction families. Each case composes two table operations, so it shows how a -// DDL change, a snapshot reference, a maintenance procedure and a property setting behave against -// each other on a plain copy-on-write table. The cases run on parquet and orc. +// The standard interaction families. Each case composes two table operations, so it shows how a DDL change, a snapshot +// reference, a maintenance procedure and a property setting behave against each other on a plain copy-on-write table. +// The cases run on parquet and orc. trait InteractionScenarios extends ScenarioKit { import Rows._ /** - * After ADD COLUMN and an insert into the new column, time travel to the pre-DDL snapshot reads - * the old schema with 3 rows, while a current read sees the new column. + * After ADD COLUMN and an insert into the new column, time travel to the pre-DDL snapshot reads the old schema with 3 + * rows, while a current read sees the new column. */ private def interactDdlTtAfterAddColumnCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -58,9 +58,8 @@ trait InteractionScenarios extends ScenarioKit { } /** - * Rolling back to the pre-DDL snapshot after ADD COLUMN and an insert keeps the evolved schema, - * restores 3 rows reading null for the new column, and the table still accepts writes into that - * column. + * Rolling back to the pre-DDL snapshot after ADD COLUMN and an insert keeps the evolved schema, restores 3 rows + * reading null for the new column, and the table still accepts writes into that column. */ private def interactDdlRestoreAfterAddColumnCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -109,8 +108,8 @@ trait InteractionScenarios extends ScenarioKit { } /** - * DROP COLUMN on a column that holds data is rejected, the column's data remains readable, and - * the table remains writable. + * DROP COLUMN on a column that holds data is rejected, the column's data remains readable, and the table remains + * writable. */ private def interactDdlDropColAfterDataCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -144,10 +143,7 @@ trait InteractionScenarios extends ScenarioKit { "rejected drop should leave the table writable") } - /** - * The DDL interactions. Every case starts from three seed rows in a table in the given file - * format. - */ + /** The DDL interactions. Every case starts from three seed rows in a table in the given file format. */ def interactionDdlCases(format: String): List[Plan.Case] = { val preparation = TablePreparation( format, @@ -164,8 +160,8 @@ trait InteractionScenarios extends ScenarioKit { } /** - * Compacting a table after an ADD COLUMN and inserts into the new column preserves all rows, the - * new column's non-null values, and null for rows written before the column was added. + * Compacting a table after an ADD COLUMN and inserts into the new column preserves all rows, the new column's + * non-null values, and null for rows written before the column was added. */ private def interactMaintCompactEvolvedCase( basePreparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -206,10 +202,7 @@ trait InteractionScenarios extends ScenarioKit { s"pre-evolution rows should remain null, got $nullValueCount") } - /** - * The maintenance interactions. The case starts from three seed rows in a table in the given - * file format. - */ + /** The maintenance interactions. The case starts from three seed rows in a table in the given file format. */ def interactionMiscellaneousCases( format: String): List[Plan.Case] = { val basePreparation = TablePreparation( diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala index 41bd7e848..7bb86d40c 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala @@ -14,16 +14,15 @@ trait MaintControlScenarios extends ScenarioKit { import Rows._ /** - * A five-row table across two snapshots in the given file format: a 3-row seed commit, then a - * 2-row insert committed at a later timestamp. Time travel, restore and maintenance all start - * from this state. + * A five-row table across two snapshots in the given file format: a 3-row seed commit, then a 2-row insert committed + * at a later timestamp. Time travel, restore and maintenance all start from this state. */ private def twoSnapshotPreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation(format, coreTwoSnapshots(format)) /** - * VERSION AS OF the first snapshot ID reads the 3 rows the seed commit wrote, and VERSION AS OF - * the second reads all 5 rows. + * VERSION AS OF the first snapshot ID reads the 3 rows the seed commit wrote, and VERSION AS OF the second reads all + * 5 rows. */ private def timeTravelVersionAsOfCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("timeTravel.versionAsOf") { table => @@ -66,8 +65,8 @@ trait MaintControlScenarios extends ScenarioKit { } /** - * The snapshots and history metadata tables each report the table's 2 snapshots, and the files - * and manifests metadata tables report at least 1 row. + * The snapshots and history metadata tables each report the table's 2 snapshots, and the files and manifests metadata + * tables report at least 1 row. */ private def timeTravelMetadataTablesCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -151,10 +150,7 @@ trait MaintControlScenarios extends ScenarioKit { restoreSetCurrentSnapshotCase(preparation)) } - /** - * expire_snapshots with retain_last=1 removes the seed snapshot and leaves all 5 current rows - * unchanged. - */ + /** expire_snapshots with retain_last=1 removes the seed snapshot and leaves all 5 current rows unchanged. */ private def maintenanceExpireSnapshotsCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("maintenance.expireSnapshots") { table => @@ -208,10 +204,9 @@ trait MaintControlScenarios extends ScenarioKit { } /** - * POSTing a table lock causes a following Spark UPDATE to be rejected server-side with - * LOCKED_TABLE_OPERATION, and DELETEing the lock lets a later UPDATE apply. The lock endpoint has - * no SQL surface, so the case drives it over HTTP against the embedded server, which runs the - * same TablesController and TablesServiceImpl as production. + * POSTing a table lock causes a following Spark UPDATE to be rejected server-side with LOCKED_TABLE_OPERATION, and + * DELETEing the lock lets a later UPDATE apply. The lock endpoint has no SQL surface, so the case drives it over HTTP + * against the embedded server, which runs the same TablesController and TablesServiceImpl as production. */ def controlLockEnforcement(ctx: Ctx): Unit = { val spark = ctx.spark diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala index d5bba4048..753e6d48b 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala @@ -15,10 +15,7 @@ trait NegativeDdlScenarios extends ScenarioKit { private val S = CoreTable.string0.columnName - /** - * DELETE with a WHERE clause on a nonexistent column is rejected with an AnalysisException - * naming that column. - */ + /** DELETE with a WHERE clause on a nonexistent column is rejected with an AnalysisException naming that column. */ private def negativeNonExistentColumnCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("negative.nonExistentColumn") { table => @@ -30,8 +27,7 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * DELETE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an - * AnalysisException about determinism. + * DELETE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an AnalysisException about determinism. */ private def negativeNonDeterministicDeleteCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -45,8 +41,7 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * UPDATE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an - * AnalysisException about determinism. + * UPDATE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an AnalysisException about determinism. */ private def negativeNonDeterministicUpdateCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -60,8 +55,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * INSERT INTO with too few values for the table's columns is rejected with an AnalysisException - * about the missing data columns. + * INSERT INTO with too few values for the table's columns is rejected with an AnalysisException about the missing + * data columns. */ private def negativeInsertArityCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("negative.insertArity") { table => @@ -75,8 +70,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * A MERGE whose UPDATE SET assigns the same target column twice is rejected with an - * AnalysisException about multiple assignments. + * A MERGE whose UPDATE SET assigns the same target column twice is rejected with an AnalysisException about multiple + * assignments. */ private def negativeMergeConflictingUpdatesCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -94,8 +89,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * A MERGE whose source has two rows matching the same target row fails with a - * cardinality-violation error naming the multi-row match. + * A MERGE whose source has two rows matching the same target row fails with a cardinality-violation error naming the + * multi-row match. */ private def negativeMergeCardinalityViolationCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -121,8 +116,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * CREATE TABLE PARTITIONED BY a nonexistent column is rejected with an AnalysisException naming - * that column, and no scratch table is left behind. + * CREATE TABLE PARTITIONED BY a nonexistent column is rejected with an AnalysisException naming that column, and no + * scratch table is left behind. */ private def negativePartitionByNonExistentCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -151,10 +146,7 @@ trait NegativeDdlScenarios extends ScenarioKit { negativePartitionByNonExistentCase(preparation)) } - /** - * ALTER TABLE DROP COLUMN is rejected with a BadRequestException naming the column that would be - * dropped. - */ + /** ALTER TABLE DROP COLUMN is rejected with a BadRequestException naming the column that would be dropped. */ private def ddlNegDropColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.neg.dropColumn") { table => val exception = Check.intercept[BadRequestException]( @@ -170,8 +162,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * ALTER TABLE ALTER COLUMN to a narrower type (bigint to int) is rejected with an - * AnalysisException about the unsupported column change. + * ALTER TABLE ALTER COLUMN to a narrower type (bigint to int) is rejected with an AnalysisException about the + * unsupported column change. */ private def ddlNegNarrowTypeCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.neg.narrowType") { table => @@ -185,8 +177,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * ALTER TABLE ALTER COLUMN SET NOT NULL on a nullable column is rejected with an - * AnalysisException about the nullable-to-non-nullable change. + * ALTER TABLE ALTER COLUMN SET NOT NULL on a nullable column is rejected with an AnalysisException about the + * nullable-to-non-nullable change. */ private def ddlNegSetNotNullCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.neg.setNotNull") { table => @@ -223,8 +215,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * SET TBLPROPERTIES on the reserved openhouse.tableUUID property is rejected with a - * BadRequestException about the restriction. + * SET TBLPROPERTIES on the reserved openhouse.tableUUID property is rejected with a BadRequestException about the + * restriction. */ private def ddlPropsReservedOpenhouseCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -240,8 +232,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * Even though format-version=1 was requested at creation, the table is forced to - * format-version=2 and remains writable. + * Even though format-version=1 was requested at creation, the table is forced to format-version=2 and remains + * writable. */ private def ddlPropsFormatVersionForcedCase( formatVersionPreparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -256,10 +248,7 @@ trait NegativeDdlScenarios extends ScenarioKit { "table not writable at the forced format-version") } - /** - * The write.metadata.previous-versions-max property requested at creation is honored and reads - * back as 7. - */ + /** The write.metadata.previous-versions-max property requested at creation is honored and reads back as 7. */ private def ddlPropsPreviousVersionsHonoredCase( previousVersionsPreparation: TablePreparation[CoreTable.type]): Plan.Case = previousVersionsPreparation.test("ddl.props.previousVersionsHonored") { table => @@ -272,9 +261,9 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * The table-property cases. Two of them start from the preparedCoreFormats preparation for the - * file format, one from a table created with format-version=1 requested, and one from an unseeded - * table created with write.metadata.previous-versions-max=7. + * The table-property cases. Two of them start from the preparedCoreFormats preparation for the file format, one from + * a table created with format-version=1 requested, and one from an unseeded table created with + * write.metadata.previous-versions-max=7. */ val ddlPropertyCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => val format = preparation.label @@ -313,8 +302,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * ALTER TABLE WRITE ORDERED BY multiple columns sets range distribution and the table remains - * writable, growing from 3 to 5 rows after a follow-up insert. + * ALTER TABLE WRITE ORDERED BY multiple columns sets range distribution and the table remains writable, growing from + * 3 to 5 rows after a follow-up insert. */ private def ddlSortOrderOrderedByMultiCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -334,9 +323,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * ALTER TABLE RENAME TO moves the table to the new name with its 3 rows intact, and the old name - * stops resolving. A second rename puts the table back under its original name, which teardown - * drops. + * ALTER TABLE RENAME TO moves the table to the new name with its 3 rows intact, and the old name stops resolving. A + * second rename puts the table back under its original name, which teardown drops. */ private def ddlRenameTableCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.renameTable") { table => @@ -351,10 +339,7 @@ trait NegativeDdlScenarios extends ScenarioKit { table.spark.sql(s"ALTER TABLE $renamedTable RENAME TO ${table.name}") } - /** - * ALTER TABLE RENAME TO a name that already exists is rejected with an error naming the - * conflict. - */ + /** ALTER TABLE RENAME TO a name that already exists is rejected with an error naming the conflict. */ private def ddlRenameTableConflictCase( preparation: TablePreparation[CoreTable.type], format: String): Plan.Case = @@ -375,8 +360,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * CREATE NAMESPACE is rejected with an UnsupportedOperationException, since this catalog does - * not support creating namespaces. + * CREATE NAMESPACE is rejected with an UnsupportedOperationException, since this catalog does not support creating + * namespaces. */ private def ddlNsCreateRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.ns.createRejected") { table => @@ -389,8 +374,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * DROP NAMESPACE is rejected with an UnsupportedOperationException, since this catalog does not - * support dropping namespaces. + * DROP NAMESPACE is rejected with an UnsupportedOperationException, since this catalog does not support dropping + * namespaces. */ private def ddlNsDropRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.ns.dropRejected") { table => @@ -432,10 +417,7 @@ trait NegativeDdlScenarios extends ScenarioKit { "table not queryable after SET POLICY (SHARING)") } - /** - * SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20) records the history policy and the table remains - * queryable. - */ + /** SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20) records the history policy and the table remains queryable. */ private def ddlPolicyHistoryCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.policy.history") { table => table.spark.sql( @@ -452,8 +434,7 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * SET POLICY (REPLICATION) followed by UNSET POLICY (REPLICATION) leaves the table queryable - * with its 3 rows intact. + * SET POLICY (REPLICATION) followed by UNSET POLICY (REPLICATION) leaves the table queryable with its 3 rows intact. */ private def ddlPolicyReplicationCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.policy.replication") { table => @@ -466,15 +447,15 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * SET POLICY (RETENTION = 30d ON COLUMN datepartition ...) records the retention policy and the - * table remains queryable. + * SET POLICY (RETENTION = 30d ON COLUMN foo_col_date ...) records the retention policy and the table remains + * queryable. */ private def ddlPolicyRetentionCase( retentionPreparation: TablePreparation[CoreTable.type]): Plan.Case = retentionPreparation.test("ddl.policy.retention") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET POLICY (" + - "RETENTION = 30d ON COLUMN datepartition WHERE pattern = 'yyyy-MM-dd-HH')") + s"RETENTION = 30d ON COLUMN ${Core.date0.columnName} WHERE pattern = 'yyyy-MM-dd-HH')") val policies = tableProps(table.spark, table.name).getOrElse("policies", "") @@ -487,8 +468,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * SET POLICY (HISTORY MAX_AGE=5D) exceeds the allowed range and is rejected with a - * BadRequestException stating the 1-to-3-day limit. + * SET POLICY (HISTORY MAX_AGE=5D) exceeds the allowed range and is rejected with a BadRequestException stating the + * 1-to-3-day limit. */ private def ddlPolicyNegHistoryMaxAgeCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -503,8 +484,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * SET POLICY (HISTORY VERSIONS=200) exceeds the allowed range and is rejected with a - * BadRequestException stating the 2-to-100-version limit. + * SET POLICY (HISTORY VERSIONS=200) exceeds the allowed range and is rejected with a BadRequestException stating the + * 2-to-100-version limit. */ private def ddlPolicyNegHistoryVersionsCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -519,9 +500,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * The table-policy cases. They start from the preparedCoreFormats preparation for the file - * format, except the retention case, which starts from three seed rows in a table partitioned by - * datepartition. + * The table-policy cases. They start from the preparedCoreFormats preparation for the file format, except the + * retention case, which starts from three seed rows in a table partitioned by the date column. */ val ddlPolicyCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => val format = preparation.label @@ -530,7 +510,7 @@ trait NegativeDdlScenarios extends ScenarioKit { TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "PARTITIONED BY (datepartition) " + + s"PARTITIONED BY (${Core.date0.columnName}) " + s"TBLPROPERTIES ('write.format.default'='$format')")() .insert(3)()) @@ -544,8 +524,7 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * ALTER TABLE MODIFY COLUMN SET TAG = (PII) tags a column without masking or changing the values - * that queries return. + * ALTER TABLE MODIFY COLUMN SET TAG = (PII) tags a column without masking or changing the values that queries return. */ private def ddlColTagCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.colTag") { table => @@ -567,8 +546,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * GRANT SELECT on a table that is not marked shared is rejected with an IllegalArgumentException - * stating the table is not shared. + * GRANT SELECT on a table that is not marked shared is rejected with an IllegalArgumentException stating the table is + * not shared. */ private def ddlAclGrantUnsharedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("ddl.acl.grantUnshared") { table => @@ -581,8 +560,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * On a shared table, GRANT SELECT TO PUBLIC makes SHOW GRANTS list SELECT for PUBLIC and the - * table stays queryable; REVOKE SELECT then removes that grant from SHOW GRANTS. + * On a shared table, GRANT SELECT TO PUBLIC makes SHOW GRANTS list SELECT for PUBLIC and the table stays queryable; + * REVOKE SELECT then removes that grant from SHOW GRANTS. */ private def ddlAclGrantSharedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation @@ -618,8 +597,7 @@ trait NegativeDdlScenarios extends ScenarioKit { "authorization service.")) /** - * The write.distribution-mode=none property requested at creation is honored and the table - * remains writable under it. + * The write.distribution-mode=none property requested at creation is honored and the table remains writable under it. */ private def ddlFeatureFlagDistributionModeCase( distributionModePreparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -636,8 +614,8 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * ALTER TABLE SET TBLPROPERTIES ('openhouse.tableType'='REPLICA_TABLE') is rejected with a - * BadRequestException, since table type cannot be changed after creation. + * ALTER TABLE SET TBLPROPERTIES ('openhouse.tableType'='REPLICA_TABLE') is rejected with a BadRequestException, since + * table type cannot be changed after creation. */ private def ddlReplTableTypeImmutableCase( preparation: TablePreparation[CoreTable.type]): Plan.Case = @@ -653,9 +631,9 @@ trait NegativeDdlScenarios extends ScenarioKit { } /** - * The column-tag, ACL and feature-flag cases. They start from the preparedCoreFormats preparation - * for the file format, except the distribution-mode case, which starts from three seed rows in a - * table created with write.distribution-mode=none. + * The column-tag, ACL and feature-flag cases. They start from the preparedCoreFormats preparation for the file + * format, except the distribution-mode case, which starts from three seed rows in a table created with + * write.distribution-mode=none. */ val ddlTagAclFeatureCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => val format = preparation.label diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala index 6083cce15..1ff8e6e13 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala @@ -25,8 +25,8 @@ trait NestedTypesScenarios extends ScenarioKit { TableTest(NestedTable).sql("create")(layout.create)().insert(numberOfRows)() /** - * Selecting the top-level id alongside struct, array, map and nested-struct fields reads back - * exactly the seeded values for all 3 rows. + * Selecting the top-level id alongside struct, array, map and nested-struct fields reads back exactly the seeded + * values for all 3 rows. */ private def nestedRoundtripCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = preparation.test("nested.roundtrip") { table => @@ -57,10 +57,7 @@ trait NestedTypesScenarios extends ScenarioKit { assert(actual == expected) } - /** - * Selecting only a nested struct field (s.x) returns just that field's values for all 3 rows, in - * id order. - */ + /** Selecting only a nested struct field (s.x) returns just that field's values for all 3 rows, in id order. */ private def nestedProjectFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = preparation.test("nested.projectField") { table => val actual = table.spark @@ -85,10 +82,7 @@ trait NestedTypesScenarios extends ScenarioKit { assert(actual == Seq(2L)) } - /** - * UPDATE SET s.x = 99 WHERE id = 2 changes only that row's nested field and leaves every other - * row unchanged. - */ + /** UPDATE SET s.x = 99 WHERE id = 2 changes only that row's nested field and leaves every other row unchanged. */ private def nestedUpdateStructFieldCase( preparation: TablePreparation[NestedTable.type]): Plan.Case = preparation.test("nested.updateStructField") { table => @@ -108,8 +102,8 @@ trait NestedTypesScenarios extends ScenarioKit { } /** - * MERGE WHEN NOT MATCHED THEN INSERT with a fully nested source row adds a 4th row whose nested - * struct field reads back as inserted. + * MERGE WHEN NOT MATCHED THEN INSERT with a fully nested source row adds a 4th row whose nested struct field reads + * back as inserted. */ private def nestedMergeInsertCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = preparation.test("nested.mergeInsert") { table => @@ -140,10 +134,7 @@ trait NestedTypesScenarios extends ScenarioKit { .getInt(0) == 4) } - /** - * DELETE WHERE s.x = 2 filtering on a nested struct field removes only the matching row, leaving - * ids 1 and 3. - */ + /** DELETE WHERE s.x = 2 filtering on a nested struct field removes only the matching row, leaving ids 1 and 3. */ private def nestedDeleteByNestedFieldCase( preparation: TablePreparation[NestedTable.type]): Plan.Case = preparation @@ -164,8 +155,8 @@ trait NestedTypesScenarios extends ScenarioKit { "rewrite.")) /** - * Inserting a row with NULL struct, empty array and empty map reads back a null struct and an - * empty array for that row. + * Inserting a row with NULL struct, empty array and empty map reads back a null struct and an empty array for that + * row. */ private def nestedNullValuesCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = preparation.test("nested.nullValues") { table => @@ -186,8 +177,8 @@ trait NestedTypesScenarios extends ScenarioKit { } /** - * The nested-type cases. Each preparation holds three seed rows with struct, array, map and - * doubly-nested struct fields in one unpartitioned nested layout. + * The nested-type cases. Each preparation holds three seed rows with struct, array, map and doubly-nested struct + * fields in one unpartitioned nested layout. */ val nestedCases: List[Plan.Case] = nestedLayouts @@ -228,8 +219,8 @@ trait NestedTypesScenarios extends ScenarioKit { s"DATE '${timestamp.take(10)}', TIMESTAMP '$timestamp', TIMESTAMP_NTZ '$timestamp')" /** - * Selecting id, n, x, dec and str for the first seeded row reads back the exact long, int, - * double, decimal and string values that were seeded. + * Selecting id, n, x, dec and str for the first seeded row reads back the exact long, int, double, decimal and string + * values that were seeded. */ private def typesRoundtripCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = preparation.test("types.roundtrip") { table => @@ -249,8 +240,8 @@ trait NestedTypesScenarios extends ScenarioKit { } /** - * Inserting a row with every non-key column NULL reads back as null for the int, double, string, - * timestamp and timestamp_ntz columns. + * Inserting a row with every non-key column NULL reads back as null for the int, double, string, timestamp and + * timestamp_ntz columns. */ private def typesNullsCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = preparation.test("types.nulls") { table => @@ -267,10 +258,7 @@ trait NestedTypesScenarios extends ScenarioKit { assert((0 to 4).forall(row.isNullAt)) } - /** - * Inserting rows with double('NaN') and double('Infinity') reads back as NaN and positive - * infinity respectively. - */ + /** Inserting rows with double('NaN') and double('Infinity') reads back as NaN and positive infinity respectively. */ private def typesSpecialFloatsCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = preparation.test("types.specialFloats") { table => table.spark.sql( @@ -293,8 +281,8 @@ trait NestedTypesScenarios extends ScenarioKit { } /** - * Inserting a row at Long.MaxValue, Int.MaxValue and a max-precision decimal reads those - * boundary values back unchanged. + * Inserting a row at Long.MaxValue, Int.MaxValue and a max-precision decimal reads those boundary values back + * unchanged. */ private def typesBoundariesCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = preparation.test("types.boundaries") { table => @@ -341,8 +329,8 @@ trait NestedTypesScenarios extends ScenarioKit { } /** - * The type-edge cases. Each preparation holds three seed rows covering the int, double, decimal, - * string, binary, date, timestamp and timestamp_ntz columns in one unpartitioned types layout. + * The type-edge cases. Each preparation holds three seed rows covering the int, double, decimal, string, binary, + * date, timestamp and timestamp_ntz columns in one unpartitioned types layout. */ val typesCases: List[Plan.Case] = typesLayouts @@ -362,10 +350,9 @@ trait NestedTypesScenarios extends ScenarioKit { // Partition transforms and evolution. /** - * One supported partition transform: a table PARTITIONED BY that transform reports a single - * partition field with the expected name in its partitions metadata table, and the seeded rows - * land in the expected number of distinct partitions. The transform, its partition field name, - * and that partition count are the parameters. + * One supported partition transform: a table PARTITIONED BY that transform reports a single partition field with the + * expected name in its partitions metadata table, and the seeded rows land in the expected number of distinct + * partitions. The transform, its partition field name, and that partition count are the parameters. */ private def supportedPartitionTransformCase( format: String, @@ -405,9 +392,9 @@ trait NestedTypesScenarios extends ScenarioKit { } /** - * One rejected partition transform: CREATE TABLE PARTITIONED BY that transform fails with a - * RuntimeException carrying the expected message, and no scratch table is left behind. The - * transform and the expected message are the parameters. + * One rejected partition transform: CREATE TABLE PARTITIONED BY that transform fails with a RuntimeException carrying + * the expected message, and no scratch table is left behind. The transform and the expected message are the + * parameters. */ private def rejectedPartitionTransformCase( format: String, @@ -464,9 +451,9 @@ trait NestedTypesScenarios extends ScenarioKit { } /** - * On three seed rows in an unpartitioned table in the given file format, ALTER TABLE ADD - * PARTITION FIELD is rejected with an exception stating that evolution of table partitioning is - * unsupported, which leaves recreating the table as the way to change partitioning. + * On three seed rows in an unpartitioned table in the given file format, ALTER TABLE ADD PARTITION FIELD is rejected + * with an exception stating that evolution of table partitioning is unsupported, which leaves recreating the table as + * the way to change partitioning. */ private def partitionEvolutionAddRejectedCase(format: String): Plan.Case = TablePreparation( @@ -479,16 +466,15 @@ trait NestedTypesScenarios extends ScenarioKit { .test("partition.evolutionAdd.rejected") { table => val exception = Check.intercept[Exception]( table.spark.sql( - s"ALTER TABLE ${table.name} ADD PARTITION FIELD datepartition")) + s"ALTER TABLE ${table.name} ADD PARTITION FIELD ${Core.date0.columnName}")) assert( exception.getMessage.contains("Evolution of table partitioning")) } /** - * On three seed rows in a table partitioned by datepartition in the given file format, ALTER - * TABLE DROP PARTITION FIELD is rejected with an exception stating that evolution of table - * partitioning is unsupported. + * On three seed rows in a table partitioned by the date column in the given file format, ALTER TABLE DROP PARTITION + * FIELD is rejected with an exception stating that evolution of table partitioning is unsupported. */ private def partitionEvolutionDropRejectedCase(format: String): Plan.Case = TablePreparation( @@ -496,13 +482,13 @@ trait NestedTypesScenarios extends ScenarioKit { TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "PARTITIONED BY (datepartition) " + + s"PARTITIONED BY (${Core.date0.columnName}) " + s"TBLPROPERTIES ('write.format.default'='$format')")() .insert(3)()) .test("partition.evolutionDrop.rejected") { table => val exception = Check.intercept[Exception]( table.spark.sql( - s"ALTER TABLE ${table.name} DROP PARTITION FIELD datepartition")) + s"ALTER TABLE ${table.name} DROP PARTITION FIELD ${Core.date0.columnName}")) assert( exception.getMessage.contains("Evolution of table partitioning")) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala index 32ad8acad..58aa8ae1a 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala @@ -15,9 +15,9 @@ object Plan { def bugReason(testCase: Case): Option[String] = testCase.knownBugReason.map(reason => s"bug: $reason") - // The interaction, surface, reader/writer and hazard families are crossed with these two file - // formats. Each family runs on one format before the next format starts, so the format loop is the - // outer one and every contribution below keeps the catalog position it holds today. + // The interaction, surface, reader/writer and hazard families are crossed with these two file formats. Each family + // runs on one format before the next format starts, so the format loop is the outer one and every contribution below + // keeps the catalog position it holds today. private val crossedFormats: List[String] = List("parquet", "orc") private def interactionContributions: List[Case] = @@ -62,8 +62,8 @@ object Plan { ).flatten } - // Every DDL-consumer family runs against one evolved preparation before the next preparation - // starts, so the preparation loop is the outer one here. + // Every DDL-consumer family runs against one evolved preparation before the next preparation starts, so the + // preparation loop is the outer one here. private def ddlConsumerContributions: List[Case] = Scenarios.ddlConsumerPreparations.flatMap { preparation => List( diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala index a2ebf9ddf..d0b358a2f 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala @@ -10,22 +10,21 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -// Shared foundation for every Scenario trait: the standard table/layout/prep "kit". All domain -// traits (DmlScenarios, ForkScenarios, ...) extend this, so mixing them into `object Scenarios` puts -// ScenarioKit first in the linearization, so its vals initialize before any domain's, exactly as -// in the original single object. It holds the copy-on-write layouts and preparations only; each -// feature layer carries its own kit that extends this one. `protected` members are the shared kit; -// `public` ones are also consumed by `object Plan`. +// Shared foundation for every Scenario trait: the standard table/layout/prep "kit". All domain traits (DmlScenarios, +// ForkScenarios, ...) extend this, so mixing them into `object Scenarios` puts ScenarioKit first in the linearization, +// so its vals initialize before any domain's, exactly as in the original single object. It holds the copy-on-write +// layouts and preparations only; each feature layer carries its own kit that extends this one. `protected` members are +// the shared kit; `public` ones are also consumed by `object Plan`. trait ScenarioKit { import Rows._ protected val Core = CoreTable // brevity in the typed column references below protected val cols = Core.columnNames.mkString(", ") // source column list, so renames propagate - // The rows a case reads back are ordered by the long column and carry exactly the core columns in - // their declared order, so an expected row set is written as the rows the case started from, - // filtered, mapped through `withColumnValue`, extended with literal rows, and re-sorted. Both - // helpers address columns by position so they also work on the literal rows a case writes out. + // The rows a case reads back are ordered by the long column and carry exactly the core columns in their declared + // order, so an expected row set is written as the rows the case started from, filtered, mapped through + // `withColumnValue`, extended with literal rows, and re-sorted. Both helpers address columns by position so they also + // work on the literal rows a case writes out. private def columnPosition(column: Column[_]): Int = Core.columnNames.indexOf(column.columnName) protected def withColumnValue[T](row: Row, column: Column[T], value: T): Row = @@ -35,28 +34,26 @@ trait ScenarioKit { rows.sortBy(_.getLong(columnPosition(Core.long0))) // --- layouts: one file format and one partitioning per starting table shape --- - // A layout is one starting table shape. Each layout is a plain literal CREATE statement: the - // column list is one shared literal `columnDefinitions`, and format and partitioning are literal - // fragments. createSchema cross-checks the literal against CoreTable's declared columns, so the - // two stay in step. A layout belongs to the preparation, so one test case is written once and - // runs on every layout. + // A layout is one starting table shape. Each layout is a plain literal CREATE statement: the column list is one + // shared literal `columnDefinitions`, and format and partitioning are literal fragments. createSchema cross-checks + // the literal against CoreTable's declared columns, so the two stay in step. A layout belongs to the preparation, so + // one test case is written once and runs on every layout. protected val columnDefinitions = - "foo_col_long bigint, foo_col_int int, foo_col_string string, foo_col_double double, foo_col_boolean boolean, datepartition string" + "foo_col_long bigint, foo_col_int int, foo_col_string string, foo_col_double double, " + + "foo_col_boolean boolean, foo_col_date string" - /** One starting table shape: the label that names it in a case ID and the CREATE statement that - * builds it. */ + /** One starting table shape: the label that names it in a case ID and the CREATE statement that builds it. */ final case class Layout(label: String, create: String => String) - /** One partitioning choice: the label that names it in a case ID and the CREATE clause that - * applies it. */ + /** One partitioning choice: the label that names it in a case ID and the CREATE clause that applies it. */ final case class Partitioning(label: String, clause: String) /** The empty partitioning clause: the table keeps all its rows in one unpartitioned file set. */ protected val unpartitioned = Partitioning("unpartitioned", "") - /** Partitions the table by datepartition, so each distinct date value owns one partition. */ + /** Partitions the table by its date column, so each distinct date value owns one partition. */ protected val partitionedByDate = - Partitioning("partitioned", "PARTITIONED BY (datepartition)") + Partitioning("partitioned", s"PARTITIONED BY (${Core.date0.columnName})") protected val partitionings: List[Partitioning] = List(unpartitioned, partitionedByDate) @@ -77,13 +74,13 @@ trait ScenarioKit { partitioning <- partitionings } yield coreLayout(partitioning, format) - /** The core layouts partitioned by datepartition, one per file format. */ + /** The core layouts partitioned by the date column, one per file format. */ val partitionedLayouts: List[Layout] = fileFormats.map(format => coreLayout(partitionedByDate, format)) /** - * The Parquet and ORC core layouts, each crossed with both partitionings, for the bespoke DDL - * cases that do not need the full file-format cross. + * The Parquet and ORC core layouts, each crossed with both partitionings, for the bespoke DDL cases that do not need + * the full file-format cross. */ val parquetAndOrcLayouts: List[Layout] = for { @@ -102,10 +99,7 @@ trait ScenarioKit { layout.label, createAndSeed(layout, 3))) - /** - * One preparation per datepartition-partitioned core layout: three seed rows with keys 1, 2 and - * 3, one row per datepartition value. - */ + /** One preparation per date-partitioned core layout: three seed rows with keys 1, 2 and 3, one row per date value. */ val preparedPartitionedCoreTables: List[TablePreparation[CoreTable.type]] = partitionedLayouts.map(layout => TablePreparation( @@ -113,8 +107,8 @@ trait ScenarioKit { createAndSeed(layout, 3))) /** - * One preparation per core layout: three seed rows, then ALTER TABLE WRITE ORDERED BY the long - * key, so the table carries that write sort order. + * One preparation per core layout: three seed rows, then ALTER TABLE WRITE ORDERED BY the long key, so the table + * carries that write sort order. */ val preparedOrderedCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => @@ -124,8 +118,8 @@ trait ScenarioKit { "prep.ordered:")) /** - * One preparation per core layout: three seed rows, then ADD COLUMN prep_extra int, so the table - * carries one column beyond the seed row shape and the seeded rows read null for it. + * One preparation per core layout: three seed rows, then ADD COLUMN prep_extra int, so the table carries one column + * beyond the seed row shape and the seeded rows read null for it. */ val preparedEvolvedCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => @@ -134,18 +128,14 @@ trait ScenarioKit { createAndSeedEvolved(layout, 3), "prep.evolved:")) - /** - * One preparation per core layout: the table is created and left unseeded, so it holds no rows. - */ + /** One preparation per core layout: the table is created and left unseeded, so it holds no rows. */ val preparedEmptyCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation( layout.label, TableTest(Core).sql("create")(layout.create)())) - /** - * One preparation per Parquet and ORC unpartitioned layout: three seed rows with keys 1, 2 and 3. - */ + /** One preparation per Parquet and ORC unpartitioned layout: three seed rows with keys 1, 2 and 3. */ val preparedCoreFormats: List[TablePreparation[CoreTable.type]] = List("parquet", "orc").map { format => val layout = coreLayout(unpartitioned, format) @@ -155,23 +145,22 @@ trait ScenarioKit { } /** - * Creates and seeds the table under `layout`, then gives it a write sort order on the long key. - * The column list stays as seeded, so every DML case runs on the result. + * Creates and seeds the table under `layout`, then gives it a write sort order on the long key. The column list stays + * as seeded, so every DML case runs on the result. */ def createAndSeedOrdered(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = createAndSeed(layout, numberOfRows).sql("prep.ordered")(t => s"ALTER TABLE $t WRITE ORDERED BY ${CoreTable.long0.columnName}")() /** - * Creates and seeds the table under `layout`, then adds the prep_extra column. The column list - * grows past the seed row shape, so the cases that address columns by name run on the result: - * the reads, the deletes and the updates. + * Creates and seeds the table under `layout`, then adds the prep_extra column. The column list grows past the seed + * row shape, so the cases that address columns by name run on the result: the reads, the deletes and the updates. */ def createAndSeedEvolved(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = createAndSeed(layout, numberOfRows).sql("prep.evolved")(t => s"ALTER TABLE $t ADD COLUMN prep_extra int")() /** - * The same starting state with a fourth row whose key is 99 and whose string column is null, so - * exactly one row of the table reads null for that column. + * The same starting state with a fourth row whose key is 99 and whose string column is null, so exactly one row of + * the table reads null for that column. */ protected def withNullStringRow( basePreparation: TablePreparation[CoreTable.type] @@ -189,8 +178,8 @@ trait ScenarioKit { preparedOrderedCoreTables.map(withNullStringRow) /** - * Every data file the preparation wrote carries the extension of the table's declared - * write.format.default, and listing the files leaves the table state unchanged. + * Every data file the preparation wrote carries the extension of the table's declared write.format.default, and + * listing the files leaves the table state unchanged. */ private def formatMaterializationCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = preparation.test("format.materialization") { table => @@ -213,8 +202,8 @@ trait ScenarioKit { } /** - * The format-materialization case for each preparation given. It applies to any preparation that - * leaves data files behind, so each feature layer passes the list its own preparations produce. + * The format-materialization case for each preparation given. It applies to any preparation that leaves data files + * behind, so each feature layer passes the list its own preparations produce. */ def layoutFormatCasesFor( preparations: List[TablePreparation[CoreTable.type]] @@ -254,8 +243,8 @@ trait ScenarioKit { // Shared helpers used across domain traits. /** - * Creates a table in the given file format, seeds three rows as the first snapshot, then inserts - * rows 4 and 5 as a second snapshot committed at a later timestamp. + * Creates a table in the given file format, seeds three rows as the first snapshot, then inserts rows 4 and 5 as a + * second snapshot committed at a later timestamp. */ protected def coreTwoSnapshots(fmt: String): TableTest[CoreTable.type] = TableTest(Core) @@ -268,8 +257,8 @@ trait ScenarioKit { /** The two-snapshot table in parquet. */ protected def coreTwoSnapshots: TableTest[CoreTable.type] = coreTwoSnapshots("parquet") - // Snapshots in ancestry order (root first), following the parent_id chain. This is deterministic even - // if two commits happen to share a committed_at millisecond (which `ORDER BY committed_at` is not). + // Snapshots in ancestry order (root first), following the parent_id chain. This is deterministic even if two commits + // happen to share a committed_at millisecond (which `ORDER BY committed_at` is not). protected def snapshotIds(spark: SparkSession, table: String): Seq[Long] = { val rows = spark.sql(s"SELECT snapshot_id, parent_id FROM $table.snapshots").collect().toSeq val ids = rows.map(_.getLong(0)).toSet @@ -288,8 +277,8 @@ trait ScenarioKit { protected val L = CoreTable.long0.columnName - // The Spark data source used by CREATE TABLE statements. The LinkedIn adapter overrides this before - // building Plan.cases. Catalog procedure calls still use the catalog name "openhouse". + // The Spark data source used by CREATE TABLE statements. The LinkedIn adapter overrides this before building + // Plan.cases. Catalog procedure calls still use the catalog name "openhouse". var dataSource: String = "iceberg" protected def coreCreateParquet(table: String): String = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala index b01392fda..5be1e36fb 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala @@ -10,11 +10,10 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -// The standard surface families. A surface case pins one edge of what the catalog exposes on a -// plain copy-on-write table: a reader, a procedure, a metadata table, a concurrency outcome, a -// schema change, or a write property. Each family builds the starting states it needs, so a family -// reads on its own. The concurrency helpers below are feature neutral, so a feature layer reuses -// them through a self-type on this trait. The cases run on parquet and orc. +// The standard surface families. A surface case pins one edge of what the catalog exposes on a plain copy-on-write +// table: a reader, a procedure, a metadata table, a concurrency outcome, a schema change, or a write property. Each +// family builds the starting states it needs, so a family reads on its own. The concurrency helpers below are feature +// neutral, so a feature layer reuses them through a self-type on this trait. The cases run on parquet and orc. trait SurfaceScenarios extends ScenarioKit { import Rows._ @@ -70,8 +69,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * Three seed rows with keys 1, 2 and 3 in an unpartitioned table in the given file format. This - * is the plainest starting state here, so the feature layers build their cases on it too. + * Three seed rows with keys 1, 2 and 3 in an unpartitioned table in the given file format. This is the plainest + * starting state here, so the feature layers build their cases on it too. */ protected def surfaceBasePreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation( @@ -83,8 +82,8 @@ trait SurfaceScenarios extends ScenarioKit { .insert(3)()) /** - * Five rows across two snapshots, a three-row seed then a two-row insert, in an unpartitioned - * table in the given file format. + * Five rows across two snapshots, a three-row seed then a two-row insert, in an unpartitioned table in the given file + * format. */ private def surfaceTwoSnapshotPreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation( @@ -109,7 +108,7 @@ trait SurfaceScenarios extends ScenarioKit { s"TBLPROPERTIES ('write.format.default'='$format')")()) /** - * Three seed rows in a table in the given file format, partitioned by datepartition and carrying + * Three seed rows in a table in the given file format, partitioned by the date column and carrying * write.distribution-mode=hash. */ private def surfaceHashPreparation(format: String): TablePreparation[CoreTable.type] = @@ -118,15 +117,14 @@ trait SurfaceScenarios extends ScenarioKit { TableTest(Core) .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.datePartition.columnName}) " + + s"PARTITIONED BY (${Core.date0.columnName}) " + "TBLPROPERTIES (" + s"'write.format.default'='$format', " + "'write.distribution-mode'='hash')")() .insert(3)()) /** - * Three seed rows in an unpartitioned table in the given file format, carrying - * write.target-file-size-bytes=1048576. + * Three seed rows in an unpartitioned table in the given file format, carrying write.target-file-size-bytes=1048576. */ private def surfaceTargetFileSizePreparation(format: String): TablePreparation[CoreTable.type] = TablePreparation( @@ -140,8 +138,8 @@ trait SurfaceScenarios extends ScenarioKit { .insert(3)()) /** - * A Spark structured streaming read of the table, run in AvailableNow batch mode, delivers all 3 - * seed rows to a memory sink within 120 seconds. + * A Spark structured streaming read of the table, run in AvailableNow batch mode, delivers all 3 seed rows to a + * memory sink within 120 seconds. */ private def surfaceStreamReadCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.stream.read") { table => @@ -166,8 +164,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * A Spark structured streaming append of two rows through the iceberg write-stream format lands - * both rows, growing the table from 3 to 5 rows. + * A Spark structured streaming append of two rows through the iceberg write-stream format lands both rows, growing + * the table from 3 to 5 rows. */ private def surfaceStreamWriteCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.stream.write") { table => @@ -183,7 +181,7 @@ trait SurfaceScenarios extends ScenarioKit { s"concat('row-', value) AS ${Core.string0.columnName}", s"CAST(value AS DOUBLE) AS ${Core.double0.columnName}", s"true AS ${Core.boolean0.columnName}", - s"'2024-01-01-00' AS ${Core.datePartition.columnName}") + s"'2024-01-01-00' AS ${Core.date0.columnName}") val checkpoint = java.nio.file.Files.createTempDirectory("ck-write").toString val query = rows.writeStream @@ -201,10 +199,7 @@ trait SurfaceScenarios extends ScenarioKit { "streaming write should append two rows") } - /** - * create_changelog_view over an append-only history reports 5 changes, all of change type - * INSERT. - */ + /** create_changelog_view over an append-only history reports 5 changes, all of change type INSERT. */ private def surfaceCdcChangelogViewCase(format: String): Plan.Case = surfaceTwoSnapshotPreparation(format).test("surface.cdc.changelogView") { table => val view = table.spark @@ -239,8 +234,8 @@ trait SurfaceScenarios extends ScenarioKit { surfaceCdcChangelogViewCase(format)) /** - * After 5 single-row inserts fragment the manifest list, rewrite_manifests compacts it to fewer - * manifests while preserving all 5 rows. + * After 5 single-row inserts fragment the manifest list, rewrite_manifests compacts it to fewer manifests while + * preserving all 5 rows. */ private def surfaceProcRewriteManifestsCase(format: String): Plan.Case = surfaceEmptyPreparation(format).test("surface.proc.rewriteManifests") { table => @@ -276,8 +271,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * The rewrite procedure that compacts the manifest set. The case starts from an unseeded table - * in the given file format and fragments the manifest list itself. + * The rewrite procedure that compacts the manifest set. The case starts from an unseeded table in the given file + * format and fragments the manifest list itself. */ def surfaceRewriteProcedureCases(format: String): List[Plan.Case] = List( @@ -299,8 +294,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * remove_orphan_files deletes a planted, backdated stray file next to a real data file while the - * table's 3 live rows remain intact. + * remove_orphan_files deletes a planted, backdated stray file next to a real data file while the table's 3 live rows + * remain intact. */ private def surfaceProcRemoveOrphanRealCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.proc.removeOrphanReal") { table => @@ -335,8 +330,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * The procedures that read snapshot ancestry and remove orphan files. Ancestry runs on a - * two-snapshot table and orphan removal on a seeded table, each in the given file format. + * The procedures that read snapshot ancestry and remove orphan files. Ancestry runs on a two-snapshot table and + * orphan removal on a seeded table, each in the given file format. */ def surfaceSnapshotProcedureCases(format: String): List[Plan.Case] = List( @@ -344,8 +339,8 @@ trait SurfaceScenarios extends ScenarioKit { surfaceProcRemoveOrphanRealCase(format)) /** - * Selecting the hidden metadata columns _file, _pos, _spec_id and _partition returns one row per - * seed row, each with a populated file path and a non-negative position. + * Selecting the hidden metadata columns _file, _pos, _spec_id and _partition returns one row per seed row, each with + * a populated file path and a non-negative position. */ private def surfaceMetaHiddenColumnsCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.meta.hiddenColumns") { @@ -369,9 +364,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * Every Iceberg metadata table (entries, files, manifests, snapshots, history, refs, partitions, - * and their all_* variants) is queryable without error, and the snapshots metadata table reports - * the table's 2 snapshots. + * Every Iceberg metadata table (entries, files, manifests, snapshots, history, refs, partitions, and their all_* + * variants) is queryable without error, and the snapshots metadata table reports the table's 2 snapshots. */ private def surfaceMetaTableSweepCase(format: String): Plan.Case = surfaceTwoSnapshotPreparation(format).test("surface.meta.tableSweep") { table => @@ -409,9 +403,8 @@ trait SurfaceScenarios extends ScenarioKit { surfaceMetaTableSweepCase(format)) /** - * Two threads concurrently insert 3 rows each; every insert either commits or fails with a typed - * commit-conflict exception, and the final row count matches 3 plus the number of inserts that - * actually committed. + * Two threads concurrently insert 3 rows each; every insert either commits or fails with a typed commit-conflict + * exception, and the final row count matches 3 plus the number of inserts that actually committed. */ private def surfaceConcAppendAppendCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.conc.appendAppend") { table => @@ -453,9 +446,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * Two threads concurrently UPDATE the same row to different values; the row count stays at 3, - * and the final value is one of the two competing updates or the original seed value, with any - * failure being a typed commit conflict. + * Two threads concurrently UPDATE the same row to different values; the row count stays at 3, and the final value is + * one of the two competing updates or the original seed value, with any failure being a typed commit conflict. */ private def surfaceConcUpdateUpdateCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.conc.updateUpdate") { table => @@ -496,18 +488,13 @@ trait SurfaceScenarios extends ScenarioKit { "concurrent updates should not change row count") } - /** - * Two writers racing on one table. Every outcome is either a commit or a typed commit conflict. - */ + /** Two writers racing on one table. Every outcome is either a commit or a typed commit conflict. */ def surfaceConcurrencyCases(format: String): List[Plan.Case] = List( surfaceConcAppendAppendCase(format), surfaceConcUpdateUpdateCase(format)) - /** - * On a side table, dropping NOT NULL from a column allows a subsequent insert of a null value - * for that column. - */ + /** On a side table, dropping NOT NULL from a column allows a subsequent insert of a null value for that column. */ private def surfaceSchemaRelaxNotNullCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.schema.relaxNotNull") { table => val sideTable = s"${table.name}_nn" @@ -532,8 +519,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * On a side table, widening a decimal column's precision preserves the original row and accepts - * a new row whose value only fits the wider precision. + * On a side table, widening a decimal column's precision preserves the original row and accepts a new row whose value + * only fits the wider precision. */ private def surfaceSchemaDecimalWidenCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.schema.decimalWiden") { table => @@ -563,8 +550,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * On a side table, ADD COLUMN of a new nested struct field null-fills it for the existing row - * and accepts a new row that sets the field. + * On a side table, ADD COLUMN of a new nested struct field null-fills it for the existing row and accepts a new row + * that sets the field. */ private def surfaceSchemaNestedAddFieldCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.schema.nestedAddField") { table => @@ -602,8 +589,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * On a side table, ALTER TABLE DROP COLUMN of a nested struct field is rejected with an - * exception, and the field remains readable afterward. + * On a side table, ALTER TABLE DROP COLUMN of a nested struct field is rejected with an exception, and the field + * remains readable afterward. */ private def surfaceSchemaNestedDropFieldCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.schema.nestedDropField") { table => @@ -631,10 +618,7 @@ trait SurfaceScenarios extends ScenarioKit { } } - /** - * ALTER TABLE ALTER COLUMN ... FIRST moves that column to the front of the schema while - * preserving all 3 rows. - */ + /** ALTER TABLE ALTER COLUMN ... FIRST moves that column to the front of the schema while preserving all 3 rows. */ private def surfaceSchemaReorderExistingCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.schema.reorderExisting") { table => table.spark.sql( @@ -665,8 +649,7 @@ trait SurfaceScenarios extends ScenarioKit { surfaceSchemaReorderExistingCase(format)) /** - * The write.distribution-mode=hash property requested at creation is retained and the table - * holds its 3 seed rows. + * The write.distribution-mode=hash property requested at creation is retained and the table holds its 3 seed rows. */ private def surfaceWriteDistributionHashCase(format: String): Plan.Case = surfaceHashPreparation(format).test("surface.write.distributionHash") { table => @@ -685,8 +668,8 @@ trait SurfaceScenarios extends ScenarioKit { } /** - * The write.target-file-size-bytes=1048576 property requested at creation is retained and the - * table holds its 3 seed rows. + * The write.target-file-size-bytes=1048576 property requested at creation is retained and the table holds its 3 seed + * rows. */ private def surfaceWriteTargetFileSizeCase(format: String): Plan.Case = surfaceTargetFileSizePreparation(format).test("surface.write.targetFileSize") { table => @@ -713,10 +696,9 @@ trait SurfaceScenarios extends ScenarioKit { surfaceWriteTargetFileSizeCase(format)) /** - * register_table onto a new name makes the source table's snapshot readable there (3 rows) and - * leaves the source unchanged, and dropping the registered table leaves the source unchanged. - * The system.snapshot and system.add_files procedures each reject their unsupported inputs with - * an exception. + * register_table onto a new name makes the source table's snapshot readable there (3 rows) and leaves the source + * unchanged, and dropping the registered table leaves the source unchanged. The system.snapshot and system.add_files + * procedures each reject their unsupported inputs with an exception. */ private def surfacePinImportProcsCase(format: String): Plan.Case = surfaceBasePreparation(format).test("surface.pin.importProcs") { table => diff --git a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala index d8bcf4abc..2e8761f90 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala @@ -4,9 +4,9 @@ import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} import org.junit.jupiter.api.Test /** - * Pins the shape the standard DML tests are written in: one list of test cases, one list of - * preparations, and a bucket that is the cross of the two. Each feature layer pins its own buckets - * in its own test. Reading these lists does not execute a case or start Spark. + * Pins the shape the standard DML tests are written in: one list of test cases, one list of preparations, and a bucket + * that is the cross of the two. Each feature layer pins its own buckets in its own test. Reading these lists does not + * execute a case or start Spark. */ final class DmlCaseCatalogTest { private val expectedReadTestCaseIds = List("read.projection", "read.filter") diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala index 0a6ed45d1..46b09b539 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala @@ -4,9 +4,9 @@ import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} import org.junit.jupiter.api.Test /** - * Pins how a preparation turns a test body into a catalog case: the ID it builds, the post-test - * hook every case from that preparation runs, and the known-bug reason a DML test case carries into - * its cases. Building a case runs no SQL, so these assertions need no Spark session. + * Pins how a preparation turns a test body into a catalog case: the ID it builds, the post-test hook every case from + * that preparation runs, and the known-bug reason a DML test case carries into its cases. Building a case runs no SQL, + * so these assertions need no Spark session. */ final class TablePreparationTest { private val emptyPreparation = TableTest(CoreTable) diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala index 30987963c..e77244ebe 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala @@ -11,9 +11,8 @@ import org.junit.jupiter.api.Assertions.{ import org.junit.jupiter.api.Test /** - * Pins fresh table identity and ownership cleanup: generated names stay namespace-scoped and - * unique across counter resets, cleanup starts after the ownership mark, and a cleanup failure is - * suppressed behind the primary test failure. + * Pins fresh table identity and ownership cleanup: generated names stay namespace-scoped and unique across counter + * resets, cleanup starts after the ownership mark, and a cleanup failure is suppressed behind the primary test failure. */ final class TableTestTest { @Test From 1f849dfe99f59d141ac0024e5e12f9dd098a6a3a Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Tue, 1 Sep 2026 17:09:45 -0700 Subject: [PATCH 12/24] refactor(delta-harness): index by capability Replace provenance and consequence buckets with capability-owned scenario files whose public contribution surfaces explain the catalog at a glance. Separate local runner code from the publishable harness, make preparations show creation and standard seeding explicitly, and reindex generic case IDs. Use generated table names and failure-preserving ownership boundaries for every case-owned table, view, registration, rename, and lock lifecycle. Move column-default coverage out of the standard layer for a dedicated follow-up PR while pinning the remaining 1,177-case catalog. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- integrations/spark/delta-harness/build.gradle | 22 +- .../openhouse/AccessControlScenarios.scala | 196 +++++ .../openhouse/ChangelogScenarios.scala | 219 +++++ .../openhouse/ColumnTagScenarios.scala | 43 + .../CompactionPlanningScenarios.scala | 137 ++++ .../openhouse/ConcurrencyScenarios.scala | 155 ++++ .../harness/openhouse/DataTypeScenarios.scala | 161 ++++ .../harness/openhouse/DmlScenarios.scala | 529 +++--------- .../openhouse/DmlValidationScenarios.scala | 131 +++ .../openhouse/EncryptionScenarios.scala | 44 + .../main/scala/harness/openhouse/Env.scala | 162 +--- .../openhouse/FileFormatScenarios.scala | 55 ++ .../openhouse/FileReplicationScenarios.scala | 82 ++ .../harness/openhouse/ForkScenarios.scala | 463 ----------- .../scala/harness/openhouse/Framework.scala | 124 ++- .../HazardReaderWriterScenarios.scala | 643 --------------- .../ImplementationPinScenarios.scala | 59 -- .../openhouse/IncrementalReadScenarios.scala | 93 +++ .../openhouse/InteractionScenarios.scala | 219 ----- .../scala/harness/openhouse/LocalRunner.scala | 143 ++++ .../harness/openhouse/LockingScenarios.scala | 114 +++ .../openhouse/MaintControlScenarios.scala | 240 ------ .../openhouse/MaintenanceScenarios.scala | 169 ++++ .../openhouse/MetadataTableScenarios.scala | 96 +++ .../openhouse/NamespaceScenarios.scala | 48 ++ .../openhouse/NegativeDdlScenarios.scala | 656 --------------- .../openhouse/NestedTypeScenarios.scala | 260 ++++++ .../openhouse/NestedTypesScenarios.scala | 505 ------------ .../harness/openhouse/OpenHouseMatrix.scala | 42 +- .../PartitionEvolutionScenarios.scala | 60 ++ .../PartitionTransformScenarios.scala | 155 ++++ .../main/scala/harness/openhouse/Plan.scala | 134 +-- .../openhouse/ProcedureScenarios.scala | 113 +++ .../harness/openhouse/RenameScenarios.scala | 68 ++ .../openhouse/ScanPlanningScenarios.scala | 115 +++ .../scala/harness/openhouse/ScenarioKit.scala | 310 ++++--- .../openhouse/SchemaEvolutionScenarios.scala | 324 ++++++++ .../openhouse/SnapshotRestoreScenarios.scala | 89 ++ .../openhouse/SortOrderScenarios.scala | 61 ++ .../openhouse/StreamingScenarios.scala | 213 +++++ .../harness/openhouse/SurfaceScenarios.scala | 767 ------------------ ...TableEvolutionCompatibilityScenarios.scala | 170 ++++ .../openhouse/TablePropertyScenarios.scala | 146 ++++ .../openhouse/TimeTravelScenarios.scala | 98 +++ .../WriteDistributionScenarios.scala | 161 ++++ .../WriterCompatibilityScenarios.scala | 50 ++ .../test/scala/harness/CaseCatalogTest.scala | 102 ++- .../scala/harness/DmlCaseCatalogTest.scala | 70 +- .../scala/harness/TableLifecycleTest.scala | 272 +++++++ .../scala/harness/TablePreparationTest.scala | 5 +- .../test/scala/harness/TableTestTest.scala | 2 + 51 files changed, 4890 insertions(+), 4405 deletions(-) create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/AccessControlScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ColumnTagScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/CompactionPlanningScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencyScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/DataTypeScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlValidationScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/EncryptionScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileFormatScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileReplicationScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/IncrementalReadScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/LocalRunner.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/LockingScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintenanceScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/MetadataTableScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/NamespaceScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypeScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionEvolutionScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionTransformScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ProcedureScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/RenameScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScanPlanningScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/SchemaEvolutionScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/SnapshotRestoreScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/SortOrderScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/StreamingScenarios.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/TableEvolutionCompatibilityScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/TablePropertyScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/TimeTravelScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriteDistributionScenarios.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriterCompatibilityScenarios.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/TableLifecycleTest.scala diff --git a/integrations/spark/delta-harness/build.gradle b/integrations/spark/delta-harness/build.gradle index 55cc11717..ba6609d4e 100644 --- a/integrations/spark/delta-harness/build.gradle +++ b/integrations/spark/delta-harness/build.gradle @@ -8,8 +8,9 @@ plugins { // it against the embedded catalog, and the LinkedIn acceptance tests supply a remote environment. // // Only the portable scenario/framework sources are published. Env.scala boots the embedded OpenHouse -// server and pulls in its test fixtures, so it is excluded from the published library and compiled in -// the local source set. +// server and pulls in its test fixtures, and LocalRunner.scala carries the harness.Main launch class +// that drives it, so both are excluded from the published library and compiled in the local source +// set. ext { icebergVersion = rootProject.ext.iceberg_1_5_version @@ -17,18 +18,21 @@ ext { scalaLibVersion = '2.12.18' } +// The embedded boot wiring and the local run loop. They are excluded from the portable library and +// are the whole of the local source set. +def embeddedOnlySources = ['harness/openhouse/Env.scala', 'harness/openhouse/LocalRunner.scala'] + sourceSets { main { scala { srcDirs = ['src/main/scala'] - // Embedded-only boot and run wiring is not part of the portable library. - exclude 'harness/openhouse/Env.scala' + exclude embeddedOnlySources } } local { scala { srcDirs = ['src/main/scala'] - include 'harness/openhouse/Env.scala' + include embeddedOnlySources } compileClasspath += sourceSets.main.output runtimeClasspath += sourceSets.main.output @@ -57,14 +61,18 @@ dependencies { testImplementation("com.linkedin.iceberg:iceberg-spark-runtime-3.5_2.12:${icebergVersion}") { exclude group: 'io.netty' } - testImplementation(project(path: ':integrations:spark:spark-3.5:openhouse-spark-3.5-runtime_2.12', configuration: 'shadow')) + testImplementation( + project(path: ':integrations:spark:spark-3.5:openhouse-spark-3.5-runtime_2.12', + configuration: 'shadow')) localImplementation sourceSets.main.output localImplementation "org.scala-lang:scala-library:${scalaLibVersion}" localImplementation("org.apache.spark:spark-sql_2.12:${sparkVersion}") { exclude group: 'io.netty' } - localImplementation(project(path: ':integrations:spark:spark-3.5:openhouse-spark-3.5-runtime_2.12', configuration: 'shadow')) { + localImplementation( + project(path: ':integrations:spark:spark-3.5:openhouse-spark-3.5-runtime_2.12', + configuration: 'shadow')) { exclude group: 'org.apache.commons', module: 'commons-lang3' } localImplementation project(':tables-test-fixtures:tables-test-fixtures-iceberg-1.5_2.12') diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/AccessControlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/AccessControlScenarios.scala new file mode 100644 index 000000000..12ab6965d --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/AccessControlScenarios.scala @@ -0,0 +1,196 @@ +package harness + +import org.apache.iceberg.exceptions.BadRequestException + +/** + * Access control: the SET POLICY statements that govern how a table may be shared, retained and replicated, and the + * GRANT and REVOKE statements that decide who may read it. + * + * Operations: SET POLICY (SHARING), SET POLICY (HISTORY), SET POLICY (REPLICATION) followed by UNSET POLICY + * (REPLICATION), SET POLICY (RETENTION) on the date column, the out-of-range SET POLICY (HISTORY MAX_AGE) and SET + * POLICY (HISTORY VERSIONS) forms, GRANT SELECT on an unshared table, and GRANT then REVOKE SELECT on a shared table + * with SHOW GRANTS in between. + * + * Preparation axes: the standard seeded core table in each of the two columnar formats, except the retention family, + * which starts from a date-partitioned core table seeded with the standard rows because RETENTION names a partition + * column. + * + * Case families: eight families contributing 16 cases. + */ +trait AccessControlScenarios extends ScenarioKit { + + /** Every access-control case, one file format at a time. */ + lazy val accessControlCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + policySharingCase(preparedStandardTable(format)), + policyHistoryCase(preparedStandardTable(format)), + policyReplicationCase(preparedStandardTable(format)), + policyRetentionCase(format), + policyHistoryMaxAgeRejectedCase(preparedStandardTable(format)), + policyHistoryVersionsRejectedCase(preparedStandardTable(format)), + grantUnsharedRejectedCase(preparedStandardTable(format)), + grantAndRevokeCase(preparedStandardTable(format))) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** SET POLICY (SHARING=TRUE) records the sharing policy and the table remains queryable. */ + private def policySharingCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("accessControl.policy.sharing") { table => + table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") + + val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + + assert( + policies.toLowerCase.contains("true") || policies.toLowerCase.contains("sharing"), + s"sharing policy not stored: $policies") + assert( + table.rows.size == standardSeedRowCount, + "table not queryable after SET POLICY (SHARING)") + } + + /** SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20) records the history policy and the table remains queryable. */ + private def policyHistoryCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("accessControl.policy.history") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20)") + + val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + + assert( + policies.contains("20") || policies.toLowerCase.contains("history"), + s"history policy not stored: $policies") + assert( + table.rows.size == standardSeedRowCount, + "table not queryable after SET POLICY (HISTORY)") + } + + /** + * SET POLICY (REPLICATION) followed by UNSET POLICY (REPLICATION) leaves the table queryable with its 3 rows intact. + */ + private def policyReplicationCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("accessControl.policy.replication") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") + table.spark.sql( + s"ALTER TABLE ${table.name} UNSET POLICY (REPLICATION)") + + assert(table.rows.size == standardSeedRowCount) + } + + /** + * SET POLICY (RETENTION = 30d ON COLUMN foo_col_date ...) records the retention policy and the table remains + * queryable. + */ + private def policyRetentionCase(format: String): Plan.Case = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"PARTITIONED BY (${Core.date0.columnName}) " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(standardSeedRowCount)()) + .test("accessControl.policy.retention") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (" + + s"RETENTION = 30d ON COLUMN ${Core.date0.columnName} WHERE pattern = 'yyyy-MM-dd-HH')") + + val policies = tableProps(table.spark, table.name).getOrElse("policies", "") + + assert( + policies.toLowerCase.contains("retention") || policies.contains("30"), + s"retention policy not stored: $policies") + assert( + table.rows.size == standardSeedRowCount, + "table not queryable after SET POLICY (RETENTION)") + } + + /** + * SET POLICY (HISTORY MAX_AGE=5D) exceeds the allowed range and is rejected with a BadRequestException stating the + * 1-to-3-day limit. + */ + private def policyHistoryMaxAgeRejectedCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("accessControl.policy.history.maxAge.rejected") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=5D)")) + + assert( + exception.getMessage.contains("max age must be between 1 to 3 days"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + + /** + * SET POLICY (HISTORY VERSIONS=200) exceeds the allowed range and is rejected with a BadRequestException stating the + * 2-to-100-version limit. + */ + private def policyHistoryVersionsRejectedCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("accessControl.policy.history.versions.rejected") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (HISTORY VERSIONS=200)")) + + assert( + exception.getMessage.contains("must be between 2 to 100 versions"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + + /** + * GRANT SELECT on a table that is not marked shared is rejected with an IllegalArgumentException stating the table + * is not shared. + */ + private def grantUnsharedRejectedCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("accessControl.grantUnshared.rejected") { table => + val exception = Check.intercept[IllegalArgumentException]( + table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC")) + + assert( + exception.getMessage.contains("is not a shared table"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + + /** + * On a shared table, GRANT SELECT TO PUBLIC makes SHOW GRANTS list SELECT for PUBLIC and the table stays queryable; + * REVOKE SELECT then removes that grant from SHOW GRANTS. + */ + private def grantAndRevokeCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation + .test("accessControl.grantAndRevoke") { table => + table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") + table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC") + + val grantsAfterGrant = table.spark + .sql(s"SHOW GRANTS ON TABLE ${table.name}") + .collect() + .map(row => (row.getString(0), row.getString(1))) + .toSet + assert( + grantsAfterGrant.contains(("SELECT", "PUBLIC")), + s"SHOW GRANTS did not include SELECT for PUBLIC: $grantsAfterGrant") + assert( + table.rows.size == standardSeedRowCount, + "the shared and granted table should stay queryable") + + table.spark.sql(s"REVOKE SELECT ON TABLE ${table.name} FROM PUBLIC") + val grantsAfterRevoke = table.spark + .sql(s"SHOW GRANTS ON TABLE ${table.name}") + .collect() + .map(row => (row.getString(0), row.getString(1))) + .toSet + assert( + !grantsAfterRevoke.contains(("SELECT", "PUBLIC")), + s"SHOW GRANTS retained SELECT for PUBLIC: $grantsAfterRevoke") + } + .copy(embeddedSkipReason = Some( + "The embedded test server has no OPA endpoint configured, so grantRole and " + + "listAclPolicies are no-ops that always report an empty ACL list. GRANT and REVOKE " + + "succeed without error, while SHOW GRANTS always returns an empty ACL list. The " + + "li-openhouse acceptance environment runs the assertions against its configured " + + "authorization service.")) + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogScenarios.scala new file mode 100644 index 000000000..4a445ed38 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogScenarios.scala @@ -0,0 +1,219 @@ +package harness + +/** + * One changelog operation: the name its case carries, the statement it runs against the prepared table, and the + * change-type histogram the changelog view reports for the snapshot range that statement opened. + */ +final case class ChangelogOperation( + name: String, + statement: String => String, + expectedChangeCounts: Map[String, Long] +) + +/** + * Changelog: the row-level change feed `create_changelog_view` reports for a snapshot range, and what it reports once + * the start of that range has been expired. + * + * Operations: five reusable changelog operations (an append, an INSERT OVERWRITE that drops one row, a row-level + * DELETE, an UPDATE, and a MERGE that updates one row and inserts another), each followed by a changelog view opened + * at the seed snapshot; a changelog view over an append-only history with no start snapshot; and a changelog view + * opened at three start points inside an expired snapshot range. + * + * Preparation axes: in each of the two columnar formats, the standard seeded core table for the five operations and + * for the expired-range family, and the two-snapshot core table for the append-only history family. The operations are + * data, so a feature layer covers its own table mode by crossing `changelogOperations` with its own preparations. + * + * Case families: three families contributing 14 cases, 10 operation cases, 2 append-only history cases and 2 + * expired-range cases. + */ +trait ChangelogScenarios extends ScenarioKit { + + /** Every changelog case, one file format at a time. */ + lazy val changelogCases: List[Plan.Case] = + standardFormats.flatMap { format => + changelogOperationCasesFor(List(preparedStandardTable(format))) ++ + List( + appendOnlyHistoryCase(preparedTwoSnapshotTable(format)), + expiredRangeCase(preparedStandardTable(format))) + } + + /** + * The five row-level operations whose change feed the catalog reports. Every one starts from the standard three-row + * seed, so its expected histogram holds on any preparation that seeds those rows. + */ + lazy val changelogOperations: List[ChangelogOperation] = + List( + ChangelogOperation( + "changelog.append", + table => + s"INSERT INTO $table VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')", + Map("INSERT" -> 1L)), + ChangelogOperation( + "changelog.overwrite", + table => + s"INSERT OVERWRITE $table SELECT * FROM $table " + + s"WHERE ${Core.long0.columnName} <= 2", + Map("DELETE" -> 1L)), + ChangelogOperation( + "changelog.delete", + table => s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1", + Map("DELETE" -> 1L)), + ChangelogOperation( + "changelog.update", + table => + s"UPDATE $table SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2", + Map("DELETE" -> 1L, "INSERT" -> 1L)), + ChangelogOperation( + "changelog.merge", + table => + s"MERGE INTO $table target " + + "USING (SELECT CAST(2 AS BIGINT) key " + + "UNION ALL SELECT CAST(9 AS BIGINT)) source " + + s"ON target.${Core.long0.columnName} = source.key " + + s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + + "WHEN NOT MATCHED THEN INSERT " + + s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + + s"${Core.string0.columnName}, ${Core.double0.columnName}, " + + s"${Core.boolean0.columnName}, ${Core.date0.columnName}) " + + "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')", + Map("DELETE" -> 1L, "INSERT" -> 2L))) + + /** The changelog cases for every operation on every preparation given, one preparation at a time. */ + def changelogOperationCasesFor( + preparations: List[TablePreparation[CoreTable.type]] + ): List[Plan.Case] = + preparations.flatMap(preparation => + changelogOperations.map(operation => changelogOperationCase(preparation, operation))) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** The change-type histogram the named changelog view reports. */ + private def changeCounts(table: PreparedTable[CoreTable.type], view: String): Map[String, Long] = + table.spark + .sql(s"SELECT _change_type, count(*) FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + /** + * Running the operation against a seeded table and opening a changelog view at the seed snapshot reports exactly the + * change types and counts that operation is defined to produce. + */ + private def changelogOperationCase( + preparation: TablePreparation[CoreTable.type], + operation: ChangelogOperation): Plan.Case = + preparation.test(operation.name) { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql(operation.statement(table.name)) + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$seedSnapshotId'))") + .collect()(0) + .getString(0) + + val actualChangeCounts = changeCounts(table, view) + + assert( + actualChangeCounts == operation.expectedChangeCounts, + s"${operation.name} reported $actualChangeCounts, expected ${operation.expectedChangeCounts}") + } + + /** create_changelog_view over an append-only history reports 5 changes, all of change type INSERT. */ + private def appendOnlyHistoryCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("changelog.appendOnlyHistory") { table => + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}')") + .collect()(0) + .getString(0) + val actualChangeCounts = changeCounts(table, view) + + assert( + actualChangeCounts == Map("INSERT" -> 5L), + s"append-only changelog should report five inserts: $actualChangeCounts") + } + + /** + * After expire_snapshots removes a changelog start point, create_changelog_view over that start point either throws + * or reports fewer changes than the table's history holds, and any message it throws leaves expiration unnamed. The + * case covers three start points: an expired snapshot ID, a timestamp older than the whole history, and a timestamp + * inside the expired range. + */ + private def expiredRangeCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("changelog.expiredRange") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + val snapshots = snapshotIds(table.spark, table.name) + val firstTimestamp = table.spark + .sql( + s"SELECT committed_at FROM ${table.name}.snapshots " + + "ORDER BY committed_at LIMIT 1") + .collect()(0) + .getTimestamp(0) + val middleTimestamp = table.spark + .sql( + s"SELECT committed_at FROM ${table.name}.snapshots " + + s"WHERE snapshot_id = ${snapshots(1)}") + .collect()(0) + .getTimestamp(0) + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + def changelogOutcome( + optionKey: String, + optionValue: String, + trueChangeCount: Long): String = + try { + val view = table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('$optionKey', '$optionValue'))") + .collect()(0) + .getString(0) + val actualChangeCount = table.spark + .sql(s"SELECT count(*) FROM $view") + .collect()(0) + .getLong(0) + if (actualChangeCount < trueChangeCount) { + s"SILENT under-report: $actualChangeCount of $trueChangeCount true changes" + } else { + s"FULL: $actualChangeCount of $trueChangeCount" + } + } catch { + case exception: Throwable => + s"TYPED: ${exception.getClass.getSimpleName} :: " + + Option(exception.getMessage).getOrElse("").take(140) + } + + val outcomes = List( + "explicitExpiredId" -> changelogOutcome("start-snapshot-id", snapshots.head.toString, 5), + "timestampBeforeHistory" -> + changelogOutcome("start-timestamp", (firstTimestamp.getTime - 1000).toString, 5), + "timestampInsideExpiredRange" -> + changelogOutcome("start-timestamp", (middleTimestamp.getTime - 1).toString, 2)) + + outcomes.foreach { case (startPoint, outcome) => + println(s"DIAG changelog.expiredRange $startPoint: $outcome") + assert( + !outcome.startsWith("FULL"), + s"expired-lineage changelog returned full truth for $startPoint") + assert( + !outcome.toLowerCase.contains("expir"), + s"expired-lineage message now names expiration for $startPoint") + } + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ColumnTagScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ColumnTagScenarios.scala new file mode 100644 index 000000000..ed72d1288 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ColumnTagScenarios.scala @@ -0,0 +1,43 @@ +package harness + +/** + * Column tags: ALTER TABLE MODIFY COLUMN SET TAG records a classification on a column and leaves the values that + * column returns exactly as they were written. + * + * Operations: SET TAG = (PII) on the string column, followed by a read of that column. + * + * Preparation axes: the standard seeded core table in each of the two columnar formats. + * + * Case families: one family contributing 2 cases. + */ +trait ColumnTagScenarios extends ScenarioKit { + + /** The column-tag case, one file format at a time. */ + lazy val columnTagCases: List[Plan.Case] = + standardFormats.map(format => setTagCase(preparedStandardTable(format))) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * ALTER TABLE MODIFY COLUMN SET TAG = (PII) tags a column, and queries keep returning the values the seed wrote. + */ + private def setTagCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("columnTag.setTag") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} MODIFY COLUMN " + + s"${Core.string0.columnName} SET TAG = (PII)") + + val values = table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getString(0)) + + assert( + values == Seq("row-1", "row-2", "row-3"), + s"SET TAG changed the values the column returns: $values") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/CompactionPlanningScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/CompactionPlanningScenarios.scala new file mode 100644 index 000000000..bf4001709 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/CompactionPlanningScenarios.scala @@ -0,0 +1,137 @@ +package harness + +import org.apache.spark.sql.SparkSession + +/** + * Compaction planning: rewrite_data_files packs data files into rewrite groups weighted by file length and spends a + * budget in file-sequence-number order, and the rewrite it commits preserves every row. + * + * Operations: rewrite_data_files with rewrite-all over a table whose data files are unevenly sized, and + * rewrite_data_files with rewrite-all over a table whose live data-file entries carry distinct, increasing + * file_sequence_numbers. + * + * Preparation axes: one table per family, built inside the case with write.distribution-mode=none so each insert + * commits its own data file. The bin-packing family runs in each of the two columnar formats. Sequence numbers order + * commits the same way in every file format, so the ordering family runs on Parquet alone. + * + * Case families: two families contributing 3 cases. + */ +trait CompactionPlanningScenarios extends ScenarioKit { + + /** The bin-packing case in each columnar format, then the file-sequence ordering case on Parquet. */ + lazy val compactionPlanningCases: List[Plan.Case] = + standardFormats.map(format => + Plan.Case( + s"compactionPlanning.binPackByFileLength @ $format", + binPackByFileLengthCase(format))) ++ + List( + Plan.Case("compactionPlanning.fileSequenceOrder @ parquet", fileSequenceOrderCase)) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** The count and the total byte size of the table's current data files. */ + private def dataFileStats(spark: SparkSession, table: String): (Long, Long) = { + val stats = spark + .sql(s"SELECT count(*), coalesce(sum(file_size_in_bytes), 0) FROM $table.data_files") + .collect()(0) + (stats.getLong(0), stats.getLong(1)) + } + + private def rewriteAll(spark: SparkSession, table: String): Unit = + spark.sql( + s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', " + + "options => map('rewrite-all', 'true'))") + + /** + * Compacting a table whose data files are unevenly sized preserves the row count and every row's value, which is the + * observable result of packing rewrite groups by file length; the weighting itself is a planner decision that no SQL + * surface exposes. + */ + private def binPackByFileLengthCase(format: String)(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = TableTest.nextQualifiedTableName(ctx.namespace) + + withOwnedTable(spark.sql(_), table)( + spark.sql( + s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'write.distribution-mode'='none')")) { + // Unevenly sized data files: a tiny one, a small one, and a big one. + spark.sql(s"INSERT INTO $table VALUES (1,'a')") + spark.sql(s"INSERT INTO $table VALUES (2,'b'),(3,'c')") + spark.sql(s"INSERT INTO $table SELECT id, repeat('x', 200) FROM range(100, 400)") + + val (filesBefore, bytesBefore) = dataFileStats(spark, table) + assert(filesBefore >= 3, s"[$format] expected at least 3 uneven data files, got $filesBefore") + val rowsBefore = countOf(spark, s"SELECT count(*) FROM $table") + + rewriteAll(spark, table) + + val (filesAfter, bytesAfter) = dataFileStats(spark, table) + assert( + countOf(spark, s"SELECT count(*) FROM $table") == rowsBefore, + s"[$format] rewrite_data_files changed the row count from $rowsBefore") + val smallestRowValue = + spark.sql(s"SELECT s FROM $table WHERE id = 1").collect()(0).getString(0) + assert(smallestRowValue == "a", s"[$format] rewrite altered a row: id=1 s=$smallestRowValue") + + println( + s"DIAG compactionPlanning.binPackByFileLength[$format]: filesBefore=$filesBefore " + + s"bytesBefore=$bytesBefore filesAfter=$filesAfter bytesAfter=$bytesAfter rows=$rowsBefore") + } + } + + /** + * file_sequence_number is exposed on the live data-file entries of the entries metadata table and increases + * monotonically across commits, and rewrite_data_files with rewrite-all preserves the row count and the row set. A + * budgeted rewrite spends its budget in file-sequence-number order, so that column is the observable half of the + * ordering decision. + */ + private def fileSequenceOrderCase(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = TableTest.nextQualifiedTableName(ctx.namespace) + + withOwnedTable(spark.sql(_), table)( + spark.sql( + s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES (" + + "'write.format.default'='parquet', 'write.distribution-mode'='none')")) { + // Several commits produce several data files with distinct, increasing file-sequence-numbers. + val numberOfCommits = 4 + (0 until numberOfCommits).foreach { commitIndex => + spark.sql(s"INSERT INTO $table VALUES (${commitIndex}L, 'c$commitIndex')") + } + + val sequenceNumbers = spark + .sql( + s"SELECT file_sequence_number FROM $table.entries " + + "WHERE status != 2 AND data_file.content = 0 ORDER BY file_sequence_number") + .collect() + .toSeq + .map(_.getLong(0)) + assert( + sequenceNumbers.size >= numberOfCommits, + s"expected at least $numberOfCommits live data-file entries with sequence numbers, " + + s"got ${sequenceNumbers.size}: $sequenceNumbers") + assert( + sequenceNumbers == sequenceNumbers.sorted, + s"file sequence numbers not monotonic: $sequenceNumbers") + assert( + sequenceNumbers.distinct.size >= 2, + s"expected multiple distinct file sequence numbers, got ${sequenceNumbers.distinct}") + val rowsBefore = countOf(spark, s"SELECT count(*) FROM $table") + + rewriteAll(spark, table) + + assert( + countOf(spark, s"SELECT count(*) FROM $table") == rowsBefore, + s"rewrite changed the row count from $rowsBefore") + val keys = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) + assert(keys == (0 until numberOfCommits).map(_.toLong), s"rewrite altered the row set: $keys") + + println( + s"DIAG compactionPlanning.fileSequenceOrder: fileSequenceNumbers=" + + s"${sequenceNumbers.mkString(",")} filesAfter=" + + s"${dataFileStats(spark, table)._1} rows=$rowsBefore") + } + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencyScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencyScenarios.scala new file mode 100644 index 000000000..71fad133a --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencyScenarios.scala @@ -0,0 +1,155 @@ +package harness + +import java.util.concurrent.{ConcurrentLinkedQueue, CountDownLatch, TimeUnit} +import java.util.concurrent.atomic.AtomicInteger + +/** + * Concurrency: two writers racing on one table. Every write either commits or fails with a typed commit-conflict + * exception, and the table the race leaves behind is consistent with the writes that committed. + * + * Operations: two threads each running three single-row INSERTs against the same table, and two threads each running + * an UPDATE of the same row to a different value. + * + * Preparation axes: the standard seeded core table in each of the two columnar formats. The concurrency helpers are + * feature neutral, so a feature layer reuses them for its own table mode. + * + * Case families: two families contributing 4 cases. + */ +trait ConcurrencyScenarios extends ScenarioKit { + + /** Every concurrency case, one file format at a time. */ + lazy val concurrencyCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + appendAppendCase(preparedStandardTable(format)), + updateUpdateCase(preparedStandardTable(format))) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * Runs every function on its own daemon thread, releases them together, and waits up to three minutes for all of + * them. Returns the throwables the threads raised, plus one for each thread still running at the deadline. + */ + protected def runConcurrently(functions: Seq[() => Unit]): Seq[Throwable] = { + val errors = new ConcurrentLinkedQueue[Throwable]() + val start = new CountDownLatch(1) + val threads = functions.zipWithIndex.map { case (function, index) => + val thread = new Thread( + () => + try { + start.await() + function() + } catch { + case interrupted: InterruptedException => + Thread.currentThread().interrupt() + errors.add(interrupted) + case throwable: Throwable => + errors.add(throwable) + }, + s"delta-harness-concurrent-$index") + thread.setDaemon(true) + thread + } + threads.foreach(_.start()) + start.countDown() + + val deadline = System.nanoTime() + TimeUnit.MINUTES.toNanos(3) + threads.foreach { thread => + val remainingNanos = deadline - System.nanoTime() + if (remainingNanos > 0) { + TimeUnit.NANOSECONDS.timedJoin(thread, remainingNanos) + } + } + + threads.filter(_.isAlive).foreach { thread => + errors.add( + new AssertionError(s"${thread.getName} did not complete within 3 minutes")) + thread.interrupt() + } + errors.toArray(Array.empty[Throwable]).toSeq + } + + /** A commit conflict the catalog reports through one of its typed commit, validation or transport exceptions. */ + protected def isTypedCommitConflict(throwable: Throwable): Boolean = + Exceptions.causeChain(throwable).exists { cause => + val className = cause.getClass.getName + className.contains("CommitFailed") || + className.contains("CommitStateUnknown") || + className.contains("Validation") || + className.contains("BadRequest") || + className.contains("WebClientResponse") + } + + /** + * Two threads concurrently insert 3 rows each; every insert either commits or fails with a typed commit-conflict + * exception, and the final row count matches 3 plus the number of inserts that actually committed. + */ + private def appendAppendCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("concurrency.appendAppend") { table => + val failureCount = new AtomicInteger(0) + def writer(base: Int): () => Unit = () => + (0 until 3).foreach { offset => + val value = base + offset + try { + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + s"(CAST($value AS BIGINT), $value, 'row-c', 1.5, true, '2024-01-09-01')") + } catch { + case exception: Throwable => + assert( + isTypedCommitConflict(exception), + "concurrent append failed with an untyped error: " + + s"${exception.getClass.getName}") + failureCount.incrementAndGet() + } + } + val threadErrors = runConcurrently(Seq(writer(100), writer(200))) + val expectedRowCount = 3 + 6 - failureCount.get + + assert( + threadErrors.isEmpty, + s"writer thread failed outside the insert loop: $threadErrors") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == expectedRowCount.toString, + s"expected $expectedRowCount rows after ${failureCount.get} of 6 inserts hit a conflict") + } + + /** + * Two threads concurrently UPDATE the same row to different values; the row count stays at 3, and the final value is + * one of the two competing updates or the original seed value, with any failure being a typed commit conflict. + */ + private def updateUpdateCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("concurrency.updateUpdate") { table => + def updater(value: String): () => Unit = () => + try { + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = '$value' " + + s"WHERE ${Core.long0.columnName} = 2") + } catch { + case exception: Throwable => + assert( + isTypedCommitConflict(exception), + "concurrent update failed with an untyped error: " + + s"${exception.getClass.getName}") + } + val threadErrors = runConcurrently(Seq(updater("AAA"), updater("BBB"))) + val finalValue = table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 2") + .collect()(0) + .getString(0) + + assert( + threadErrors.isEmpty, + s"updater thread failed with a non-conflict error: $threadErrors") + assert( + finalValue == "AAA" || finalValue == "BBB" || finalValue == "row-2", + s"concurrent updates produced a torn value: $finalValue") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", + "concurrent updates should leave the row count at 3") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DataTypeScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DataTypeScenarios.scala new file mode 100644 index 000000000..15d281710 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DataTypeScenarios.scala @@ -0,0 +1,161 @@ +package harness + +import java.math.BigDecimal + +/** + * Scalar data types: how the long, int, double, decimal, string, binary, date, timestamp and timestamp_ntz columns + * round trip, and how the catalog stores the edge values of each one. + * + * Operations: a round trip of the seeded long, int, double, decimal and string values; an INSERT of an all-null row; + * an INSERT of the special double values NaN and Infinity; an INSERT at the long, int and decimal boundaries; and an + * INSERT of a unicode string and an empty string. + * + * Preparation axes: one unpartitioned TypesTable layout per file format, each seeded with three rows covering every + * scalar column. + * + * Case families: five families over three layouts, contributing 15 cases. + */ +trait DataTypeScenarios extends ScenarioKit { + + /** Every scalar-type case, one layout at a time. */ + lazy val dataTypeCases: List[Plan.Case] = + preparedTypesTables.flatMap(preparation => + List( + roundtripCase(preparation), + nullsCase(preparation), + specialFloatsCase(preparation), + boundariesCase(preparation), + unicodeAndEmptyCase(preparation))) + + /** One unpartitioned scalar-type table per file format. */ + lazy val typesLayouts: List[Layout] = + fileFormats.map(format => + Layout( + s"types-unpartitioned/$format", + table => + s"CREATE TABLE $table (${TypesTable.columnDefinitions}) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")) + + /** One preparation per scalar-type layout: the table is created, then seeded with three fully valued rows. */ + lazy val preparedTypesTables: List[TablePreparation[TypesTable.type]] = + typesLayouts.map(layout => + TablePreparation( + layout.label, + TableTest(TypesTable).sql("create")(layout.create)().insert(standardSeedRowCount)())) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + // A fully valued TypesTable row with the given id; each case supplies the columns it is about. + private def typesRow(id: Long, n: String, x: String, dec: String, str: String): String = + s"(CAST($id AS BIGINT), $n, $x, $dec, $str, CAST('b' AS binary), DATE '2024-01-01', " + + s"TIMESTAMP '2024-01-01 00:00:00', TIMESTAMP_NTZ '2024-01-01 00:00:00')" + + /** + * Selecting id, n, x, dec and str for the first seeded row reads back the exact long, int, double, decimal and + * string values that were seeded. + */ + private def roundtripCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + preparation.test("types.roundtrip") { table => + val row = table.spark + .sql( + s"SELECT id, n, x, dec, str FROM ${table.name} WHERE id = 1") + .collect()(0) + + assert( + row.getLong(0) == 1L && + row.getInt(1) == 1 && + row.getDouble(2) == 1.5) + assert( + row.getDecimal(3).compareTo(new BigDecimal("1.50")) == 0) + assert(row.getString(4) == "row-1") + } + + /** + * Inserting a row with every non-key column NULL reads back as null for the int, double, string, timestamp and + * timestamp_ntz columns. + */ + private def nullsCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + preparation.test("types.nulls") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES (" + + "CAST(10 AS BIGINT), NULL, NULL, NULL, NULL, " + + "NULL, NULL, NULL, NULL)") + + val row = table.spark + .sql( + s"SELECT n, x, str, ts, tsntz FROM ${table.name} WHERE id = 10") + .collect()(0) + + assert((0 to 4).forall(row.isNullAt)) + } + + /** Inserting rows with double('NaN') and double('Infinity') reads back as NaN and positive infinity respectively. */ + private def specialFloatsCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + preparation.test("types.specialFloats") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + s"${typesRow(11, "0", "double('NaN')", "CAST(0 AS decimal(10,2))", "'x'")}, " + + s"${typesRow(12, "0", "double('Infinity')", "CAST(0 AS decimal(10,2))", "'y'")}") + + assert( + table.spark + .sql(s"SELECT x FROM ${table.name} WHERE id = 11") + .collect()(0) + .getDouble(0) + .isNaN) + assert( + table.spark + .sql(s"SELECT x FROM ${table.name} WHERE id = 12") + .collect()(0) + .getDouble(0) + .isInfinite) + } + + /** + * Inserting a row at Long.MaxValue, Int.MaxValue and a max-precision decimal reads those boundary values back + * unchanged. + */ + private def boundariesCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + preparation.test("types.boundaries") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + typesRow( + Long.MaxValue, + Int.MaxValue.toString, + "0.0", + "CAST(99999999.99 AS decimal(10,2))", + "'max'")) + + val row = table.spark + .sql( + s"SELECT id, n, dec FROM ${table.name} WHERE str = 'max'") + .collect()(0) + + assert( + row.getLong(0) == Long.MaxValue && + row.getInt(1) == Int.MaxValue) + assert( + row.getDecimal(2).compareTo(new BigDecimal("99999999.99")) == 0) + } + + /** Inserting rows with a unicode string and an empty string reads each back unchanged. */ + private def unicodeAndEmptyCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + preparation.test("types.unicodeAndEmpty") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + s"${typesRow(13, "0", "0.0", "CAST(0 AS decimal(10,2))", "'\u65e5\u672c\u8a9e \uD83C\uDF89'")}, " + + s"${typesRow(14, "0", "0.0", "CAST(0 AS decimal(10,2))", "''")}") + + assert( + table.spark + .sql(s"SELECT str FROM ${table.name} WHERE id = 13") + .collect()(0) + .getString(0) == "\u65e5\u672c\u8a9e \uD83C\uDF89") + assert( + table.spark + .sql(s"SELECT str FROM ${table.name} WHERE id = 14") + .collect()(0) + .getString(0) == "") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala index 8251d9f38..08ad5884a 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala @@ -4,24 +4,117 @@ import org.apache.spark.sql.Row import org.apache.spark.sql.functions.lit /** - * Defines 54 reusable DML operations over the six CoreTable columns: bigint, int, string, double, boolean, and a - * string-encoded date. The operation catalog contains 2 reads, 14 deletes, 13 updates, 16 merges, 6 inserts or - * overwrites, 1 null-string delete, and 2 partition-scoped overwrites. Each operation covers a distinct SQL or - * DataFrame form or a distinct observable state change within its family. + * Data manipulation on the core table: the reads, deletes, updates, merges, inserts and overwrites the catalog + * supports, and the row and snapshot change each one commits. * - * ScenarioKit supplies the starting-state axes. Six core layouts cross three file formats with partitioned and - * unpartitioned tables. Three date-partitioned layouts receive partition-scoped writes. Six write-ordered layouts - * exercise the same catalog under sort order. Six evolved layouts receive the 29 operations that address columns by - * name. Null-string variants isolate the one operation that requires a null value. + * Operations: 54 reusable DML operations over the six CoreTable columns (bigint, int, string, double, boolean, and a + * string-encoded date), made up of 2 reads, 14 deletes, 13 updates, 16 merges, 6 inserts or overwrites, 1 null-string + * delete, and 2 partition-scoped overwrites. Each operation covers a distinct SQL or DataFrame form, or a distinct + * observable state change within its family. Every operation is defined once here, so a feature layer covers its own + * table mode by crossing these same definitions with its own preparations. * - * The final section defines 13 bespoke DDL follow-up cases. Six consume a table after a DDL state transition, and seven - * verify schema creation or evolution. These cases stay outside the DML cross-product because the DDL transition is - * part of the behavior under test. + * Preparation axes: ScenarioKit supplies the starting states. Six core layouts cross three file formats with + * partitioned and unpartitioned tables. Three date-partitioned layouts receive the partition-scoped writes. Six + * write-ordered layouts exercise the same catalog under a sort order. Six evolved layouts receive the 29 operations + * that address columns by name. Null-string variants isolate the one operation that requires a null value. + * + * Case families: 804 cases in four families, `coreDmlCases` (312), `partitionedDmlCases` (6), `orderedDmlCases` (312) + * and `evolvedDmlCases` (174). */ trait DmlScenarios extends ScenarioKit { import Rows._ - // --- the DML test cases --- + /** Every DML case, in preparation order: core, partition-scoped, write-ordered, then evolved. */ + lazy val dmlCases: List[Plan.Case] = + coreDmlCases ++ partitionedDmlCases ++ orderedDmlCases ++ evolvedDmlCases + + /** + * The reads. They select columns by name and write nothing, so they run on any preparation that starts from the three + * seed rows, including one whose column list has grown past that shape. + */ + lazy val readTestCases: List[DmlTestCase[CoreTable.type]] = List( + readProjection, + readFilter) + + /** + * The DELETE that selects a null string. It applies to a preparation that already holds a row whose string column is + * null, and it removes exactly that row. + */ + lazy val nullStringRowTestCases: List[DmlTestCase[CoreTable.type]] = List( + deleteByNullCondition) + + /** + * The partition-scoped writes. They replace whole partitions, so they apply to a preparation that partitions the + * table, and they cross with the partitioned preparations alone. + */ + lazy val partitionedTableTestCases: List[DmlTestCase[CoreTable.type]] = List( + insertDynamicOverwrite, + overwritePartitions) + + // --- which cases a preparation is compatible with --- + // Compatibility is a property of the starting state, so each list names the states it fits. + + /** Every DML case. Runs on any preparation that starts from three seed rows of the seed shape. */ + lazy val allDmlTestCases: List[DmlTestCase[CoreTable.type]] = + readTestCases ++ + deleteTestCases ++ + updateTestCases ++ + mergeTestCases ++ + insertAndOverwriteTestCases + + /** The row-mutating cases: every DELETE, UPDATE and MERGE. */ + lazy val rowMutationTestCases: List[DmlTestCase[CoreTable.type]] = + deleteTestCases ++ updateTestCases ++ mergeTestCases + + /** + * The cases that address columns by name and never write a whole seed-shaped row, so they run on a preparation whose + * column list has grown beyond the seed rows. + */ + lazy val testCasesCompatibleWithAnAddedColumn: List[DmlTestCase[CoreTable.type]] = + readTestCases ++ deleteTestCases ++ updateTestCases + + /** + * Every DML case, with the partition-predicate DELETE marked as a known bug: the Spark and Iceberg rewrite crashes on + * it when the table carries a write order. + */ + lazy val orderedDmlTestCases: List[DmlTestCase[CoreTable.type]] = + allDmlTestCases.map { + case testCase if testCase == deleteByPartitionPredicate => + testCase.copy(knownBugReason = Some( + "DELETE by partition predicate crashes in the Spark and Iceberg rewrite when the " + + "table has a write order.")) + case testCase => + testCase + } + + // --- standard preparations crossed with the cases they are compatible with --- + + /** + * Every DML case on the core preparations, plus the null-string DELETE on the same preparations extended with a + * null-string row. + */ + lazy val coreDmlCases: List[Plan.Case] = + preparedCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) ++ + preparedNullStringCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) + + /** The partition-scoped writes on the partitioned preparations. */ + lazy val partitionedDmlCases: List[Plan.Case] = + preparedPartitionedCoreTables.flatMap(preparation => + partitionedTableTestCases.map(_.runOn(preparation))) + + /** Every DML case on the write-ordered preparations, plus the null-string DELETE on their null-string form. */ + lazy val orderedDmlCases: List[Plan.Case] = + preparedOrderedCoreTables.flatMap(preparation => orderedDmlTestCases.map(_.runOn(preparation))) ++ + preparedNullStringOrderedCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) + + /** The cases that address columns by name, on the preparations that added a column. */ + lazy val evolvedDmlCases: List[Plan.Case] = + preparedEvolvedCoreTables.flatMap(preparation => + testCasesCompatibleWithAnAddedColumn.map(_.runOn(preparation))) + + // --- the operations the surface above composes --- // Each case captures the table state, runs one operation, captures the state again, and asserts the row change and // the snapshot delta that operation caused. Deltas are relative, so a case holds on any preparation regardless of how // many snapshots the preparation itself committed. @@ -74,14 +167,6 @@ trait DmlScenarios extends ScenarioKit { assert(after == before, "a read leaves the rows and the snapshot count unchanged") }) - /** - * The reads. They select columns by name and write nothing, so they run on any preparation that starts from the three - * seed rows, including one whose column list has grown past that shape. - */ - val readTestCases: List[DmlTestCase[CoreTable.type]] = List( - readProjection, - readFilter) - /** * DELETE WHERE foo_col_string IS NULL removes exactly the prepared row whose string is null, leaves every other row * unchanged, and commits one snapshot. @@ -104,13 +189,6 @@ trait DmlScenarios extends ScenarioKit { "DELETE by a null condition commits one snapshot") }) - /** - * The DELETE that selects a null string. It applies to a preparation that already holds a row whose string column is - * null, and it removes exactly that row. - */ - val nullStringRowTestCases: List[DmlTestCase[CoreTable.type]] = List( - deleteByNullCondition) - /** * DELETE WHERE foo_col_date = '2024-01-01-00' removes the rows with that date, keeps the rest, and commits one * snapshot. @@ -775,7 +853,7 @@ trait DmlScenarios extends ScenarioKit { SELECT * FROM VALUES (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') - AS s($cols) + AS s($columnNameList) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED THEN INSERT *""") val after = table.state @@ -859,7 +937,7 @@ trait DmlScenarios extends ScenarioKit { SELECT * FROM VALUES (CAST(2 AS BIGINT), 2, 'U', 2.5, true, '2024-01-02-01'), (CAST(7 AS BIGINT), 7, 'g', 7.5, false, '2024-01-07-06') - AS s($cols) + AS s($columnNameList) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName} @@ -979,7 +1057,7 @@ trait DmlScenarios extends ScenarioKit { SELECT * FROM VALUES (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') - AS s($cols) + AS s($columnNameList) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED AND s.${Core.long0.columnName} = 4 THEN INSERT *""") val after = table.state @@ -1007,7 +1085,7 @@ trait DmlScenarios extends ScenarioKit { SELECT * FROM VALUES (CAST(2 AS BIGINT), 2, 'M2', 2.5, true, '2024-01-02-01'), (CAST(4 AS BIGINT), 4, 'row-4', 4.5, false, '2024-01-04-03') - AS s($cols) + AS s($columnNameList) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN MATCHED THEN UPDATE SET t.${Core.string0.columnName} = s.${Core.string0.columnName} @@ -1041,7 +1119,7 @@ trait DmlScenarios extends ScenarioKit { s"""MERGE INTO ${table.name} t USING ( SELECT * FROM VALUES (CAST(2 AS BIGINT), 22, 'S2', 22.5, true, '2024-06-06-06') - AS s($cols) + AS s($columnNameList) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN MATCHED THEN UPDATE SET *""") val after = table.state @@ -1162,7 +1240,7 @@ trait DmlScenarios extends ScenarioKit { SELECT * FROM VALUES (CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04') - AS s($cols) + AS s($columnNameList) ) s ON t.${Core.long0.columnName} = s.${Core.long0.columnName} WHEN NOT MATCHED THEN INSERT *""") val after = table.state @@ -1327,7 +1405,7 @@ trait DmlScenarios extends ScenarioKit { table.spark.sql( s"INSERT INTO ${table.name} SELECT * FROM VALUES " + s"(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') " + - s"AS s($cols)") + s"AS s($columnNameList)") val after = table.state assert( @@ -1352,7 +1430,7 @@ trait DmlScenarios extends ScenarioKit { .sql( s"SELECT * FROM VALUES " + s"(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05') " + - s"AS s($cols)") + s"AS s($columnNameList)") .writeTo(table.name) .append() val after = table.state @@ -1405,7 +1483,7 @@ trait DmlScenarios extends ScenarioKit { .sql( s"SELECT * FROM VALUES " + s"(CAST(8 AS BIGINT), 8, 'h', 8.5, false, '2024-01-08-07') " + - s"AS s($cols)") + s"AS s($columnNameList)") .writeTo(table.name) .overwrite(lit(true)) val after = table.state @@ -1474,7 +1552,7 @@ trait DmlScenarios extends ScenarioKit { .sql( s"SELECT * FROM VALUES " + "(CAST(10 AS BIGINT), 10, 'p', 10.5, true, '2024-01-01-00') " + - s"AS s($cols)") + s"AS s($columnNameList)") .writeTo(table.name) .overwritePartitions() val after = table.state @@ -1489,381 +1567,4 @@ trait DmlScenarios extends ScenarioKit { "a partition overwrite commits one snapshot") }) - /** - * The partition-scoped writes. They replace whole partitions, so they apply to a preparation that partitions the - * table, and they cross with the partitioned preparations alone. - */ - val partitionedTableTestCases: List[DmlTestCase[CoreTable.type]] = List( - insertDynamicOverwrite, - overwritePartitions) - - // --- which cases a preparation is compatible with --- - // Compatibility is a property of the starting state, so each list names the states it fits. - - /** Every DML case. Runs on any preparation that starts from three seed rows of the seed shape. */ - val allDmlTestCases: List[DmlTestCase[CoreTable.type]] = - readTestCases ++ - deleteTestCases ++ - updateTestCases ++ - mergeTestCases ++ - insertAndOverwriteTestCases - - /** The row-mutating cases: every DELETE, UPDATE and MERGE. */ - val rowMutationTestCases: List[DmlTestCase[CoreTable.type]] = - deleteTestCases ++ updateTestCases ++ mergeTestCases - - /** - * The cases that address columns by name and never write a whole seed-shaped row, so they run on a preparation whose - * column list has grown beyond the seed rows. - */ - val testCasesCompatibleWithAnAddedColumn: List[DmlTestCase[CoreTable.type]] = - readTestCases ++ deleteTestCases ++ updateTestCases - - /** - * Every DML case, with the partition-predicate DELETE marked as a known bug: the Spark and Iceberg rewrite crashes on - * it when the table carries a write order. - */ - val orderedDmlTestCases: List[DmlTestCase[CoreTable.type]] = - allDmlTestCases.map { - case testCase if testCase == deleteByPartitionPredicate => - testCase.copy(knownBugReason = Some( - "DELETE by partition predicate crashes in the Spark and Iceberg rewrite when the " + - "table has a write order.")) - case testCase => - testCase - } - - // --- standard preparations crossed with the cases they are compatible with --- - - /** - * Every DML case on the core preparations, plus the null-string DELETE on the same preparations extended with a - * null-string row. - */ - val coreDmlCases: List[Plan.Case] = - preparedCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) ++ - preparedNullStringCoreTables.flatMap(preparation => - nullStringRowTestCases.map(_.runOn(preparation))) - - /** The partition-scoped writes on the partitioned preparations. */ - val partitionedDmlCases: List[Plan.Case] = - preparedPartitionedCoreTables.flatMap(preparation => - partitionedTableTestCases.map(_.runOn(preparation))) - - /** Every DML case on the write-ordered preparations, plus the null-string DELETE on their null-string form. */ - val orderedDmlCases: List[Plan.Case] = - preparedOrderedCoreTables.flatMap(preparation => orderedDmlTestCases.map(_.runOn(preparation))) ++ - preparedNullStringOrderedCoreTables.flatMap(preparation => - nullStringRowTestCases.map(_.runOn(preparation))) - - /** The cases that address columns by name, on the preparations that added a column. */ - val evolvedDmlCases: List[Plan.Case] = - preparedEvolvedCoreTables.flatMap(preparation => - testCasesCompatibleWithAnAddedColumn.map(_.runOn(preparation))) - - // --- DDL consumers: a DDL evolves the table, then operations are run against it --- - - /** - * One preparation per Parquet and ORC layout and per DDL: three seed rows with keys 1, 2 and 3, then one of ADD - * COLUMN cc int, which the seed rows read as null; foo_col_int widened from int to bigint; WRITE ORDERED BY - * foo_col_long, which gives the table that write sort order; or write.distribution-mode set to range, which range - * distributes later writes. Plan walks this list so every consumer family lands on one preparation before the next - * preparation starts. - */ - val ddlConsumerPreparations: List[TablePreparation[CoreTable.type]] = - parquetAndOrcLayouts.flatMap { layout => - List( - TablePreparation( - layout.label, - createAndSeed(layout, 3) - .sql("ddl")(table => s"ALTER TABLE $table ADD COLUMN cc int")(), - "ddlConsume:addColumn."), - TablePreparation( - layout.label, - createAndSeed(layout, 3) - .sql("ddl")(table => - s"ALTER TABLE $table ALTER COLUMN ${Core.int0.columnName} TYPE bigint")(), - "ddlConsume:typeWiden."), - TablePreparation( - layout.label, - createAndSeed(layout, 3) - .sql("ddl")(table => - s"ALTER TABLE $table WRITE ORDERED BY ${Core.long0.columnName}")(), - "ddlConsume:writeOrder."), - TablePreparation( - layout.label, - createAndSeed(layout, 3) - .sql("ddl")(table => - s"ALTER TABLE $table SET TBLPROPERTIES " + - "('write.distribution-mode'='range')")(), - "ddlConsume:distMode.")) - } - - /** A plain INSERT still lands on the table after the DDL, taking it to four rows. */ - private def dmlWriteCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("dmlWrite") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "table is not writable after DDL") - } - - /** A row-level DELETE still lands on the table after the DDL, taking it to two rows. */ - private def dmlMutateCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("dmlMutate") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 2, - "mutation failed after DDL") - } - - /** The seed snapshot from before the DDL is still readable through VERSION AS OF and returns its three rows. */ - private def timeTravelCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("timeTravel") { table => - val seedSnapshotId = - snapshotIds(table.spark, table.name).head - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF $seedSnapshotId") - .collect()(0) - .getLong(0) == 3, - "seed snapshot is not readable after DDL") - } - - /** - * rollback_to_snapshot back to the seed snapshot undoes an INSERT made after the DDL and returns the table to its - * three seed rows. - */ - private def restoreCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("restore") { table => - val seedSnapshotId = - snapshotIds(table.spark, table.name).head - - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $seedSnapshotId)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 3, - "restore across DDL failed") - } - - /** expire_snapshots retaining only the newest snapshot leaves the table readable with its four current rows. */ - private def expireCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("expire") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "table is unreadable after snapshot expiration") - } - - /** The reads and writes a consumer runs against the table this preparation evolved. */ - def ddlConsumerDataCases( - preparation: TablePreparation[CoreTable.type]): List[Plan.Case] = - List( - dmlWriteCase(preparation), - dmlMutateCase(preparation), - timeTravelCase(preparation), - restoreCase(preparation), - expireCase(preparation)) - - /** rewrite_data_files compacts the files written across the DDL and preserves the four current rows. */ - private def compactCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("compact") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('min-input-files', '2'))") - - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "compaction changed rows after DDL") - } - - /** The compaction a consumer runs over the files written across this preparation's DDL. */ - def ddlConsumerCompactionCases( - preparation: TablePreparation[CoreTable.type]): List[Plan.Case] = - List( - compactCase(preparation)) - - // --- DDL that changes the schema of a seeded table --- - - /** - * The created table's schema is exactly CoreTable's columns, in declaration order and with their declared types, and - * the table holds no rows. - */ - private def createSchemaCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("create.schema") { table => - val actual = table.spark - .table(table.name) - .schema - .fields - .toList - .map(field => field.name -> field.dataType.simpleString) - val expected = Core.tableColumns.toList.map(column => (column.columnName, column.sqlType)) - - assert(actual == expected, s"schema is $actual") - assert(table.rows.isEmpty, "a table that was never seeded holds no rows") - } - - /** The created-schema case on every unseeded preparation. */ - val createSchemaCases: List[Plan.Case] = preparedEmptyCoreTables.map { preparation => - createSchemaCase(preparation) - } - - /** ADD COLUMN adds the column to the schema, the existing rows read null for it, and the row count is unchanged. */ - private def ddlAddColumnSingleCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.addColumn.single") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_int int") - - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - val nullCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name} WHERE added_int IS NULL") - .collect()(0) - .getLong(0) - - assert(columnNames.contains("added_int"), s"added_int missing: $columnNames") - assert( - nullCount == table.preparedRows.size, - s"existing rows should read null for added_int: $nullCount != ${table.preparedRows.size}") - assert(table.rows.size == table.preparedRows.size, "ADD COLUMN changed the row count") - } - - /** ADD COLUMNS with two columns in one statement adds both to the schema and leaves the row count unchanged. */ - private def ddlAddColumnMultipleCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.addColumn.multiple") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMNS (added_a int, added_b string)") - - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - - assert( - columnNames.contains("added_a") && columnNames.contains("added_b"), - s"added columns missing: $columnNames") - assert(table.rows.size == table.preparedRows.size, "ADD COLUMNS changed the row count") - } - - /** ADD COLUMN ... COMMENT stores the comment on the added column and the reader sees it. */ - private def ddlAddColumnCommentCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.addColumn.comment") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_c int COMMENT 'a note'") - - val addedColumn = table.spark - .table(table.name) - .schema - .fields - .find(_.name == "added_c") - .getOrElse(throw new AssertionError("added_c missing")) - - assert( - addedColumn.getComment().contains("a note"), - s"comment not stored: ${addedColumn.getComment()}") - } - - /** ADD COLUMN ... AFTER foo_col_long places the added column directly after that column in the schema. */ - private def ddlAddColumnPositionCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.addColumn.position") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN added_after int AFTER ${Core.long0.columnName}") - - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - - assert( - columnNames.indexOf("added_after") == columnNames.indexOf(Core.long0.columnName) + 1, - s"added_after not after long0: $columnNames") - } - - /** - * ALTER COLUMN foo_col_int TYPE bigint widens the column in the schema and the already-written values read back - * unchanged. - */ - private def ddlAlterColumnTypeWidenCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.alterColumn.typeWiden") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.int0.columnName} TYPE bigint") - - val liveColumns = table.spark.table(table.name).schema.fields.toSeq - .map(field => field.name -> field.dataType.simpleString) - .toMap - val values = table.spark - .sql( - s"SELECT ${Core.int0.columnName} FROM ${table.name} ORDER BY ${Core.long0.columnName}") - .collect() - .toSeq - .map(_.getLong(0)) - - assert( - liveColumns.get(Core.int0.columnName).contains("bigint"), - s"int0 not widened: ${liveColumns.get(Core.int0.columnName)}") - assert(values == Seq(1L, 2L, 3L), s"values not preserved after widening: $values") - } - - /** - * RENAME COLUMN renames the column in the schema: the new name is present, the old name is gone, and the row count is - * unchanged. - */ - private def ddlRenameColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation - .test("ddl.renameColumn") { table => - table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN to_rename int") - table.spark.sql(s"ALTER TABLE ${table.name} RENAME COLUMN to_rename TO renamed_col") - - val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) - - assert( - columnNames.contains("renamed_col") && !columnNames.contains("to_rename"), - s"RENAME COLUMN silently no-oped: $columnNames") - assert(table.rows.size == table.preparedRows.size, "RENAME COLUMN changed the row count") - } - .copy(knownBugReason = Some( - "RENAME COLUMN is a silent no-op because server-side schema casing normalization " + - "restores the old name.")) - - /** The schema-changing DDL cases on the core preparations. */ - val ddlSchemaCases: List[Plan.Case] = preparedCoreTables.flatMap { preparation => - List( - ddlAddColumnSingleCase(preparation), - ddlAddColumnMultipleCase(preparation), - ddlAddColumnCommentCase(preparation), - ddlAddColumnPositionCase(preparation), - ddlAlterColumnTypeWidenCase(preparation), - ddlRenameColumnCase(preparation)) - } - } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlValidationScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlValidationScenarios.scala new file mode 100644 index 000000000..fd9b3001b --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlValidationScenarios.scala @@ -0,0 +1,131 @@ +package harness + +import org.apache.spark.sql.AnalysisException + +/** + * DML validation: the DML statements the analyzer and the row-level rewrite refuse, and the message each rejection + * carries. + * + * Operations: DELETE on a column the table does not declare, DELETE and UPDATE with a nondeterministic predicate, + * INSERT INTO with fewer values than the table has columns, a MERGE whose UPDATE SET assigns one target column twice, + * and a MERGE whose source matches one target row twice. + * + * Preparation axes: the standard seeded core table in each of the two columnar formats. + * + * Case families: six families contributing 12 cases. + */ +trait DmlValidationScenarios extends ScenarioKit { + + /** Every DML-validation case, one file format at a time. */ + lazy val dmlValidationCases: List[Plan.Case] = + preparedCoreFormats.flatMap { preparation => + List( + nonExistentColumnCase(preparation), + nonDeterministicDeleteCase(preparation), + nonDeterministicUpdateCase(preparation), + insertArityCase(preparation), + mergeConflictingUpdatesCase(preparation), + mergeCardinalityViolationCase(preparation)) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** DELETE with a WHERE clause on a nonexistent column is rejected with an AnalysisException naming that column. */ + private def nonExistentColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("dmlValidation.nonExistentColumn") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"DELETE FROM ${table.name} WHERE no_such_column = 1")) + + assert(exception.getMessage.contains("no_such_column")) + } + + /** + * DELETE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an AnalysisException about + * determinism. + */ + private def nonDeterministicDeleteCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("dmlValidation.nonDeterministicDelete") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"DELETE FROM ${table.name} WHERE rand() < 0.5")) + + assert(exception.getMessage.toLowerCase.contains("deterministic")) + } + + /** + * UPDATE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an AnalysisException about + * determinism. + */ + private def nonDeterministicUpdateCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("dmlValidation.nonDeterministicUpdate") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'x' WHERE rand() < 0.5")) + + assert(exception.getMessage.toLowerCase.contains("deterministic")) + } + + /** + * INSERT INTO with too few values for the table's columns is rejected with an AnalysisException about the missing + * data columns. + */ + private def insertArityCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("dmlValidation.insertArity") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"INSERT INTO ${table.name} VALUES (CAST(1 AS BIGINT), 1)")) + + assert(exception.getMessage.toLowerCase.contains("not enough data columns")) + } + + /** + * A MERGE whose UPDATE SET assigns the same target column twice is rejected with an AnalysisException about multiple + * assignments. + */ + private def mergeConflictingUpdatesCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("dmlValidation.mergeConflictingUpdates") { table => + val keyColumn = Core.long0.columnName + val stringColumn = Core.string0.columnName + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"""MERGE INTO ${table.name} target USING ( + SELECT * FROM VALUES (CAST(2 AS BIGINT)) AS source($keyColumn) + ) source + ON target.$keyColumn = source.$keyColumn + WHEN MATCHED THEN UPDATE + SET target.$stringColumn = 'a', target.$stringColumn = 'b'""")) + + assert(exception.getMessage.contains("Multiple assignments")) + } + + /** + * A MERGE whose source has two rows matching the same target row fails with a cardinality-violation error naming the + * multi-row match. + */ + private def mergeCardinalityViolationCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("dmlValidation.mergeCardinalityViolation") { table => + val keyColumn = Core.long0.columnName + val stringColumn = Core.string0.columnName + val exception = Check.intercept[Exception]( + table.spark.sql( + s"""MERGE INTO ${table.name} target USING ( + SELECT * FROM VALUES + (CAST(2 AS BIGINT), 'a'), + (CAST(2 AS BIGINT), 'b') + AS source($keyColumn, $stringColumn) + ) source + ON target.$keyColumn = source.$keyColumn + WHEN MATCHED THEN UPDATE SET target.$stringColumn = source.$stringColumn""")) + + assert( + Exceptions.causeChain(exception).exists { cause => + Option(cause.getMessage).exists( + _.contains("matched a single row from the target table")) + }, + s"expected a MERGE cardinality-violation message, got: ${exception.getMessage}") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/EncryptionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/EncryptionScenarios.scala new file mode 100644 index 000000000..6bdb56d8b --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/EncryptionScenarios.scala @@ -0,0 +1,44 @@ +package harness + +import java.nio.file.{Files, Paths} + +/** + * Encryption: the OSS build writes table data in plaintext, because OpenHouse delegates table-data encryption to an + * external KMS plugin and the OSS build wires no KeyManagementClient into the catalog, leaving the default + * PlaintextEncryptionManager in place. + * + * Operations: read the trailing footer magic bytes of one data file. A Parquet footer reads PAR1 for plaintext and + * PARE under modular encryption regardless of compression, so that magic value settles which path wrote the file. + * + * Preparation axes: the standard seeded core table in Parquet, which is the format whose footer carries the marker. + * + * Case families: one family contributing 1 case. + */ +trait EncryptionScenarios extends ScenarioKit { + + /** The plaintext data-file case, on the standard seeded Parquet table. */ + lazy val encryptionCases: List[Plan.Case] = + List(dataFilePlaintextCase(preparedStandardTable("parquet"))) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** A data file's Parquet footer magic bytes are the plaintext PAR1 marker. */ + private def dataFilePlaintextCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("encryption.dataFilePlaintext") { table => + val dataFilePath = table.spark + .sql(s"SELECT file_path FROM ${table.name}.data_files LIMIT 1") + .collect()(0) + .getString(0) + .stripPrefix("file:") + val bytes = Files.readAllBytes(Paths.get(dataFilePath)) + + assert( + bytes.length >= 8, + s"data file is too small to inspect: ${bytes.length} bytes") + val footerMagic = new String(bytes.takeRight(4), "US-ASCII") + assert( + footerMagic == "PAR1", + s"expected plaintext Parquet footer PAR1, got $footerMagic") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala index 47aa1dd80..d75f81de0 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Env.scala @@ -1,50 +1,31 @@ package harness -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -/** Runs a case, retrying only a transient-infrastructure failure. */ -object Runner { - val MaxAttempts = 3 - - def execute(testCase: Plan.Case, context: Ctx): (Outcome, Int) = { - @tailrec def attempt(attemptIndex: Int): (Outcome, Int) = { - val outcome = - try { - testCase.run(context.copy(spark = context.spark.newSession())) - Outcome.Passed - } - catch { case NonFatal(throwable) => Outcome.Failed(throwable) } - outcome match { - case failure: Outcome.Failed - if failure.retryable && attemptIndex + 1 < MaxAttempts => - attempt(attemptIndex + 1) - case terminal => - (terminal, attemptIndex + 1) - } - } - attempt(0) - } -} - -/** Boots the embedded OpenHouse server and wires a SparkSession to the OpenHouse catalog. */ +import org.apache.spark.sql.SparkSession +import scala.io.Source + +/** + * Embedded environment wiring: boots the in-process OpenHouse server and hands back a SparkSession pointed at its + * catalog. This file is compiled into the `local` source set only, because it pulls in the server test fixtures that + * the published portable library leaves out. + */ object OpenHouseEnv { import com.linkedin.openhouse.tablestest.OpenHouseLocalServer private def authToken(): String = Option(getClass.getClassLoader.getResourceAsStream("dummy.token")) - .map(is => scala.io.Source.fromInputStream(is, "UTF-8").mkString.trim) + .map(tokenStream => Source.fromInputStream(tokenStream, "UTF-8").mkString.trim) .getOrElse("default-token") - private def wireCatalog(builder: SparkSession.Builder, name: String, uri: String, token: String): SparkSession.Builder = + private def wireCatalog( + builder: SparkSession.Builder, + name: String, + uri: String, + token: String): SparkSession.Builder = builder .config(s"spark.sql.catalog.$name", "org.apache.iceberg.spark.SparkCatalog") - .config(s"spark.sql.catalog.$name.catalog-impl", "com.linkedin.openhouse.spark.OpenHouseCatalog") + .config( + s"spark.sql.catalog.$name.catalog-impl", + "com.linkedin.openhouse.spark.OpenHouseCatalog") .config(s"spark.sql.catalog.$name.uri", uri) .config(s"spark.sql.catalog.$name.cluster", "local-cluster") .config(s"spark.sql.catalog.$name.auth-token", token) @@ -89,112 +70,3 @@ object OpenHouseEnv { } } } - -object Main { - def main(args: Array[String]): Unit = { - val (server, spark, restUri, restToken) = OpenHouseEnv.start() - var runFailure: Option[Throwable] = None - try { - spark.sparkContext.setLogLevel("ERROR") - val ctx = Ctx(spark, "openhouse.dbMatrix", restUri, restToken) - - // Each command-line argument is an include substring. A case runs when its ID contains every provided substring. - // An empty argument list selects the full catalog. - val filters = args.toList - val cases = Plan.cases.filter(testCase => - filters.forall(testCase.id.contains)) - - val header = - if (filters.isEmpty) { - "all cases" - } else { - s"filter ${filters.mkString(", ")} -> ${cases.size} cases" - } - println(s"\n=== delta-harness :: localized cases @ OpenHouse catalog ($header) ===\n") - - // Each case owns a fresh table. Worker tasks use separate Spark sessions over the shared Spark context, and - // results are printed in catalog order. - val parallelism = sys.env.get("HARNESS_PARALLELISM").map(_.toInt) - .getOrElse(math.max(1, Runtime.getRuntime.availableProcessors())) - println(s"parallelism: $parallelism worker sessions\n") - - def runOne(testCase: Plan.Case): (Plan.Case, (Outcome, Int)) = - testCase.embeddedSkipReason - .map(reason => s"embedded limitation: $reason") - .orElse(Plan.bugReason(testCase)) match { - case Some(reason) => - (testCase, (Outcome.Skipped(reason): Outcome, 0)) - case None => - (testCase, Runner.execute(testCase, ctx)) - } - - val results = - if (parallelism <= 1) { - cases.map(runOne) - } else { - val pool = java.util.concurrent.Executors.newFixedThreadPool(parallelism) - try { - val futures = cases.map(testCase => - pool.submit( - new java.util.concurrent.Callable[(Plan.Case, (Outcome, Int))] { - def call(): (Plan.Case, (Outcome, Int)) = runOne(testCase) - })) - futures.map(_.get(60, java.util.concurrent.TimeUnit.MINUTES)) - } finally { - pool.shutdownNow() - } - } - - results.foreach { case (testCase, (outcome, attempts)) => - val note = outcome match { - case failure: Outcome.Failed => - s" (${failure.reason}${if (failure.retryable) " [retryable]" else ""})" - case Outcome.Skipped(reason) => - s" ($reason)" - case Outcome.Passed => - "" - } - println(f"${outcome.label}%-4s ${testCase.id}%-52s try=$attempts$note") - } - - val failed = - results.count { case (_, (outcome, _)) => outcome.isInstanceOf[Outcome.Failed] } - val skipped = - results.count { case (_, (outcome, _)) => outcome.isInstanceOf[Outcome.Skipped] } - val passed = results.size - failed - skipped - println(f"\n$passed passed, $skipped skipped, $failed failed (${results.size} cases)") - - if (failed > 0 || passed == 0) { - throw new AssertionError( - s"delta harness finished with $passed passed, $skipped skipped, and $failed failed cases") - } - } catch { - case failure: Throwable => - runFailure = Some(failure) - throw failure - } finally { - val cleanupFailures = - List[() => Unit]( - () => spark.stop(), - () => server.stop()) - .flatMap { cleanup => - try { - cleanup() - None - } catch { - case failure: Throwable => Some(failure) - } - } - - runFailure match { - case Some(failure) => - cleanupFailures.foreach(failure.addSuppressed) - case None => - cleanupFailures.headOption.foreach { failure => - cleanupFailures.drop(1).foreach(failure.addSuppressed) - throw failure - } - } - } - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileFormatScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileFormatScenarios.scala new file mode 100644 index 000000000..20ff523e7 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileFormatScenarios.scala @@ -0,0 +1,55 @@ +package harness + +/** + * File format materialization: the write.format.default a table declares is the format its data files are actually + * written in. + * + * Operations: read the declared write.format.default from the table properties, then list the data files the + * preparation wrote and compare their extensions against it. + * + * Preparation axes: the twelve standard preparations that leave data files behind, which are the six core layouts + * (Parquet, ORC and Avro crossed with unpartitioned and date-partitioned) and the same six carrying a write sort + * order. A feature layer covers its own table mode by passing its own preparations to `layoutFormatCasesFor`. + * + * Case families: one family, `format.materialization`, contributing 12 cases. + */ +trait FileFormatScenarios extends ScenarioKit { + + /** The format-materialization case on every standard preparation that writes data files. */ + lazy val fileFormatCases: List[Plan.Case] = layoutFormatCasesFor(layoutFormatPreparations) + + /** + * The format-materialization case for each preparation given: every data file the preparation wrote carries the + * extension of the table's declared write.format.default, and listing the files leaves the rows and the snapshot + * count unchanged. It applies to any preparation that leaves data files behind, so each feature layer passes the + * list its own preparations produce. + */ + def layoutFormatCasesFor( + preparations: List[TablePreparation[CoreTable.type]] + ): List[Plan.Case] = + preparations.map { preparation => + preparation.test("format.materialization") { table => + val before = table.state + val declaredFormat = table.spark + .sql(s"SHOW TBLPROPERTIES ${table.name} ('write.format.default')") + .collect()(0) + .getString(1) + val filePaths = table.spark + .sql(s"SELECT file_path FROM ${table.name}.files") + .collect() + .toSeq + .map(_.getString(0)) + val after = table.state + + assert( + filePaths.nonEmpty && filePaths.forall(_.toLowerCase.endsWith(s".$declaredFormat")), + s"data files are not all .$declaredFormat: $filePaths") + assert(after == before, "listing files leaves the rows and the snapshot count unchanged") + } + } + + /** The standard preparations that leave data files behind: the core and the write-ordered ones. */ + lazy val layoutFormatPreparations: List[TablePreparation[CoreTable.type]] = + preparedCoreTables ++ preparedOrderedCoreTables + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileReplicationScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileReplicationScenarios.scala new file mode 100644 index 000000000..cc6e2a711 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileReplicationScenarios.scala @@ -0,0 +1,82 @@ +package harness + +import org.apache.iceberg.Table +import org.apache.iceberg.spark.Spark3Util +import java.util.{Map => JavaMap} +import scala.util.Try + +/** + * File replication: the output-file property the writer stamps so the file system can set a block replication factor + * on the files a commit produces. + * + * Operations: read OutputFileFactory.FILE_REPLICATION_FACTOR, build an OutputFileFactory carrying a replication + * factor, read the property map that factory stamps onto its output files, and write to the table afterwards. + * + * Preparation axes: one format-version-2 table built inside the case, because the case needs an Iceberg Table handle + * to build a factory from. + * + * Case families: one family contributing 1 case. + */ +trait FileReplicationScenarios extends ScenarioKit { + + /** The output-file replication property case. */ + lazy val fileReplicationCases: List[Plan.Case] = + List( + Plan.Case("fileReplication.outputFileProperty @ core", outputFilePropertyCase)) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * OutputFileFactory exposes FILE_REPLICATION_FACTOR as "file-replication-factor", and a factory built with a + * replication factor stamps that key into the property map of the output files it creates. Writes made through the + * table afterwards still return the correct rows. The key is the one HDFS reads to set block replication on an + * output file when a replication factor is supplied to the factory, and the delete-file write path is the one path + * that supplies one. Reflection reaches the builder and getProperties because some Iceberg artifacts leave them out + * of the public compiled API, so a direct reference would fail to compile against those artifacts. + */ + private def outputFilePropertyCase(ctx: Ctx): Unit = { + val spark = ctx.spark + val outputFileFactoryClass = Class.forName("org.apache.iceberg.io.OutputFileFactory") + + val replicationKeyField = Try(outputFileFactoryClass.getField("FILE_REPLICATION_FACTOR")) + assert(replicationKeyField.isSuccess, "OutputFileFactory.FILE_REPLICATION_FACTOR is absent") + val replicationKey = replicationKeyField.get.get(null).asInstanceOf[String] + assert( + replicationKey == "file-replication-factor", + s"""expected FILE_REPLICATION_FACTOR to equal "file-replication-factor", got "$replicationKey"""") + + val table = TableTest.nextQualifiedTableName(ctx.namespace) + withOwnedTable(spark.sql(_), table)( + spark.sql( + s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + + "TBLPROPERTIES ('format-version'='2')")) { + spark.sql(s"INSERT INTO $table VALUES (1,'a'),(2,'b')") + val icebergTable = Spark3Util.loadIcebergTable(spark, table) + val builder = outputFileFactoryClass + .getMethod("builderFor", classOf[Table], classOf[Int], classOf[Long]) + .invoke(null, icebergTable, Int.box(1), Long.box(1L)) + val replicationFactorMethod = + Try(builder.getClass.getMethod("replicationFactor", classOf[Short])) + assert( + replicationFactorMethod.isSuccess, + "OutputFileFactory.Builder.replicationFactor(short) is absent") + replicationFactorMethod.get.invoke(builder, Short.box(2.toShort)) + val factory = Option(builder.getClass.getMethod("build").invoke(builder)) + .getOrElse(throw new AssertionError("OutputFileFactory build returned null")) + + val getProperties = outputFileFactoryClass.getDeclaredMethod("getProperties") + getProperties.setAccessible(true) + val outputFileProperties = + getProperties.invoke(factory).asInstanceOf[JavaMap[String, String]] + assert( + outputFileProperties.get(replicationKey) == "2", + s"expected output-file property $replicationKey=2 stamped by the factory, " + + s"got ${outputFileProperties.get(replicationKey)}") + + spark.sql(s"INSERT INTO $table VALUES (3,'c')") + val keys = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) + assert(keys == Seq(1L, 2L, 3L), s"rows wrong after write: $keys") + } + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala deleted file mode 100644 index dd443536f..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ForkScenarios.scala +++ /dev/null @@ -1,463 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The fork cases pin behavior decided by LinkedIn's fork of Apache Iceberg, the com.linkedin.iceberg artifacts this -// module depends on: the column-default path, the write distribution default for a partitioned write, the output-file -// replication key, the read split size, and the compaction plan. These behaviors have no catalog SQL surface of their -// own, so a case reaches them through the Iceberg API or a Spark configuration and asserts the result a caller can -// observe. -trait ForkScenarios extends ScenarioKit { - import Rows._ - - /** - * ALTER TABLE ADD COLUMN c int DEFAULT 5 parses, and the default value stops at the parser: the committed schema - * records no default for c, pre-existing rows read null for it, and an INSERT that omits c is rejected with - * INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA. The file format is the parameter. - */ - private def forkColDefaultAddColumn(fmt: String)(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_coldef_$fmt" - spark.sql(s"DROP TABLE IF EXISTS $table") - spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')") - spark.sql(s"INSERT INTO $table VALUES (1, 'a'), (2, 'b')") - - // (1) The DDL is accepted at parse time; Spark owns the DEFAULT grammar. - spark.sql(s"ALTER TABLE $table ADD COLUMN c int DEFAULT 5") - - // (2) The default is not written into the persisted schema; column c has no default metadata. - val cDesc = spark.sql(s"DESCRIBE TABLE EXTENDED $table").collect() - .map(_.mkString("|")).filter(_.matches("(?i)^c\\|.*")).mkString(" ;; ") - assert(!cDesc.toLowerCase.contains("default") && !cDesc.contains("5"), - s"[$fmt] expected no default persisted for c, but DESCRIBE shows: $cDesc") - - // (3) The default is not backfilled on read; pre-existing rows read null, not 5. - val nulls = spark.sql(s"SELECT count(*) FROM $table WHERE c IS NULL").collect()(0).getLong(0) - assert(nulls == 2, - s"[$fmt] expected the default not applied on read (2 nulls), got $nulls") - - // (4) The default is not applied on write; an insert that omits c is rejected. - val omit = Check.intercept[org.apache.spark.sql.AnalysisException] { - spark.sql(s"INSERT INTO $table (id, s) VALUES (3, 'c')") - } - val omitMsg = Exceptions.causeChain(omit).flatMap(e => Option(e.getMessage)).mkString(" | ") - assert(omitMsg.contains("CANNOT_FIND_DATA"), - s"[$fmt] expected omit-insert rejected with CANNOT_FIND_DATA, got: $omitMsg") - - println(s"fork.colDefault[$fmt]: accepted=yes persistedDefault=no readBackfill=no writeApply=no(CANNOT_FIND_DATA)") - spark.sql(s"DROP TABLE IF EXISTS $table") - } - - /** - * A NestedField built with an initial default serializes initial-default into the schema JSON, and that value - * survives a fromJson then toJson round trip. SchemaParser.toJson takes no format-version parameter, so the key - * serializes the same at every format version. On an artifact whose NestedField exposes no builder, the - * column-default API is absent entirely, down to the initialDefault and writeDefault accessors, and the case pins - * that absence. Reflection reaches the builder because some Iceberg release jars leave it out, which a direct - * reference would fail to compile against. - */ - private def forkColDefaultApiSerialization(ctx: Ctx): Unit = { - val nestedFieldCls = Class.forName("org.apache.iceberg.types.Types$NestedField") - val builderM = scala.util.Try(nestedFieldCls.getMethod("builder")) - if (builderM.isFailure) { - // The column-default API is absent on this artifact; assert that absence is total. - println("fork.colDefault.api: NestedField.builder absent, column-default API unsupported on this artifact") - val ms = nestedFieldCls.getMethods.map(_.getName).toSet - assert(!ms.contains("initialDefault") && !ms.contains("writeDefault"), - "NestedField exposes initial/write-default accessors but no builder()") - return - } - // The column-default API is present; build `optional int c` carrying initial-default=5. - val builder0 = builderM.get.invoke(null) - def chain(b: AnyRef, m: String, argT: Class[_], arg: AnyRef): AnyRef = - b.getClass.getMethod(m, argT).invoke(b, arg) - def chain0(b: AnyRef, m: String): AnyRef = b.getClass.getMethod(m).invoke(b) - val intType = Class.forName("org.apache.iceberg.types.Types$IntegerType") - .getMethod("get").invoke(null) - var b = chain(builder0, "withId", java.lang.Integer.TYPE, java.lang.Integer.valueOf(3)) - b = chain(b, "withName", classOf[String], "c") - b = chain(b, "ofType", Class.forName("org.apache.iceberg.types.Type"), intType) - b = chain0(b, "asOptional") - b = chain(b, "withInitialDefault", classOf[Object], java.lang.Integer.valueOf(5)) - val field = b.getClass.getMethod("build").invoke(b) - .asInstanceOf[org.apache.iceberg.types.Types.NestedField] - - // Assemble a schema [id, c(default=5)] and serialize it; no format version is passed to toJson. - val idField = org.apache.iceberg.types.Types.NestedField.required( - 1, "id", org.apache.iceberg.types.Types.LongType.get()) - val schema = new org.apache.iceberg.Schema(java.util.Arrays.asList(idField, field)) - val json = org.apache.iceberg.SchemaParser.toJson(schema) - println(s"fork.colDefault.api: column-default API present, serialized schema JSON = $json") - - // (a) The default is serialized into the schema JSON. - assert(json.contains("initial-default"), - s"expected SchemaParser to serialize 'initial-default' into the schema JSON, got: $json") - // (b) toJson takes no format-version argument, so the key serializes the same regardless of format version. (c) The - // value round-trips through fromJson then toJson. - val reparsed = org.apache.iceberg.SchemaParser.fromJson(json) - val json2 = org.apache.iceberg.SchemaParser.toJson(reparsed) - assert(json2.contains("initial-default"), - s"expected 'initial-default' to survive the fromJson/toJson round trip, got: $json2") - println("fork.colDefault.api: initial-default serialized with no format-version argument and round-trips") - } - - /** - * Reflectively builds an optional int NestedField carrying the given initial default. Returns None when the builder - * API is absent, so a caller can assert that absence directly. - */ - private def buildDefaultedIntField(id: Int, name: String, dflt: Int): Option[org.apache.iceberg.types.Types.NestedField] = { - val nfCls = Class.forName("org.apache.iceberg.types.Types$NestedField") - val bm = scala.util.Try(nfCls.getMethod("builder")) - if (bm.isFailure) return None - def chain(b: AnyRef, m: String, at: Class[_], a: AnyRef): AnyRef = b.getClass.getMethod(m, at).invoke(b, a) - def chain0(b: AnyRef, m: String): AnyRef = b.getClass.getMethod(m).invoke(b) - val intType = Class.forName("org.apache.iceberg.types.Types$IntegerType").getMethod("get").invoke(null) - var b = chain(bm.get.invoke(null), "withId", java.lang.Integer.TYPE, java.lang.Integer.valueOf(id)) - b = chain(b, "withName", classOf[String], name) - b = chain(b, "ofType", Class.forName("org.apache.iceberg.types.Type"), intType) - b = chain0(b, "asOptional") - b = chain(b, "withInitialDefault", classOf[Object], java.lang.Integer.valueOf(dflt)) - Some(b.getClass.getMethod("build").invoke(b).asInstanceOf[org.apache.iceberg.types.Types.NestedField]) - } - - /** - * A column default added after data files exist persists into the committed schema. The schema evolution goes through - * the low-level TableMetadata API because the public UpdateSchema surface has no set-default operation. The - * documented read contract covers schema persistence only. The case prints the OSS Spark read result for pre-existing - * rows as diagnostic output, while its assertions stop at the persisted schema. - */ - private def forkColDefaultReadApplyProbe(ctx: Ctx): Unit = { - val spark = ctx.spark - val nfCls = Class.forName("org.apache.iceberg.types.Types$NestedField") - val apiPresent = scala.util.Try(nfCls.getMethod("builder")).isSuccess - if (!apiPresent) { - // No builder API means there is no way to set a default, so assert that absence directly. - println("fork.colDefault.readApplyProbe: column-default builder API is absent, nothing to probe") - assert(!nfCls.getMethods.map(_.getName).toSet.contains("initialDefault"), - "NestedField exposes initialDefault but builder() is absent") - return - } - val cat = "coldefroapply" - val wh = s"/tmp/coldef-readapply-${System.nanoTime()}" - spark.conf.set(s"spark.sql.catalog.$cat", "org.apache.iceberg.spark.SparkCatalog") - spark.conf.set(s"spark.sql.catalog.$cat.type", "hadoop") - spark.conf.set(s"spark.sql.catalog.$cat.warehouse", wh) - val t = s"$cat.d.t_readapply" - spark.sql(s"DROP TABLE IF EXISTS $t") - spark.sql(s"CREATE TABLE $t (id bigint) USING $dataSource") - spark.sql(s"INSERT INTO $t VALUES (1),(2)") // data files physically contain only id - - // Evolve the schema to [id, c int DEFAULT 5] directly through TableMetadata. - val table = org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, t) - val cur = table.schema() - val nextId = cur.highestFieldId() + 1 - val cField = buildDefaultedIntField(nextId, "c", 5).getOrElse( - throw new AssertionError("builder API present but field build failed")) - val cols = new java.util.ArrayList[org.apache.iceberg.types.Types.NestedField](cur.columns()) - cols.add(cField) - val s2 = new org.apache.iceberg.Schema(cols) - val ops = table.asInstanceOf[org.apache.iceberg.HasTableOperations].operations() - val base = ops.current() - val updated = org.apache.iceberg.TableMetadata.buildFrom(base).setCurrentSchema(s2, s2.highestFieldId()).build() - ops.commit(base, updated) - - // The default persists into the committed schema. - val persisted = org.apache.iceberg.SchemaParser.toJson( - org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, t).schema()) - assert(persisted.contains("initial-default"), - s"expected initial-default to persist into the committed schema, got: $persisted") - - // Recorded for reference only: the read path's treatment of the defaulted column over old files is not part of this - // connector's documented contract. - spark.sql(s"REFRESH TABLE $t") - val vals = spark.sql(s"SELECT c FROM $t ORDER BY id").collect() - .map(r => if (r.isNullAt(0)) "NULL" else r.getInt(0).toString) - println(s"fork.colDefault.readApplyProbe: read of defaulted column over pre-existing rows = " + - s"[${vals.mkString(",")}] (recorded for reference, not asserted)") - spark.sql(s"DROP TABLE IF EXISTS $t") - } - - /** - * A partitioned write defaults write.distribution-mode to NONE, so every input task writes every partition it holds - * and one append produces up to (input tasks times partitions) data files. Under an explicit HASH distribution the - * writer shuffles rows so one task owns each partition, clustering the append to roughly one file per partition. - * Appending the same multi-task DataFrame into a 4-partition table under each mode therefore yields at least as many - * files under the default as under HASH. The file format is the parameter. - */ - private def forkPartitionDistDefault(fmt: String)(ctx: Ctx): Unit = { - val spark = ctx.spark - val nParts = 4 - val nTasks = 8 - def buildAndCountFiles(tbl: String, extraProps: String): Long = { - spark.sql(s"DROP TABLE IF EXISTS $tbl") - spark.sql(s"CREATE TABLE $tbl (id bigint, p int) USING $dataSource PARTITIONED BY (p) " + - s"TBLPROPERTIES ('format-version'='2', 'write.format.default'='$fmt'$extraProps)") - // nTasks input partitions, each holding rows for all nParts table partitions. - val df = spark.range(0, 400) - .selectExpr("id", s"cast(id % $nParts as int) as p") - .repartition(nTasks) - df.writeTo(tbl).append() - val n = spark.sql(s"SELECT count(*) FROM $tbl.data_files").collect()(0).getLong(0) - spark.sql(s"DROP TABLE IF EXISTS $tbl") - n - } - val nDefault = buildAndCountFiles(s"${ctx.namespace}.t_dist_def_$fmt", "") - val nHash = buildAndCountFiles(s"${ctx.namespace}.t_dist_hash_$fmt", ", 'write.distribution-mode'='hash'") - println(s"fork.partitionDist[$fmt]: defaultFiles=$nDefault hashFiles=$nHash (parts=$nParts tasks=$nTasks)") - // Explicit hash clusters by partition, with slack for spill. - assert(nHash <= nParts * 2, - s"[$fmt] write.distribution-mode=hash should cluster to about $nParts files, got $nHash") - assert(nDefault > nHash, - s"[$fmt] expected the default distribution mode to produce more files than hash " + - s"(default=$nDefault hash=$nHash)") - } - - /** Returns the count and the total byte size of the table's current data files. */ - private def dataFileStats(spark: SparkSession, table: String): (Long, Long) = { - val r = spark.sql(s"SELECT count(*), coalesce(sum(file_size_in_bytes), 0) FROM $table.data_files").collect()(0) - (r.getLong(0), r.getLong(1)) - } - - /** - * OutputFileFactory exposes FILE_REPLICATION_FACTOR as "file-replication-factor", and a factory built with a - * replication factor stamps that key into the property map of the output files it creates. Writes made through the - * table afterward still return the correct rows. It is not a settable table property; it is the key HDFS reads to set - * block replication on an output file when a replication factor is supplied to the factory, and the delete-file write - * path is the one path that supplies one. Reflection reaches the builder and getProperties because some Iceberg - * artifacts leave them out of the public compiled API. - */ - private def forkFileReplicationFactor(ctx: Ctx): Unit = { - val spark = ctx.spark - val offCls = Class.forName("org.apache.iceberg.io.OutputFileFactory") - - // (1) Assert the exact key string. - val keyFieldT = scala.util.Try(offCls.getField("FILE_REPLICATION_FACTOR")) - assert(keyFieldT.isSuccess, "OutputFileFactory.FILE_REPLICATION_FACTOR is absent") - val key = keyFieldT.get.get(null).asInstanceOf[String] - assert(key == "file-replication-factor", - s"""expected FILE_REPLICATION_FACTOR to equal "file-replication-factor", got "$key"""") - - // Need a real Iceberg Table to build a factory. - val table = s"${ctx.namespace}.t_filerepl" - spark.sql(s"DROP TABLE IF EXISTS $table") - spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES ('format-version'='2')") - spark.sql(s"INSERT INTO $table VALUES (1,'a'),(2,'b')") - val icebergTable = org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, table) - - // (2) Build an OutputFileFactory carrying replicationFactor=2. - val builder = offCls.getMethod("builderFor", classOf[org.apache.iceberg.Table], java.lang.Integer.TYPE, java.lang.Long.TYPE) - .invoke(null, icebergTable, java.lang.Integer.valueOf(1), java.lang.Long.valueOf(1L)) - val replMT = scala.util.Try(builder.getClass.getMethod("replicationFactor", java.lang.Short.TYPE)) - assert(replMT.isSuccess, "OutputFileFactory.Builder.replicationFactor(short) is absent") - replMT.get.invoke(builder, java.lang.Short.valueOf(2.toShort)) - val factory = Option(builder.getClass.getMethod("build").invoke(builder)) - .getOrElse(throw new AssertionError("OutputFileFactory build returned null")) - - // (3) The factory stamps FILE_REPLICATION_FACTOR -> "2" into the per-output-file property map. - val gp = offCls.getDeclaredMethod("getProperties"); gp.setAccessible(true) - val props = gp.invoke(factory).asInstanceOf[java.util.Map[String, String]] - assert(props.get(key) == "2", - s"expected output-file property $key=2 stamped by the factory, got ${props.get(key)}") - - // (4) Writes still succeed and rows are correct. - spark.sql(s"INSERT INTO $table VALUES (3,'c')") - val rows = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) - assert(rows == Seq(1L, 2L, 3L), s"rows wrong after write: $rows") - - println(s"fork.fileReplicationFactor: key='$key'; factory stamps $key=${props.get(key)} into output-file props; " + - s"writes ok rows=${rows.mkString(",")}") - spark.sql(s"DROP TABLE IF EXISTS $table") - } - - /** - * spark.sql.iceberg.split-size decides how the read path combines data files into read tasks. Over several small - * files, a large split size combines them into fewer read tasks and a tiny split size splits them into more, visible - * through rdd.getNumPartitions, and both reads return the same rows. The planner shows the same effect directly: a - * split size above the whole table plans one task group, and a split size below one file plans one group per file. - * The file format is the parameter. - */ - private def forkSplitSize(fmt: String)(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_splitsize_$fmt" - spark.sql(s"DROP TABLE IF EXISTS $table") - // distribution=none plus several separate inserts produces several distinct data files. An open-file-cost of 1 sets - // each file's planning weight to its byte length, making split-size the knob that governs task-group count. - spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$fmt', 'write.distribution-mode'='none', 'read.split.open-file-cost'='1')") - val numberOfFiles = 6 - (0 until numberOfFiles).foreach { fileIndex => - spark.sql(s"INSERT INTO $table SELECT ${fileIndex}L, repeat('r$fileIndex', 4000)") - } - val fileCount = spark.sql(s"SELECT count(*) FROM $table.data_files").collect()(0).getLong(0) - assert(fileCount >= 2, s"[$fmt] expected multiple data files for a split test, got $fileCount") - - val key = org.apache.iceberg.spark.SparkSQLProperties.SPLIT_SIZE // "spark.sql.iceberg.split-size" - val saved = spark.conf.getOption(key) - def keys(): Seq[Long] = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) - def rddParts(): Int = spark.sql(s"SELECT * FROM $table").rdd.getNumPartitions - val expected = (0 until numberOfFiles).map(_.toLong) - try { - // (a) Set spark.sql.iceberg.split-size directly and read the multi-file table under a large and a tiny split - // size; the row set must be invariant either way. - spark.conf.set(key, (512L * 1024 * 1024).toString) - val bigRows = keys(); val bigRdd = rddParts() - spark.conf.set(key, "1") - val smallRows = keys(); val smallRdd = rddParts() - assert(bigRows == expected && smallRows == expected, - s"[$fmt] split-size must not change the row set: big=$bigRows small=$smallRows expected=$expected") - assert(smallRdd >= bigRdd, - s"[$fmt] a smaller split-size must not decrease the read RDD partition count: small=$smallRdd big=$bigRdd") - - // (b) The same knob checked directly at the planner: with open-file-cost=1, each file's planning weight is its - // byte length, so a split-size below one file combines nothing (one task group per file) while a split-size above - // the whole table combines everything into one group. - val ice = org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark, table) - val szKey = org.apache.iceberg.TableProperties.SPLIT_SIZE // "read.split.target-size" - def planGroups(splitBytes: Long): Int = { - val it = ice.newScan().option(szKey, splitBytes.toString).planTasks().iterator() - var n = 0; while (it.hasNext) { it.next(); n += 1 } - n - } - val bigGroups = planGroups(512L * 1024 * 1024) // one combined group - val smallGroups = planGroups(1L) // one group per file - assert(bigGroups == 1, s"[$fmt] a split-size above the whole table should plan 1 task group, got $bigGroups") - assert(smallGroups == fileCount, - s"[$fmt] a split-size below one file should plan one task group per file ($fileCount), got $smallGroups") - - println(s"fork.splitSize[$fmt]: key='$key' files=$fileCount " + - s"rddParts(big=$bigRdd,small=$smallRdd) plannedTaskGroups(bigSplit=$bigGroups,smallSplit=$smallGroups)") - } finally { - saved match { case Some(v) => spark.conf.set(key, v); case None => spark.conf.unset(key) } - spark.sql(s"DROP TABLE IF EXISTS $table") - } - } - - /** - * rewrite_data_files packs data files into rewrite groups weighted by file length. Compacting a table whose data - * files are unevenly sized preserves the row count and every row's value, which is the observable result of that - * packing; the weighting itself is a planner decision that no SQL surface exposes. The file format is the parameter. - */ - private def forkBinPackByLength(fmt: String)(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_binpack_$fmt" - spark.sql(s"DROP TABLE IF EXISTS $table") - spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$fmt', 'write.distribution-mode'='none')") - // Unevenly sized data files: a tiny one, a small one, and a big one. - spark.sql(s"INSERT INTO $table VALUES (1,'a')") - spark.sql(s"INSERT INTO $table VALUES (2,'b'),(3,'c')") - spark.sql(s"INSERT INTO $table SELECT id, repeat('x', 200) FROM range(100, 400)") - val before = dataFileStats(spark, table) - assert(before._1 >= 3, s"[$fmt] expected at least 3 uneven data files, got ${before._1}") - val totalRows = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) - - spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") - - val after = dataFileStats(spark, table) - val totalRows2 = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) - assert(totalRows2 == totalRows, s"[$fmt] rewrite_data_files changed the row count: $totalRows -> $totalRows2") - val probe = spark.sql(s"SELECT s FROM $table WHERE id = 1").collect()(0).getString(0) - assert(probe == "a", s"[$fmt] rewrite altered a row: id=1 s=$probe") - - println(s"fork.binPackByLength[$fmt]: beforeFiles=${before._1} beforeBytes=${before._2} " + - s"afterFiles=${after._1} afterBytes=${after._2} rows=$totalRows") - spark.sql(s"DROP TABLE IF EXISTS $table") - } - - /** - * file_sequence_number is exposed on the live data-file entries of the entries metadata table and increases - * monotonically across commits, and rewrite_data_files with rewrite-all preserves the row count and the row set. A - * budgeted rewrite spends its budget in file-sequence-number order, so that column is the observable half of the - * ordering decision. Sequence numbers order commits the same way in every file format, so parquet alone covers this - * behavior. - */ - private def forkCompactionOrder(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_compord" - spark.sql(s"DROP TABLE IF EXISTS $table") - spark.sql(s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + - "TBLPROPERTIES ('write.format.default'='parquet', 'write.distribution-mode'='none')") - // Several commits produce several data files with distinct, increasing file-sequence-numbers. - val numberOfCommits = 4 - (0 until numberOfCommits).foreach { commitIndex => - spark.sql(s"INSERT INTO $table VALUES (${commitIndex}L, 'c$commitIndex')") - } - val seqs = spark.sql( - s"SELECT file_sequence_number FROM $table.entries WHERE status != 2 AND data_file.content = 0 " + - s"ORDER BY file_sequence_number").collect().toSeq.map(_.getLong(0)) - assert( - seqs.size >= numberOfCommits, - s"expected at least $numberOfCommits live data-file entries with sequence numbers, got ${seqs.size}: $seqs") - assert(seqs == seqs.sorted, s"file sequence numbers not monotonic: $seqs") - assert(seqs.distinct.size >= 2, s"expected multiple distinct file sequence numbers, got ${seqs.distinct}") - val totalRows = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) - - spark.sql(s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', options => map('rewrite-all', 'true'))") - - val totalRowsAfter = spark.sql(s"SELECT count(*) FROM $table").collect()(0).getLong(0) - assert(totalRowsAfter == totalRows, s"rewrite changed the row count: $totalRows -> $totalRowsAfter") - val filesAfter = spark.sql(s"SELECT count(*) FROM $table.data_files").collect()(0).getLong(0) - val keys = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) - assert(keys == (0 until numberOfCommits).map(_.toLong), s"rewrite altered the row set: $keys") - - println(s"fork.compactionOrder: fileSeqNumbers=${seqs.mkString(",")} filesBefore=${seqs.size} " + - s"filesAfter=$filesAfter rows=$totalRows") - spark.sql(s"DROP TABLE IF EXISTS $table") - } - - /** The column-default and write-distribution fork cases. */ - val forkColumnDefaultAndDistributionCases: List[Plan.Case] = - List( - Plan.Case( - "fork.colDefault.addColumnInert @ parquet", - forkColDefaultAddColumn("parquet")), - Plan.Case( - "fork.colDefault.addColumnInert @ orc", - forkColDefaultAddColumn("orc")), - Plan.Case( - "fork.colDefault.apiSerialization @ core", - forkColDefaultApiSerialization), - Plan.Case( - "fork.colDefault.readApplyProbe @ core", - forkColDefaultReadApplyProbe), - Plan.Case( - "fork.partitionDist.default @ parquet", - forkPartitionDistDefault("parquet")), - Plan.Case( - "fork.partitionDist.default @ orc", - forkPartitionDistDefault("orc"))) - - /** - * The output-file, split-size and compaction fork cases. They are the second of two fork contribution lists: one more - * fork entry sits between the two in the catalog, supplied by the layer that owns it, and Plan keeps that order. - */ - val forkFileAndCompactionCases: List[Plan.Case] = - List( - Plan.Case( - "fork.fileReplicationFactor @ core", - forkFileReplicationFactor), - Plan.Case( - "fork.splitSize @ parquet", - forkSplitSize("parquet")), - Plan.Case( - "fork.splitSize @ orc", - forkSplitSize("orc")), - Plan.Case( - "fork.binPackByLength @ parquet", - forkBinPackByLength("parquet")), - Plan.Case( - "fork.binPackByLength @ orc", - forkBinPackByLength("orc")), - Plan.Case( - "fork.compactionOrder @ parquet", - forkCompactionOrder)) - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala index ddb735703..3b46ad169 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala @@ -1,10 +1,14 @@ package harness import org.apache.spark.sql.{Row, SparkSession} +import java.math.{BigDecimal => JavaBigDecimal} +import java.net.{ConnectException, SocketException, SocketTimeoutException} +import java.sql.{Date, Timestamp} import java.time.LocalDate import java.time.LocalDateTime import java.time.format.DateTimeFormatter import java.util.UUID +import java.util.concurrent.atomic.AtomicInteger import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal @@ -25,20 +29,24 @@ object Rest { .header("Authorization", s"Bearer ${ctx.restToken}") .header("Content-Type", "application/json") def post(ctx: Ctx, path: String, body: String): (Int, String) = { - val r = client.send(base(ctx, path).POST(HttpRequest.BodyPublishers.ofString(body)).build(), HttpResponse.BodyHandlers.ofString()) - (r.statusCode(), r.body()) + val response = client.send( + base(ctx, path).POST(HttpRequest.BodyPublishers.ofString(body)).build(), + HttpResponse.BodyHandlers.ofString()) + (response.statusCode(), response.body()) } def delete(ctx: Ctx, path: String): (Int, String) = { - val r = client.send(base(ctx, path).DELETE().build(), HttpResponse.BodyHandlers.ofString()) - (r.statusCode(), r.body()) + val response = client.send(base(ctx, path).DELETE().build(), HttpResponse.BodyHandlers.ofString()) + (response.statusCode(), response.body()) } def put(ctx: Ctx, path: String, body: String): (Int, String) = { - val r = client.send(base(ctx, path).PUT(HttpRequest.BodyPublishers.ofString(body)).build(), HttpResponse.BodyHandlers.ofString()) - (r.statusCode(), r.body()) + val response = client.send( + base(ctx, path).PUT(HttpRequest.BodyPublishers.ofString(body)).build(), + HttpResponse.BodyHandlers.ofString()) + (response.statusCode(), response.body()) } def get(ctx: Ctx, path: String): (Int, String) = { - val r = client.send(base(ctx, path).GET().build(), HttpResponse.BodyHandlers.ofString()) - (r.statusCode(), r.body()) + val response = client.send(base(ctx, path).GET().build(), HttpResponse.BodyHandlers.ofString()) + (response.statusCode(), response.body()) } } @@ -78,10 +86,11 @@ object Exceptions { * assertion failures surface on their first attempt. */ def isTransient(throwable: Throwable): Boolean = causeChain(throwable).exists { - case _: java.net.SocketTimeoutException => true - case _: java.net.ConnectException => true - case e: java.net.SocketException => Option(e.getMessage).exists(_.toLowerCase.contains("reset")) - case _ => false + case _: SocketTimeoutException => true + case _: ConnectException => true + case socketFailure: SocketException => + Option(socketFailure.getMessage).exists(_.toLowerCase.contains("reset")) + case _ => false } } @@ -154,11 +163,22 @@ object CoreTable extends Schema { // A schema exercising complex/nested types: a struct, an array, a map, and a struct-in-struct. Struct/array read back // as Row/Seq; map as a Map. `id` is first so it is the ordering key. object NestedTable extends Schema { - val id: Column[Long] = Column("id", "bigint", rowIndex => rowIndex.toString) - val s: Column[Row] = Column("s", "struct", rowIndex => s"named_struct('x', $rowIndex, 'y', 'row-$rowIndex')") - val arr: Column[Seq[Int]] = Column("arr", "array", rowIndex => s"array($rowIndex, ${rowIndex + 1})") - val m: Column[Map[String, Int]] = Column("m", "map", rowIndex => s"map('k', $rowIndex)") - val nested: Column[Row] = Column("nested", "struct>", rowIndex => s"named_struct('inner', named_struct('z', $rowIndex))") + val id: Column[Long] = + Column("id", "bigint", rowIndex => rowIndex.toString) + val s: Column[Row] = + Column( + "s", + "struct", + rowIndex => s"named_struct('x', $rowIndex, 'y', 'row-$rowIndex')") + val arr: Column[Seq[Int]] = + Column("arr", "array", rowIndex => s"array($rowIndex, ${rowIndex + 1})") + val m: Column[Map[String, Int]] = + Column("m", "map", rowIndex => s"map('k', $rowIndex)") + val nested: Column[Row] = + Column( + "nested", + "struct>", + rowIndex => s"named_struct('inner', named_struct('z', $rowIndex))") def tableColumns: Seq[Column[_]] = Seq(id, s, arr, m, nested) val columnDefinitions: String = @@ -168,24 +188,30 @@ object NestedTable extends Schema { // A schema for type-edge coverage: the common scalar types, exercised with nulls, special float values, boundary // values, and unicode/empty strings. object TypesTable extends Schema { - val id: Column[Long] = Column("id", "bigint", rowIndex => rowIndex.toString) - val n: Column[Int] = Column("n", "int", rowIndex => rowIndex.toString) - val x: Column[Double] = Column("x", "double", rowIndex => s"$rowIndex.5") - val dec: Column[java.math.BigDecimal] = Column("dec", "decimal(10,2)", rowIndex => s"CAST($rowIndex.50 AS decimal(10,2))") - val str: Column[String] = Column("str", "string", rowIndex => s"'row-$rowIndex'") - val bin: Column[Array[Byte]] = Column("bin", "binary", rowIndex => s"CAST('bin-$rowIndex' AS binary)") - val dt: Column[java.sql.Date] = + val id: Column[Long] = + Column("id", "bigint", rowIndex => rowIndex.toString) + val n: Column[Int] = + Column("n", "int", rowIndex => rowIndex.toString) + val x: Column[Double] = + Column("x", "double", rowIndex => s"$rowIndex.5") + val dec: Column[JavaBigDecimal] = + Column("dec", "decimal(10,2)", rowIndex => s"CAST($rowIndex.50 AS decimal(10,2))") + val str: Column[String] = + Column("str", "string", rowIndex => s"'row-$rowIndex'") + val bin: Column[Array[Byte]] = + Column("bin", "binary", rowIndex => s"CAST('bin-$rowIndex' AS binary)") + val dt: Column[Date] = Column( "dt", "date", rowIndex => s"DATE '${DateEpoch.plusDays((rowIndex - 1).toLong)}'") - val ts: Column[java.sql.Timestamp] = + val ts: Column[Timestamp] = Column( "ts", "timestamp", rowIndex => s"TIMESTAMP '${TimestampEpoch.plusHours((rowIndex - 1).toLong).format(TimestampFormat)}'") - val tsntz: Column[java.time.LocalDateTime] = + val tsntz: Column[LocalDateTime] = Column( "tsntz", "timestamp_ntz", @@ -194,7 +220,8 @@ object TypesTable extends Schema { def tableColumns: Seq[Column[_]] = Seq(id, n, x, dec, str, bin, dt, ts, tsntz) val columnDefinitions: String = - "id bigint, n int, x double, dec decimal(10,2), str string, bin binary, dt date, ts timestamp, tsntz timestamp_ntz" + "id bigint, n int, x double, dec decimal(10,2), str string, bin binary, dt date, ts timestamp, " + + "tsntz timestamp_ntz" private val DateEpoch = LocalDate.of(2024, 1, 1) private val TimestampEpoch = LocalDateTime.of(2024, 1, 1, 0, 0) @@ -322,33 +349,44 @@ final class TableTest[S <: Schema] private (val schema: S, val steps: Vector[Ste } private[harness] object OwnedTableLifecycle { - def withOwnership(dropOwnedTable: => Unit)(use: (() => Unit) => Unit): Unit = { - var tableCreated = false - var testFailure: Option[Throwable] = None + /** + * Runs `use`, then runs `cleanUp` on every outcome. A failure from `use` is the failure the caller sees, with a + * cleanup failure attached to it as a suppressed exception. When `use` returns normally a cleanup failure is the + * failure the caller sees, so cleanup that silently fails cannot pass for a clean run. + */ + def withCleanup(cleanUp: => Unit)(use: => Unit): Unit = { + var primaryFailure: Option[Throwable] = None try { - use(() => tableCreated = true) + use } catch { case failure: Throwable => - testFailure = Some(failure) + primaryFailure = Some(failure) throw failure } finally { - if (tableCreated) { - try { - dropOwnedTable - } catch { - case cleanupFailure: Throwable => - testFailure match { - case Some(failure) => failure.addSuppressed(cleanupFailure) - case None => throw cleanupFailure - } - } + try { + cleanUp + } catch { + case cleanupFailure: Throwable => + primaryFailure match { + case Some(failure) => failure.addSuppressed(cleanupFailure) + case None => throw cleanupFailure + } } } } + + /** + * Runs `use` with a mark it calls once the table exists. `dropOwnedTable` runs only when that mark was set, so a + * create that fails leaves whatever already answered to the name untouched. + */ + def withOwnership(dropOwnedTable: => Unit)(use: (() => Unit) => Unit): Unit = { + var tableCreated = false + withCleanup(if (tableCreated) dropOwnedTable)(use(() => tableCreated = true)) + } } object TableTest { - private val counter = new java.util.concurrent.atomic.AtomicInteger(0) + private val counter = new AtomicInteger(0) def apply[S <: Schema](schema: S): TableTest[S] = new TableTest(schema, Vector.empty) def seedCounter(value: Int): Unit = counter.set(value) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala deleted file mode 100644 index 82cd9e392..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/HazardReaderWriterScenarios.scala +++ /dev/null @@ -1,643 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The copy-on-write reader, writer and hazard families. The reader and writer cases pin the changelog view, the -// incremental read and the structured-streaming reader and writer against a plain copy-on-write table. The hazard cases -// pin what happens when two operations that can interfere are run against the same table. Plan crosses every family -// here with the parquet and orc file formats. `cowCreate` states the standard copy-on-write table shape, so a feature -// layer reaches it through a self-type on this trait. -trait HazardReaderWriterScenarios extends ScenarioKit { - import Rows._ - - /** The CREATE statement for a copy-on-write table in the given file format. */ - protected def cowCreate(t: String, fmt: String): String = - s"CREATE TABLE $t ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')" - - /** - * Three seed rows in a copy-on-write table in the given file format. Each family builds its own table from this - * recipe, so a family reads on its own. - */ - private def cowPreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => cowCreate(table, format))() - .insert(3)()) - - /** A changelog view over an appended row reports exactly one INSERT and no DELETE. */ - private def readerWriterChangelogAppendCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.changelog.append") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.append: $changeTypes") - assert( - changeTypes.getOrElse("INSERT", 0L) == 1 && - !changeTypes.contains("DELETE"), - s"append changelog must contain one INSERT and no DELETE: $changeTypes") - } - - /** The changelog case for an append, on three seed rows in the given file format. */ - def readerWriterChangelogAppendCases(format: String): List[Plan.Case] = - List( - readerWriterChangelogAppendCase(format)) - - /** A changelog view over an INSERT OVERWRITE that drops one row reports exactly that row as a DELETE. */ - private def readerWriterChangelogOverwriteCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.changelog.overwrite") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT OVERWRITE ${table.name} " + - s"SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.overwrite: $changeTypes") - assert( - changeTypes == Map("DELETE" -> 1L), - s"overwrite changelog must contain the one removed row: $changeTypes") - } - - /** The changelog case for an INSERT OVERWRITE, on three seed rows in the given file format. */ - def readerWriterChangelogOverwriteCases(format: String): List[Plan.Case] = - List( - readerWriterChangelogOverwriteCase(format)) - - /** A changelog view over a DELETE reports exactly one DELETE and no INSERT. */ - private def readerWriterChangelogDeleteCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.changelog.delete") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.delete: $changeTypes") - assert( - changeTypes.getOrElse("DELETE", 0L) == 1 && - !changeTypes.contains("INSERT"), - s"delete changelog must contain one DELETE and no INSERT: $changeTypes") - } - - /** The changelog case for a DELETE, on three seed rows in the given file format. */ - def readerWriterChangelogDeleteCases(format: String): List[Plan.Case] = - List( - readerWriterChangelogDeleteCase(format)) - - /** A changelog view over an UPDATE reports the old row as a DELETE and the new value as an INSERT. */ - private def readerWriterChangelogUpdateCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.changelog.update") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + - s"WHERE ${Core.long0.columnName} = 2") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.update: $changeTypes") - assert( - changeTypes == Map("DELETE" -> 1L, "INSERT" -> 1L), - s"update changelog must contain the old and new row versions: $changeTypes") - } - - /** The changelog case for an UPDATE, on three seed rows in the given file format. */ - def readerWriterChangelogUpdateCases(format: String): List[Plan.Case] = - List( - readerWriterChangelogUpdateCase(format)) - - /** A changelog view over a MERGE that updates one row and inserts another reports one DELETE and two INSERTs. */ - private def readerWriterChangelogMergeCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.changelog.merge") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"MERGE INTO ${table.name} target " + - "USING (SELECT CAST(2 AS BIGINT) key " + - "UNION ALL SELECT CAST(9 AS BIGINT)) source " + - s"ON target.${Core.long0.columnName} = source.key " + - s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + - "WHEN NOT MATCHED THEN INSERT " + - s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + - s"${Core.string0.columnName}, ${Core.double0.columnName}, " + - s"${Core.boolean0.columnName}, ${Core.date0.columnName}) " + - "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')") - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - val changeTypes = table.spark - .sql(s"SELECT _change_type, count(*) AS c FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - println(s"DIAG changelog.merge: $changeTypes") - assert( - changeTypes == Map("DELETE" -> 1L, "INSERT" -> 2L), - s"merge changelog must contain one update and one insert: $changeTypes") - } - - /** The changelog case for a MERGE, on three seed rows in the given file format. */ - def readerWriterChangelogMergeCases(format: String): List[Plan.Case] = - List( - readerWriterChangelogMergeCase(format)) - - /** An incremental scan spanning an appended row returns exactly that one row. */ - private def readerWriterIncrementalAppendCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.incremental.append") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() - - println(s"DIAG incremental.append: added=$addedRowCount") - assert( - addedRowCount == 1, - s"append incremental scan should contain one row, got $addedRowCount") - } - - /** An incremental scan spanning a DELETE-only snapshot returns no rows. */ - private def readerWriterIncrementalDeleteCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.incremental.delete") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() - - println(s"DIAG incremental.delete: added=$addedRowCount") - assert( - addedRowCount == 0, - s"delete-only incremental scan must not return appended rows: $addedRowCount") - } - - /** An incremental scan spanning an INSERT OVERWRITE that only removes rows returns no rows. */ - private def readerWriterIncrementalOverwriteCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.incremental.overwrite") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"INSERT OVERWRITE ${table.name} " + - s"SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} <= 2") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() - - println(s"DIAG incremental.overwrite: added=$addedRowCount") - assert( - addedRowCount == 0, - s"overwrite-only incremental scan must not return appended rows: $addedRowCount") - } - - /** An incremental scan spanning an UPDATE-only snapshot returns no rows. */ - private def readerWriterIncrementalUpdateCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.incremental.update") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql( - s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'upd' " + - s"WHERE ${Core.long0.columnName} = 2") - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", seedSnapshotId) - .option("end-snapshot-id", currentSnapshotId) - .load(table.name) - .count() - - println(s"DIAG incremental.update: added=$addedRowCount") - assert( - addedRowCount == 0, - s"update-only incremental scan must not return appended rows: $addedRowCount") - } - - /** - * A streaming read of the table delivers the seed rows on first run and the newly inserted row after restart, into a - * destination table. - */ - private def readerWriterStreamAppendCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.stream.append") { table => - val destination = s"${table.name}_s" - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - table.spark.sql(cowCreate(destination, format)) - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-rw").toString - def runStream(): Unit = { - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("iceberg") - .outputMode("append") - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .toTable(destination) - assert(query.awaitTermination(120000), "stream did not finish") - query.stop() - } - - try { - runStream() - assert( - countOf(table.spark, s"SELECT count(*) FROM $destination") == "3", - "initial stream did not deliver the seed") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - runStream() - assert( - countOf(table.spark, s"SELECT count(*) FROM $destination") == "4", - "stream restart did not deliver the appended row") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - } - } - - /** - * An append-only stream restarted after a DELETE snapshot was written fails, with an error mentioning delete or - * overwrite. - */ - private def readerWriterStreamDeleteRejectedCase(format: String): Plan.Case = - cowPreparation(format).test("readerWriter.stream.deleteRejected") { table => - val destination = s"${table.name}_sd" - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - table.spark.sql(cowCreate(destination, format)) - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-rwd").toString - def runStream(): Unit = { - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("iceberg") - .outputMode("append") - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .toTable(destination) - assert(query.awaitTermination(120000), "stream did not finish") - query.stop() - } - - try { - runStream() - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val exception = Check.intercept[Exception](runStream()) - - println( - "DIAG stream.afterDelete: " + - s"${exception.getClass.getSimpleName} :: " + - Option(exception.getMessage).getOrElse("").take(140)) - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage).exists(message => - message.toLowerCase.contains("delete") || - message.toLowerCase.contains("overwrite"))), - "append-only stream should reject a delete snapshot") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - } - } - - /** - * The incremental reads between two snapshots and the structured-streaming reader and writer, on three seed rows in - * the given file format. - */ - def readerWriterIncrementalAndStreamCases(format: String): List[Plan.Case] = - List( - readerWriterIncrementalAppendCase(format), - readerWriterIncrementalDeleteCase(format), - readerWriterIncrementalOverwriteCase(format), - readerWriterIncrementalUpdateCase(format), - readerWriterStreamAppendCase(format), - readerWriterStreamDeleteRejectedCase(format)) - - /** - * A streaming read that resumes after its earliest offset snapshot has been expired fails, with an error naming the - * expired or missing snapshot. - */ - private def hazardStreamExpiredCheckpointCase( - format: String, - basePreparation: TablePreparation[CoreTable.type]): Plan.Case = - basePreparation.test("hazard.stream.expiredCheckpoint") { table => - val destination = s"${table.name}_sink" - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - table.spark.sql(cowCreate(destination, format)) - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-hazard").toString - def runStream(): Unit = { - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("iceberg") - .outputMode("append") - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .toTable(destination) - assert(query.awaitTermination(120000), "stream did not finish") - query.stop() - } - - try { - runStream() - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $destination") == "3", - "initial stream should deliver the seed") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - runStream() - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $destination") == "4", - "control restart should deliver one incremental row") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - val exception = Check.intercept[Exception](runStream()) - - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage).exists(message => - message.contains("expired or removed") || - message.contains("Cannot load current offset") || - message.contains("Cannot find snapshot"))), - "stream restart should report the expired checkpoint offset") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $destination") - } - } - - /** - * After expire_snapshots removes a changelog start point, create_changelog_view over that start point either throws - * or reports fewer changes than the table's history holds, and any message it throws leaves expiration unnamed. The - * case covers three start points: an expired snapshot ID, a timestamp older than the whole history, and a timestamp - * inside the expired range. - */ - private def hazardCdcExpiredRangeCase( - basePreparation: TablePreparation[CoreTable.type]): Plan.Case = - basePreparation.test("hazard.cdc.expiredRange") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - val snapshots = snapshotIds(table.spark, table.name) - val firstTimestamp = table.spark - .sql( - s"SELECT committed_at FROM ${table.name}.snapshots " + - "ORDER BY committed_at LIMIT 1") - .collect()(0) - .getTimestamp(0) - val middleTimestamp = table.spark - .sql( - s"SELECT committed_at FROM ${table.name}.snapshots " + - s"WHERE snapshot_id = ${snapshots(1)}") - .collect()(0) - .getTimestamp(0) - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - def changelog( - optionKey: String, - optionValue: String, - trueChangeCount: Long): String = - try { - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('$optionKey', '$optionValue'))") - .collect()(0) - .getString(0) - val actualChangeCount = table.spark - .sql(s"SELECT count(*) FROM $view") - .collect()(0) - .getLong(0) - if (actualChangeCount < trueChangeCount) { - s"SILENT under-report: $actualChangeCount of " + - s"$trueChangeCount true changes" - } else { - s"FULL: $actualChangeCount of $trueChangeCount" - } - } catch { - case exception: Throwable => - s"TYPED: ${exception.getClass.getSimpleName} :: " + - Option(exception.getMessage).getOrElse("").take(140) - } - val explicitSnapshotOutcome = - changelog("start-snapshot-id", snapshots.head.toString, 5) - val beforeHistoryOutcome = - changelog( - "start-timestamp", - (firstTimestamp.getTime - 1000).toString, - 5) - val middleHistoryOutcome = - changelog( - "start-timestamp", - (middleTimestamp.getTime - 1).toString, - 2) - - println(s"DIAG cdc.explicitExpiredId: $explicitSnapshotOutcome") - println(s"DIAG cdc.tsBeforeHistory: $beforeHistoryOutcome") - println(s"DIAG cdc.tsMidExpired: $middleHistoryOutcome") - Seq( - "explicitId" -> explicitSnapshotOutcome, - "tsBeforeHistory" -> beforeHistoryOutcome, - "tsMidExpired" -> middleHistoryOutcome).foreach { - case (label, outcome) => - assert( - !outcome.startsWith("FULL"), - s"expired-lineage changelog returned full truth for $label") - assert( - !outcome.toLowerCase.contains("expir"), - s"expired-lineage message now names expiration for $label") - } - } - - /** - * The hazards a reader or a consumer meets when maintenance lands underneath it. Every case starts from three seed - * rows in a copy-on-write table in the given file format. - */ - def hazardReaderCases(format: String): List[Plan.Case] = { - val basePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => cowCreate(table, format))() - .insert(3)()) - - List( - hazardStreamExpiredCheckpointCase(format, basePreparation), - hazardCdcExpiredRangeCase(basePreparation)) - } - - /** - * An explicit-column INSERT that worked before ADD COLUMN is rejected afterward, with an error naming the new column. - */ - private def hazardAddColumnBreaksWritersCase( - basePreparation: TablePreparation[CoreTable.type]): Plan.Case = - basePreparation.test("hazard.addColumn.breaksWriters") { table => - val allColumns = - Core.tableColumns.map(_.columnName).mkString(", ") - val writerStatement = - s"INSERT INTO ${table.name} ($allColumns) VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')" - table.spark.sql(writerStatement) - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "4", - "explicit-column writer should work before schema evolution") - - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - val exception = Check.intercept[AnalysisException]( - table.spark.sql(writerStatement)) - assert( - exception.getMessage.contains("extra_col") && - (exception.getMessage.contains("CANNOT_FIND_DATA") || - exception.getMessage.toLowerCase.contains("cannot find data")), - "pre-evolution explicit-column writer should fail after ADD COLUMN") - } - - /** - * The hazard an explicit-column writer meets after a column is added. The case starts from three seed rows in a - * copy-on-write table in the given file format. - */ - def hazardWriterCases(format: String): List[Plan.Case] = { - val basePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => cowCreate(table, format))() - .insert(3)()) - - List( - hazardAddColumnBreaksWritersCase(basePreparation)) - } - - /** - * While a table is REST-locked, an expire_snapshots call is rejected and snapshots keep accumulating. After the lock - * is deleted, expire_snapshots succeeds and the snapshot count drops, so the lock blocks every maintenance commit - * while it is held. - */ - def hazardLockStarvesMaintenance(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_lockmaint" - val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) - spark.sql(s"DROP TABLE IF EXISTS $table") - spark.sql(coreCreateParquet(table)) - spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") - spark.sql(s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - try { - val (lockStatus, lockBody) = Rest.post(ctx, s"/v1/databases/$db/tables/$tbl/lock", """{"locked":true}""") - assert(lockStatus >= 200 && lockStatus < 300, s"lock POST failed: $lockStatus $lockBody") - val snapsBefore = spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) - val e = Check.intercept[Exception](spark.sql( - s"CALL openhouse.system.expire_snapshots(table => '${table.stripPrefix("openhouse.")}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)")) - assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(_.toLowerCase.contains("locked"))), - s"expected LOCKED rejection for the maintenance commit: ${e.getClass.getName} ${Option(e.getMessage).getOrElse("").take(180)}") - spark.sql(s"REFRESH TABLE $table") - val snapsAfter = spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) - assert(snapsAfter == snapsBefore, "locked table must accumulate snapshots (maintenance starved)") - val (unlockStatus, _) = Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/lock") - assert(unlockStatus >= 200 && unlockStatus < 300, "unlock failed") - spark.sql(s"CALL openhouse.system.expire_snapshots(table => '${table.stripPrefix("openhouse.")}', older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)") - spark.sql(s"REFRESH TABLE $table") - assert(spark.sql(s"SELECT count(*) FROM $table.snapshots").collect()(0).getLong(0) < snapsBefore, - "maintenance must proceed after unlock") - } finally { - Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/lock") - spark.sql(s"DROP TABLE IF EXISTS $table") - } - } - - /** The hazard cases the embedded REST server drives. */ - val hazardContextCases: List[Plan.Case] = - List( - Plan.Case( - "hazard.lock.starvesMaintenance @ embedded", - hazardLockStarvesMaintenance)) - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala deleted file mode 100644 index c40993df4..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ImplementationPinScenarios.scala +++ /dev/null @@ -1,59 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// Pins on the physical form of what the OSS build writes. A case here fixes an implementation detail of the shipped -// write path, so a change to that detail shows up as a failing case. The behavior a case pins is an artifact of how OSS -// is wired, not a documented product feature. -trait ImplementationPinScenarios extends ScenarioKit { - import Rows._ - - /** - * A data file's Parquet footer magic bytes are the plaintext PAR1 marker, confirming OSS writes table data in - * plaintext. OpenHouse delegates table-data encryption to an external KMS plugin and the OSS build wires no - * KeyManagementClient into the catalog, so tables use the default PlaintextEncryptionManager. A Parquet footer reads - * PAR1 for plaintext and PARE under modular encryption regardless of compression, so that magic value settles which - * path wrote the file. - */ - private def surfacePinDataPlaintextCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("surface.pin.dataPlaintext") { table => - val dataFilePath = table.spark - .sql(s"SELECT file_path FROM ${table.name}.data_files LIMIT 1") - .collect()(0) - .getString(0) - .stripPrefix("file:") - val bytes = java.nio.file.Files.readAllBytes( - java.nio.file.Paths.get(dataFilePath)) - - assert( - bytes.length >= 8, - s"data file is too small to inspect: ${bytes.length} bytes") - val footerMagic = new String(bytes.takeRight(4), "US-ASCII") - assert( - footerMagic == "PAR1", - s"expected plaintext Parquet footer PAR1, got $footerMagic") - } - - /** The encryption pin, starting from three seed rows in a parquet table. */ - lazy val encryptionPinCases: List[Plan.Case] = { - val preparation = TablePreparation( - "parquet", - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "TBLPROPERTIES ('write.format.default'='parquet')")() - .insert(3)()) - - List( - surfacePinDataPlaintextCase(preparation)) - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/IncrementalReadScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/IncrementalReadScenarios.scala new file mode 100644 index 000000000..baa8bcc4c --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/IncrementalReadScenarios.scala @@ -0,0 +1,93 @@ +package harness + +/** + * Incremental read: a scan bounded by a start and an end snapshot returns the rows the snapshots in that range + * appended, and nothing else. + * + * Operations: an incremental scan across an append, across a row-level DELETE, across an INSERT OVERWRITE that only + * removes rows, across an UPDATE, and across the second commit of a two-snapshot history. + * + * Preparation axes: in each of the two columnar formats, the standard seeded core table for the four + * operation-bounded scans, and the two-snapshot core table for the scan between the two seeded commits. + * + * Case families: five families contributing 10 cases. + */ +trait IncrementalReadScenarios extends ScenarioKit { + + /** Every incremental-read case, one file format at a time. */ + lazy val incrementalReadCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + incrementalCase( + preparedStandardTable(format), + "incrementalRead.append", + table => + s"INSERT INTO $table VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')", + 1), + incrementalCase( + preparedStandardTable(format), + "incrementalRead.delete", + table => s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1", + 0), + incrementalCase( + preparedStandardTable(format), + "incrementalRead.overwrite", + table => + s"INSERT OVERWRITE $table SELECT * FROM $table " + + s"WHERE ${Core.long0.columnName} <= 2", + 0), + incrementalCase( + preparedStandardTable(format), + "incrementalRead.update", + table => + s"UPDATE $table SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2", + 0), + betweenSnapshotsCase(preparedTwoSnapshotTable(format))) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** The number of rows an incremental scan between the two snapshot IDs returns. */ + private def incrementalRowCount( + table: PreparedTable[CoreTable.type], + startSnapshotId: Long, + endSnapshotId: Long): Long = + table.spark.read + .format("iceberg") + .option("start-snapshot-id", startSnapshotId) + .option("end-snapshot-id", endSnapshotId) + .load(table.name) + .count() + + /** + * Running the statement against a seeded table and scanning from the seed snapshot to the snapshot the statement + * committed returns exactly `expectedRowCount` rows. + */ + private def incrementalCase( + preparation: TablePreparation[CoreTable.type], + caseName: String, + statement: String => String, + expectedRowCount: Long): Plan.Case = + preparation.test(caseName) { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql(statement(table.name)) + val currentSnapshotId = snapshotIds(table.spark, table.name).last + val addedRowCount = incrementalRowCount(table, seedSnapshotId, currentSnapshotId) + + assert( + addedRowCount == expectedRowCount, + s"$caseName returned $addedRowCount rows, expected $expectedRowCount") + } + + /** An incremental read spanning both snapshots of the two-snapshot table returns the 2 rows the second one added. */ + private def betweenSnapshotsCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("incrementalRead.betweenSnapshots") { table => + val snapshots = snapshotIds(table.spark, table.name) + val addedRowCount = incrementalRowCount(table, snapshots(0), snapshots(1)) + + assert(addedRowCount == 2, s"the second commit added 2 rows, scan returned $addedRowCount") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala deleted file mode 100644 index 1653ca9ca..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/InteractionScenarios.scala +++ /dev/null @@ -1,219 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The standard interaction families. Each case composes two table operations, so it shows how a DDL change, a snapshot -// reference, a maintenance procedure and a property setting behave against each other on a plain copy-on-write table. -// The cases run on parquet and orc. -trait InteractionScenarios extends ScenarioKit { - import Rows._ - - /** - * After ADD COLUMN and an insert into the new column, time travel to the pre-DDL snapshot reads the old schema with 3 - * rows, while a current read sees the new column. - */ - private def interactDdlTtAfterAddColumnCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("interact.ddl.ttAfterAddColumn") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - val currentColumns = table.spark - .sql(s"SELECT * FROM ${table.name} LIMIT 1") - .columns - .toSeq - val historicalColumns = table.spark - .sql( - s"SELECT * FROM ${table.name} " + - s"VERSION AS OF $seedSnapshotId LIMIT 1") - .columns - .toSeq - val historicalRowCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF $seedSnapshotId") - .collect()(0) - .getLong(0) - - assert( - currentColumns.contains("extra_col"), - s"current read is missing the evolved column: $currentColumns") - assert( - !historicalColumns.contains("extra_col") && - historicalColumns.size == Core.tableColumns.size, - s"time travel should use the snapshot schema: $historicalColumns") - assert( - historicalRowCount == 3, - s"pre-DDL snapshot should contain 3 rows, got $historicalRowCount") - } - - /** - * Rolling back to the pre-DDL snapshot after ADD COLUMN and an insert keeps the evolved schema, restores 3 rows - * reading null for the new column, and the table still accepts writes into that column. - */ - private def interactDdlRestoreAfterAddColumnCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("interact.ddl.restoreAfterAddColumn") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $seedSnapshotId)") - val currentColumns = table.spark - .sql(s"SELECT * FROM ${table.name} LIMIT 1") - .columns - .toSeq - val currentRowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val nonNullEvolvedValueCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - "WHERE extra_col IS NOT NULL") - .collect()(0) - .getLong(0) - - assert( - currentColumns.contains("extra_col"), - s"rollback should retain the evolved schema: $currentColumns") - assert( - currentRowCount == 3, - s"rollback should restore 3 rows, got $currentRowCount") - assert( - nonNullEvolvedValueCount == 0, - "rolled-back rows should read the evolved column as null") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert10") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 4, - "the rolled-back table should accept evolved-schema writes") - } - - /** - * DROP COLUMN on a column that holds data is rejected, the column's data remains readable, and the table remains - * writable. - */ - private def interactDdlDropColAfterDataCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("interact.ddl.dropColAfterData") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} DROP COLUMN extra_col")) - - assert( - exception.getMessage.contains("not found in newSchema"), - s"drop rejection message changed: ${exception.getMessage.take(200)}") - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} WHERE extra_col = 42") - .collect()(0) - .getLong(0) == 1, - "rejected drop should leave the column data readable") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert10") - assert( - table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) == 5, - "rejected drop should leave the table writable") - } - - /** The DDL interactions. Every case starts from three seed rows in a table in the given file format. */ - def interactionDdlCases(format: String): List[Plan.Case] = { - val preparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - - List( - interactDdlTtAfterAddColumnCase(preparation), - interactDdlRestoreAfterAddColumnCase(preparation), - interactDdlDropColAfterDataCase(preparation)) - } - - /** - * Compacting a table after an ADD COLUMN and inserts into the new column preserves all rows, the new column's - * non-null values, and null for rows written before the column was added. - */ - private def interactMaintCompactEvolvedCase( - basePreparation: TablePreparation[CoreTable.type]): Plan.Case = - basePreparation.test("interact.maint.compactEvolved") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert10") - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}')") - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - val evolvedValueCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - "WHERE extra_col IN (42, 43)") - .collect()(0) - .getLong(0) - val nullValueCount = table.spark - .sql( - s"SELECT count(*) FROM ${table.name} WHERE extra_col IS NULL") - .collect()(0) - .getLong(0) - - assert( - rowCount == 5, - s"compaction should preserve 5 rows, got $rowCount") - assert( - evolvedValueCount == 2, - s"compaction should preserve two evolved values, got $evolvedValueCount") - assert( - nullValueCount == 3, - s"pre-evolution rows should remain null, got $nullValueCount") - } - - /** The maintenance interactions. The case starts from three seed rows in a table in the given file format. */ - def interactionMiscellaneousCases( - format: String): List[Plan.Case] = { - val basePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - - List( - interactMaintCompactEvolvedCase(basePreparation)) - } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LocalRunner.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LocalRunner.scala new file mode 100644 index 000000000..e039784ec --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LocalRunner.scala @@ -0,0 +1,143 @@ +package harness + +import java.util.concurrent.{Callable, Executors, TimeUnit} +import scala.annotation.tailrec +import scala.util.control.NonFatal + +/** + * The local runner: the `harness.Main` launch class the `runOpenHouse` task starts, plus the retry policy it executes + * each case under. This file is compiled into the `local` source set only, so the published portable library carries + * the catalog and the framework without a run loop of its own. + */ +object Runner { + val MaxAttempts = 3 + + /** Runs a case, retrying only a transient-infrastructure failure. */ + def execute(testCase: Plan.Case, context: Ctx): (Outcome, Int) = { + @tailrec def attempt(attemptIndex: Int): (Outcome, Int) = { + val outcome = + try { + testCase.run(context.copy(spark = context.spark.newSession())) + Outcome.Passed + } + catch { case NonFatal(throwable) => Outcome.Failed(throwable) } + outcome match { + case failure: Outcome.Failed + if failure.retryable && attemptIndex + 1 < MaxAttempts => + attempt(attemptIndex + 1) + case terminal => + (terminal, attemptIndex + 1) + } + } + attempt(0) + } +} + +object Main { + def main(args: Array[String]): Unit = { + val (server, spark, restUri, restToken) = OpenHouseEnv.start() + var runFailure: Option[Throwable] = None + try { + spark.sparkContext.setLogLevel("ERROR") + val ctx = Ctx(spark, "openhouse.dbMatrix", restUri, restToken) + + // Each command-line argument is an include substring. A case runs when its ID contains every provided substring. + // An empty argument list selects the full catalog. + val filters = args.toList + val cases = Plan.cases.filter(testCase => + filters.forall(testCase.id.contains)) + + val header = + if (filters.isEmpty) { + "all cases" + } else { + s"filter ${filters.mkString(", ")} -> ${cases.size} cases" + } + println(s"\n=== delta-harness :: localized cases @ OpenHouse catalog ($header) ===\n") + + // Each case owns a fresh table. Worker tasks use separate Spark sessions over the shared Spark context, and + // results are printed in catalog order. + val parallelism = sys.env.get("HARNESS_PARALLELISM").map(_.toInt) + .getOrElse(math.max(1, Runtime.getRuntime.availableProcessors())) + println(s"parallelism: $parallelism worker sessions\n") + + def runOne(testCase: Plan.Case): (Plan.Case, (Outcome, Int)) = + testCase.embeddedSkipReason + .map(reason => s"embedded limitation: $reason") + .orElse(Plan.bugReason(testCase)) match { + case Some(reason) => + (testCase, (Outcome.Skipped(reason): Outcome, 0)) + case None => + (testCase, Runner.execute(testCase, ctx)) + } + + val results = + if (parallelism <= 1) { + cases.map(runOne) + } else { + val pool = Executors.newFixedThreadPool(parallelism) + try { + val futures = cases.map(testCase => + pool.submit( + new Callable[(Plan.Case, (Outcome, Int))] { + def call(): (Plan.Case, (Outcome, Int)) = runOne(testCase) + })) + futures.map(_.get(60, TimeUnit.MINUTES)) + } finally { + pool.shutdownNow() + } + } + + results.foreach { case (testCase, (outcome, attempts)) => + val note = outcome match { + case failure: Outcome.Failed => + s" (${failure.reason}${if (failure.retryable) " [retryable]" else ""})" + case Outcome.Skipped(reason) => + s" ($reason)" + case Outcome.Passed => + "" + } + println(f"${outcome.label}%-4s ${testCase.id}%-52s try=$attempts$note") + } + + val failed = + results.count { case (_, (outcome, _)) => outcome.isInstanceOf[Outcome.Failed] } + val skipped = + results.count { case (_, (outcome, _)) => outcome.isInstanceOf[Outcome.Skipped] } + val passed = results.size - failed - skipped + println(f"\n$passed passed, $skipped skipped, $failed failed (${results.size} cases)") + + if (failed > 0 || passed == 0) { + throw new AssertionError( + s"delta harness finished with $passed passed, $skipped skipped, and $failed failed cases") + } + } catch { + case failure: Throwable => + runFailure = Some(failure) + throw failure + } finally { + val cleanupFailures = + List[() => Unit]( + () => spark.stop(), + () => server.stop()) + .flatMap { cleanup => + try { + cleanup() + None + } catch { + case failure: Throwable => Some(failure) + } + } + + runFailure match { + case Some(failure) => + cleanupFailures.foreach(failure.addSuppressed) + case None => + cleanupFailures.headOption.foreach { failure => + cleanupFailures.drop(1).foreach(failure.addSuppressed) + throw failure + } + } + } + } +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LockingScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LockingScenarios.scala new file mode 100644 index 000000000..d8f1ec046 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LockingScenarios.scala @@ -0,0 +1,114 @@ +package harness + +/** + * Table locking: while a table carries a REST lock, the catalog rejects the commits that would change it, and + * deleting the lock lets them through again. + * + * Operations: POST the lock endpoint, then run an UPDATE and an expire_snapshots call against the locked table, then + * DELETE the lock and run each of them again. The lock endpoint has no SQL surface, so both cases drive it over HTTP + * against the embedded server, which runs the same TablesController and TablesServiceImpl as production. Both cases + * hold the lock through the shared lock boundary, which checks every lock and release response and releases the lock + * once, whichever way the case ends. + * + * Preparation axes: each case builds its own parquet core table and seeds it directly, because the REST path + * addresses the table by its database and table name. + * + * Case families: two families contributing 2 cases. + */ +trait LockingScenarios extends ScenarioKit { + + /** The lock cases, each driven over HTTP against the embedded server. */ + lazy val lockingCases: List[Plan.Case] = + List( + Plan.Case("lock.enforcement @ embedded", lockEnforcement), + Plan.Case("lock.starvesMaintenance @ embedded", lockStarvesMaintenance)) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * POSTing a table lock causes a following Spark UPDATE to be rejected server-side with LOCKED_TABLE_OPERATION, and + * DELETEing the lock lets a later UPDATE apply. + */ + private def lockEnforcement(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = TableTest.nextQualifiedTableName(ctx.namespace) + val Array(database, tableName) = table.stripPrefix("openhouse.").split("\\.", 2) + + withOwnedTable(spark.sql(_), table)(spark.sql(coreCreate(table, "parquet"))) { + spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, standardSeedRowCount)}") + withTableLock(lockRequest(ctx, database, tableName), unlockRequest(ctx, database, tableName)) { + releaseLock => + val lockedFailure = Check.intercept[Exception]( + spark.sql( + s"UPDATE $table SET ${Core.string0.columnName} = 'locked-write' " + + s"WHERE ${Core.long0.columnName} = 1")) + assert( + Exceptions.causeChain(lockedFailure).exists(cause => + Option(cause.getMessage).exists(_.toLowerCase.contains("locked"))), + s"expected a locked-table rejection, got: ${lockedFailure.getMessage.take(200)}") + + releaseLock() + spark.sql( + s"UPDATE $table SET ${Core.string0.columnName} = 'unlocked-write' " + + s"WHERE ${Core.long0.columnName} = 1") + assert( + countOf( + spark, + s"SELECT count(*) FROM $table WHERE ${Core.string0.columnName} = 'unlocked-write'") == "1", + "post-unlock update did not apply") + } + } + } + + /** + * While a table is REST-locked, an expire_snapshots call is rejected and snapshots keep accumulating. After the lock + * is deleted, expire_snapshots succeeds and the snapshot count drops, so the lock holds off every maintenance commit + * for as long as it is held. + */ + private def lockStarvesMaintenance(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = TableTest.nextQualifiedTableName(ctx.namespace) + val Array(database, tableName) = table.stripPrefix("openhouse.").split("\\.", 2) + val expireSnapshots = + s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)" + + withOwnedTable(spark.sql(_), table)(spark.sql(coreCreate(table, "parquet"))) { + spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, standardSeedRowCount)}") + spark.sql( + s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + withTableLock(lockRequest(ctx, database, tableName), unlockRequest(ctx, database, tableName)) { + releaseLock => + val snapshotsBefore = countOf(spark, s"SELECT count(*) FROM $table.snapshots") + + val lockedFailure = Check.intercept[Exception](spark.sql(expireSnapshots)) + assert( + Exceptions.causeChain(lockedFailure).exists(cause => + Option(cause.getMessage).exists(_.toLowerCase.contains("locked"))), + "expected a LOCKED rejection for the maintenance commit: " + + s"${lockedFailure.getClass.getName} " + + Option(lockedFailure.getMessage).getOrElse("").take(180)) + spark.sql(s"REFRESH TABLE $table") + assert( + countOf(spark, s"SELECT count(*) FROM $table.snapshots") == snapshotsBefore, + "a locked table keeps every snapshot it holds") + + releaseLock() + spark.sql(expireSnapshots) + spark.sql(s"REFRESH TABLE $table") + assert( + countOf(spark, s"SELECT count(*) FROM $table.snapshots").toLong < snapshotsBefore.toLong, + "maintenance must proceed after unlock") + } + } + } + + /** The POST that takes the REST lock on the named table. */ + private def lockRequest(ctx: Ctx, database: String, tableName: String): () => (Int, String) = + () => Rest.post(ctx, s"/v1/databases/$database/tables/$tableName/lock", """{"locked":true}""") + + /** The DELETE that releases the REST lock on the named table. */ + private def unlockRequest(ctx: Ctx, database: String, tableName: String): () => (Int, String) = + () => Rest.delete(ctx, s"/v1/databases/$database/tables/$tableName/lock") + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala deleted file mode 100644 index 7bb86d40c..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintControlScenarios.scala +++ /dev/null @@ -1,240 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -trait MaintControlScenarios extends ScenarioKit { - import Rows._ - - /** - * A five-row table across two snapshots in the given file format: a 3-row seed commit, then a 2-row insert committed - * at a later timestamp. Time travel, restore and maintenance all start from this state. - */ - private def twoSnapshotPreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation(format, coreTwoSnapshots(format)) - - /** - * VERSION AS OF the first snapshot ID reads the 3 rows the seed commit wrote, and VERSION AS OF the second reads all - * 5 rows. - */ - private def timeTravelVersionAsOfCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("timeTravel.versionAsOf") { table => - val snapshots = snapshotIds(table.spark, table.name) - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF ${snapshots(0)}") - .collect()(0) - .getLong(0) == 3) - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"VERSION AS OF ${snapshots(1)}") - .collect()(0) - .getLong(0) == 5) - } - - /** TIMESTAMP AS OF the first commit's time reads the 3 rows that commit wrote. */ - private def timeTravelTimestampAsOfCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("timeTravel.timestampAsOf") { table => - val firstCommitTimestamp = table.spark - .sql( - s"SELECT CAST(committed_at AS STRING) FROM ${table.name}.snapshots " + - "ORDER BY committed_at LIMIT 1") - .collect()(0) - .getString(0) - - assert( - table.spark - .sql( - s"SELECT count(*) FROM ${table.name} " + - s"TIMESTAMP AS OF '$firstCommitTimestamp'") - .collect()(0) - .getLong(0) == 3) - } - - /** - * The snapshots and history metadata tables each report the table's 2 snapshots, and the files and manifests metadata - * tables report at least 1 row. - */ - private def timeTravelMetadataTablesCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("timeTravel.metadataTables") { table => - def metadataRowCount(metadataTable: String): Long = - table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.$metadataTable") - .collect()(0) - .getLong(0) - - assert(metadataRowCount("snapshots") == 2) - assert(metadataRowCount("history") == 2) - assert( - metadataRowCount("files") >= 1 && - metadataRowCount("manifests") >= 1) - } - - /** An incremental read spanning both snapshots returns the 2 rows the second commit added. */ - private def timeTravelIncrementalReadCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("timeTravel.incrementalRead") { table => - val snapshots = snapshotIds(table.spark, table.name) - val addedRowCount = table.spark.read - .format("iceberg") - .option("start-snapshot-id", snapshots(0)) - .option("end-snapshot-id", snapshots(1)) - .load(table.name) - .count() - - assert(addedRowCount == 2) - } - - /** Time travel across both snapshots of the two-snapshot table, in parquet and in orc. */ - val timeTravelCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - val preparation = twoSnapshotPreparation(format) - - List( - timeTravelVersionAsOfCase(preparation), - timeTravelTimestampAsOfCase(preparation), - timeTravelMetadataTablesCase(preparation), - timeTravelIncrementalReadCase(preparation)) - } - - /** rollback_to_snapshot to the first snapshot restores the 3 rows the seed commit wrote. */ - private def restoreRollbackToSnapshotCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("restore.rollbackToSnapshot") { table => - val firstSnapshotId = - snapshotIds(table.spark, table.name).head - - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $firstSnapshotId)") - - assert(table.rows.size == 3) - } - - /** set_current_snapshot to the first snapshot restores the 3 rows the seed commit wrote. */ - private def restoreSetCurrentSnapshotCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("restore.setCurrentSnapshot") { table => - val firstSnapshotId = - snapshotIds(table.spark, table.name).head - - table.spark.sql( - "CALL openhouse.system.set_current_snapshot(" + - s"'${catalogRelative(table.name)}', $firstSnapshotId)") - - assert(table.rows.size == 3) - } - - /** Restore back to the seed snapshot of the two-snapshot table, in parquet and in orc. */ - val restoreRollbackCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - val preparation = twoSnapshotPreparation(format) - - List( - restoreRollbackToSnapshotCase(preparation), - restoreSetCurrentSnapshotCase(preparation)) - } - - /** expire_snapshots with retain_last=1 removes the seed snapshot and leaves all 5 current rows unchanged. */ - private def maintenanceExpireSnapshotsCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("maintenance.expireSnapshots") { table => - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - assert( - table.rows.size == 5, - "expire_snapshots changed the current data") - assert( - table.snapshotCount < table.preparedSnapshotCount, - "expire_snapshots did not remove a snapshot: " + - s"${table.preparedSnapshotCount} -> ${table.snapshotCount}") - } - - /** rewrite_data_files compacts the data files and leaves all 5 rows unchanged. */ - private def maintenanceRewriteDataFilesCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("maintenance.rewriteDataFiles") { table => - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}')") - - assert(table.rows.size == 5, "compaction changed rows") - } - - /** remove_orphan_files leaves all 5 rows unchanged. */ - private def maintenanceRemoveOrphanFilesCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("maintenance.removeOrphanFiles") { table => - table.spark.sql( - "CALL openhouse.system.remove_orphan_files(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2020-01-01 00:00:00')") - - assert(table.rows.size == 5, "orphan removal changed rows") - } - - /** The maintenance procedures run over the two-snapshot table, in parquet and in orc. */ - val maintenanceCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - val preparation = twoSnapshotPreparation(format) - - List( - maintenanceExpireSnapshotsCase(preparation), - maintenanceRewriteDataFilesCase(preparation), - maintenanceRemoveOrphanFilesCase(preparation)) - } - - /** - * POSTing a table lock causes a following Spark UPDATE to be rejected server-side with LOCKED_TABLE_OPERATION, and - * DELETEing the lock lets a later UPDATE apply. The lock endpoint has no SQL surface, so the case drives it over HTTP - * against the embedded server, which runs the same TablesController and TablesServiceImpl as production. - */ - def controlLockEnforcement(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = s"${ctx.namespace}.t_lock" - val Array(db, tbl) = table.stripPrefix("openhouse.").split("\\.", 2) - spark.sql(s"DROP TABLE IF EXISTS $table") - spark.sql(coreCreateParquet(table)) - spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, 3)}") - try { - val (lockStatus, lockBody) = Rest.post(ctx, s"/v1/databases/$db/tables/$tbl/lock", """{"locked":true}""") - assert(lockStatus >= 200 && lockStatus < 300, s"lock POST failed: $lockStatus $lockBody") - val e = Check.intercept[Exception](spark.sql( - s"UPDATE $table SET ${Core.string0.columnName} = 'locked-write' WHERE ${Core.long0.columnName} = 1")) - assert(Exceptions.causeChain(e).exists(t => Option(t.getMessage).exists(_.toLowerCase.contains("locked"))), - s"expected a locked-table rejection, got: ${e.getMessage.take(200)}") - val (unlockStatus, unlockBody) = Rest.delete(ctx, s"/v1/databases/$db/tables/$tbl/lock") - assert(unlockStatus >= 200 && unlockStatus < 300, s"unlock DELETE failed: $unlockStatus $unlockBody") - spark.sql(s"UPDATE $table SET ${Core.string0.columnName} = 'unlocked-write' WHERE ${Core.long0.columnName} = 1") - assert(spark.sql(s"SELECT count(*) FROM $table WHERE ${Core.string0.columnName} = 'unlocked-write'").collect()(0).getLong(0) == 1, - "post-unlock update did not apply") - } finally spark.sql(s"DROP TABLE IF EXISTS $table") - } - - /** The control-plane cases, each driven over HTTP against the embedded server. */ - val controlPlaneCases: List[Plan.Case] = - List( - Plan.Case( - "control.lock.enforcement @ embedded", - controlLockEnforcement)) - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintenanceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintenanceScenarios.scala new file mode 100644 index 000000000..5ce5ae921 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintenanceScenarios.scala @@ -0,0 +1,169 @@ +package harness + +import java.nio.file.{Files, Paths} +import java.nio.file.attribute.FileTime + +/** + * Maintenance: the procedures that rewrite a table's files and metadata without changing the rows a reader sees. + * + * Operations: expire_snapshots down to the newest snapshot, rewrite_data_files over the table's data files, + * remove_orphan_files over the table's directory, rewrite_manifests over a fragmented manifest list, remove_orphan_ + * files against a planted backdated stray file, and rewrite_data_files across an ADD COLUMN. + * + * Preparation axes: in each of the two columnar formats, the two-snapshot core table for the three procedures that run + * over an existing history, an unseeded core table for the manifest family, which fragments the manifest list itself, + * and the standard seeded core table for the planted-orphan and schema-evolution families. + * + * Case families: six families contributing 12 cases. + */ +trait MaintenanceScenarios extends ScenarioKit { + + /** Every maintenance case, one file format at a time. */ + lazy val maintenanceCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + expireSnapshotsCase(preparedTwoSnapshotTable(format)), + rewriteDataFilesCase(preparedTwoSnapshotTable(format)), + removeOrphanFilesCase(preparedTwoSnapshotTable(format)), + rewriteManifestsCase(preparedEmptyStandardTable(format)), + removeOrphanFilesPlantedCase(preparedStandardTable(format)), + rewriteDataFilesAfterAddColumnCase(preparedStandardTable(format))) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** expire_snapshots with retain_last=1 removes the seed snapshot and leaves all 5 current rows unchanged. */ + private def expireSnapshotsCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("maintenance.expireSnapshots") { table => + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + assert( + table.rows.size == 5, + "expire_snapshots changed the current data") + assert( + table.snapshotCount < table.preparedSnapshotCount, + "expire_snapshots did not remove a snapshot: " + + s"${table.preparedSnapshotCount} -> ${table.snapshotCount}") + } + + /** rewrite_data_files compacts the data files and leaves all 5 rows unchanged. */ + private def rewriteDataFilesCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("maintenance.rewriteDataFiles") { table => + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}')") + + assert(table.rows.size == 5, "compaction changed rows") + } + + /** remove_orphan_files over a table with no stray files leaves all 5 rows unchanged. */ + private def removeOrphanFilesCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("maintenance.removeOrphanFiles") { table => + table.spark.sql( + "CALL openhouse.system.remove_orphan_files(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2020-01-01 00:00:00')") + + assert(table.rows.size == 5, "orphan removal changed rows") + } + + /** + * After 5 single-row inserts fragment the manifest list, rewrite_manifests compacts it to fewer manifests while + * preserving all 5 rows. + */ + private def rewriteManifestsCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("maintenance.rewriteManifests") { table => + (1 to 5).foreach(index => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + coreRow(index, s"r$index"))) + val manifestCountBefore = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.manifests") + .collect()(0) + .getLong(0) + table.spark.sql( + "CALL openhouse.system.rewrite_manifests(" + + s"table => '${catalogRelative(table.name)}', " + + "use_caching => false)") + val manifestCountAfter = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.manifests") + .collect()(0) + .getLong(0) + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "5", + "rewrite_manifests should preserve the five rows") + assert( + manifestCountBefore >= 2 && + manifestCountAfter < manifestCountBefore, + "rewrite_manifests should compact the manifest set: " + + s"before=$manifestCountBefore after=$manifestCountAfter") + } + + /** + * remove_orphan_files deletes a planted, backdated stray file next to a real data file while the table's 3 live rows + * remain intact. + */ + private def removeOrphanFilesPlantedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("maintenance.removeOrphanFiles.planted") { table => + val dataFile = table.spark + .sql(s"SELECT file_path FROM ${table.name}.files LIMIT 1") + .collect()(0) + .getString(0) + .stripPrefix("file:") + val orphanFile = Paths + .get(dataFile) + .getParent + .resolve(s"${table.name.split('.').last}_orphan.parquet") + Files.write(orphanFile, "not-a-real-parquet".getBytes) + Files.setLastModifiedTime(orphanFile, FileTime.fromMillis(1546300800000L)) + + table.spark.sql( + "CALL openhouse.system.remove_orphan_files(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2020-01-01 00:00:00')") + assert( + Files.notExists(orphanFile), + "remove_orphan_files should delete the planted orphan") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", + "remove_orphan_files should preserve live data") + } + + /** + * Compacting a table after an ADD COLUMN and inserts into the new column preserves all rows, the new column's + * non-null values, and null for rows written before the column was added. + */ + private def rewriteDataFilesAfterAddColumnCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("maintenance.rewriteDataFiles.afterAddColumn") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert10") + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}')") + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "5", + "compaction should preserve 5 rows") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} WHERE extra_col IN (42, 43)") == "2", + "compaction should preserve both evolved values") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} WHERE extra_col IS NULL") == "3", + "pre-evolution rows should remain null") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MetadataTableScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MetadataTableScenarios.scala new file mode 100644 index 000000000..4684b9943 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MetadataTableScenarios.scala @@ -0,0 +1,96 @@ +package harness + +/** + * Metadata tables: the hidden metadata columns a scan exposes and the Iceberg metadata tables the catalog serves + * alongside every table. + * + * Operations: select the hidden _file, _pos, _spec_id and _partition columns; query every metadata table the catalog + * serves (entries, files, manifests, snapshots, history, refs, partitions, metadata_log_entries, data_files and the + * all_* variants); and read the snapshot, history, files and manifests counts of a two-snapshot table. + * + * Preparation axes: in each of the two columnar formats, the standard seeded core table for the hidden-column family + * and the two-snapshot core table for the two families that count metadata rows. + * + * Case families: three families contributing 6 cases. + */ +trait MetadataTableScenarios extends ScenarioKit { + + /** Every metadata-table case, one file format at a time. */ + lazy val metadataTableCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + hiddenColumnsCase(preparedStandardTable(format)), + tableSweepCase(preparedTwoSnapshotTable(format)), + snapshotAndHistoryCase(preparedTwoSnapshotTable(format))) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * Selecting the hidden metadata columns _file, _pos, _spec_id and _partition returns one row per seed row, each with + * a populated file path and a non-negative position. + */ + private def hiddenColumnsCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("metadata.hiddenColumns") { table => + val rows = table.spark + .sql(s"SELECT _file, _pos, _spec_id, _partition FROM ${table.name}") + .collect() + .toSeq + + assert( + rows.size == 3, + s"hidden metadata columns should return 3 rows, got ${rows.size}") + assert( + rows.forall(row => Option(row.getString(0)).exists(_.nonEmpty)), + "_file should be populated for every row") + assert( + rows.forall(_.getLong(1) >= 0), + "_pos should be non-negative for every row") + } + + /** + * Every Iceberg metadata table is queryable without error, and the snapshots metadata table reports the table's 2 + * snapshots. + */ + private def tableSweepCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("metadata.tableSweep") { table => + val metadataTables = Seq( + "entries", + "files", + "manifests", + "snapshots", + "history", + "refs", + "partitions", + "metadata_log_entries", + "data_files", + "all_data_files", + "all_manifests", + "all_entries", + "all_files") + metadataTables.foreach { metadataTable => + table.spark.sql(s"SELECT count(*) FROM ${table.name}.`$metadataTable`").collect() + } + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}.snapshots") == "2", + "snapshot metadata should contain two snapshots") + } + + /** + * The snapshots and history metadata tables each report the table's 2 snapshots, and the files and manifests + * metadata tables each report at least 1 row. + */ + private def snapshotAndHistoryCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("metadata.snapshotAndHistory") { table => + def metadataRowCount(metadataTable: String): Long = + table.spark + .sql(s"SELECT count(*) FROM ${table.name}.$metadataTable") + .collect()(0) + .getLong(0) + + assert(metadataRowCount("snapshots") == 2) + assert(metadataRowCount("history") == 2) + assert(metadataRowCount("files") >= 1 && metadataRowCount("manifests") >= 1) + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NamespaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NamespaceScenarios.scala new file mode 100644 index 000000000..11dd401b6 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NamespaceScenarios.scala @@ -0,0 +1,48 @@ +package harness + +/** + * Namespaces: the catalog serves the databases it is configured with, and it rejects the statements that would create + * or drop one. + * + * Operations: CREATE NAMESPACE and DROP NAMESPACE. + * + * Preparation axes: the standard seeded core table in each of the two columnar formats, which gives each case a live + * catalog session and a table lifecycle. + * + * Case families: two families contributing 4 cases. + */ +trait NamespaceScenarios extends ScenarioKit { + + /** Every namespace case, one file format at a time. */ + lazy val namespaceCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + createRejectedCase(preparedStandardTable(format)), + dropRejectedCase(preparedStandardTable(format))) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** CREATE NAMESPACE is rejected with an UnsupportedOperationException naming the unsupported operation. */ + private def createRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("namespace.create.rejected") { table => + val exception = Check.intercept[UnsupportedOperationException]( + table.spark.sql("CREATE NAMESPACE openhouse.a_new_db")) + + assert( + exception.getMessage.contains("not supported"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + + /** DROP NAMESPACE is rejected with an UnsupportedOperationException naming the unsupported operation. */ + private def dropRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("namespace.drop.rejected") { table => + val exception = Check.intercept[UnsupportedOperationException]( + table.spark.sql("DROP NAMESPACE openhouse.dbMatrix")) + + assert( + exception.getMessage.contains("not supported"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala deleted file mode 100644 index 753e6d48b..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NegativeDdlScenarios.scala +++ /dev/null @@ -1,656 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -trait NegativeDdlScenarios extends ScenarioKit { - import Rows._ - - private val S = CoreTable.string0.columnName - - /** DELETE with a WHERE clause on a nonexistent column is rejected with an AnalysisException naming that column. */ - private def negativeNonExistentColumnCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("negative.nonExistentColumn") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"DELETE FROM ${table.name} WHERE no_such_column = 1")) - - assert(exception.getMessage.contains("no_such_column")) - } - - /** - * DELETE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an AnalysisException about determinism. - */ - private def negativeNonDeterministicDeleteCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("negative.nonDeterministicDelete") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"DELETE FROM ${table.name} WHERE rand() < 0.5")) - - assert( - exception.getMessage.toLowerCase.contains("deterministic")) - } - - /** - * UPDATE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an AnalysisException about determinism. - */ - private def negativeNonDeterministicUpdateCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("negative.nonDeterministicUpdate") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"UPDATE ${table.name} SET $S = 'x' WHERE rand() < 0.5")) - - assert( - exception.getMessage.toLowerCase.contains("deterministic")) - } - - /** - * INSERT INTO with too few values for the table's columns is rejected with an AnalysisException about the missing - * data columns. - */ - private def negativeInsertArityCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("negative.insertArity") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"INSERT INTO ${table.name} VALUES (CAST(1 AS BIGINT), 1)")) - - assert( - exception.getMessage.toLowerCase.contains( - "not enough data columns")) - } - - /** - * A MERGE whose UPDATE SET assigns the same target column twice is rejected with an AnalysisException about multiple - * assignments. - */ - private def negativeMergeConflictingUpdatesCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("negative.mergeConflictingUpdates") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"""MERGE INTO ${table.name} target USING ( - SELECT * FROM VALUES (CAST(2 AS BIGINT)) AS source($L) - ) source - ON target.$L = source.$L - WHEN MATCHED THEN UPDATE - SET target.$S = 'a', target.$S = 'b'""")) - - assert(exception.getMessage.contains("Multiple assignments")) - } - - /** - * A MERGE whose source has two rows matching the same target row fails with a cardinality-violation error naming the - * multi-row match. - */ - private def negativeMergeCardinalityViolationCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("negative.mergeCardinalityViolation") { table => - val exception = Check.intercept[Exception]( - table.spark.sql( - s"""MERGE INTO ${table.name} target USING ( - SELECT * FROM VALUES - (CAST(2 AS BIGINT), 'a'), - (CAST(2 AS BIGINT), 'b') - AS source($L, $S) - ) source - ON target.$L = source.$L - WHEN MATCHED THEN UPDATE SET target.$S = source.$S""")) - - assert( - Exceptions.causeChain(exception).exists { cause => - Option(cause.getMessage).exists( - _.contains("matched a single row from the target table")) - }, - "expected a MERGE cardinality-violation message, got: " + - exception.getMessage) - } - - /** - * CREATE TABLE PARTITIONED BY a nonexistent column is rejected with an AnalysisException naming that column, and no - * scratch table is left behind. - */ - private def negativePartitionByNonExistentCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("negative.partitionByNonExistent") { table => - val scratchTable = table.name + "_x" - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"CREATE TABLE $scratchTable ($columnDefinitions) " + - s"USING $dataSource PARTITIONED BY (no_such_column) " + - s"TBLPROPERTIES ('write.format.default'='${preparation.label}')")) - - table.spark.sql(s"DROP TABLE IF EXISTS $scratchTable") - assert(exception.getMessage.contains("no_such_column")) - } - - /** The rejected DML statements, on the preparedCoreFormats preparations. */ - val negativeCases: List[Plan.Case] = - preparedCoreFormats.flatMap { preparation => - List( - negativeNonExistentColumnCase(preparation), - negativeNonDeterministicDeleteCase(preparation), - negativeNonDeterministicUpdateCase(preparation), - negativeInsertArityCase(preparation), - negativeMergeConflictingUpdatesCase(preparation), - negativeMergeCardinalityViolationCase(preparation), - negativePartitionByNonExistentCase(preparation)) - } - - /** ALTER TABLE DROP COLUMN is rejected with a BadRequestException naming the column that would be dropped. */ - private def ddlNegDropColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.neg.dropColumn") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} DROP COLUMN ${Core.int0.columnName}")) - - assert( - exception.getMessage.contains("not found in newSchema"), - s"unexpected message: ${exception.getMessage.take(160)}") - assert( - exception.getMessage.contains(Core.int0.columnName), - s"message should name the dropped column: ${exception.getMessage.take(160)}") - } - - /** - * ALTER TABLE ALTER COLUMN to a narrower type (bigint to int) is rejected with an AnalysisException about the - * unsupported column change. - */ - private def ddlNegNarrowTypeCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.neg.narrowType") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.long0.columnName} TYPE int")) - - assert( - exception.getMessage.contains("NOT_SUPPORTED_CHANGE_COLUMN"), - s"unexpected message: ${exception.getMessage.take(160)}") - } - - /** - * ALTER TABLE ALTER COLUMN SET NOT NULL on a nullable column is rejected with an AnalysisException about the - * nullable-to-non-nullable change. - */ - private def ddlNegSetNotNullCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.neg.setNotNull") { table => - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.string0.columnName} SET NOT NULL")) - - assert( - exception.getMessage.contains("Cannot change nullable column to non-nullable"), - s"unexpected message: ${exception.getMessage.take(160)}") - } - - /** The rejected schema changes, on the preparedCoreFormats preparations. */ - val ddlNegativeCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => - List( - ddlNegDropColumnCase(preparation), - ddlNegNarrowTypeCase(preparation), - ddlNegSetNotNullCase(preparation)) - } - - /** SET TBLPROPERTIES adds a user property that reads back, and UNSET TBLPROPERTIES removes it. */ - private def ddlPropsUserRoundTripCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.props.userRoundTrip") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('my_key'='my_val')") - assert( - tableProps(table.spark, table.name).get("my_key").contains("my_val"), - "user prop not set") - - table.spark.sql(s"ALTER TABLE ${table.name} UNSET TBLPROPERTIES ('my_key')") - assert( - !tableProps(table.spark, table.name).contains("my_key"), - "user prop not removed") - } - - /** - * SET TBLPROPERTIES on the reserved openhouse.tableUUID property is rejected with a BadRequestException about the - * restriction. - */ - private def ddlPropsReservedOpenhouseCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.props.reservedOpenhouse") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + - "'openhouse.tableUUID'='deadbeef')")) - - assert( - exception.getMessage.toLowerCase.contains("restriction"), - s"msg: ${exception.getMessage.take(200)}") - } - - /** - * Even though format-version=1 was requested at creation, the table is forced to format-version=2 and remains - * writable. - */ - private def ddlPropsFormatVersionForcedCase( - formatVersionPreparation: TablePreparation[CoreTable.type]): Plan.Case = - formatVersionPreparation.test("ddl.props.formatVersionForced") { table => - val formatVersion = tableProps(table.spark, table.name).get("format-version") - - assert( - formatVersion.contains("2"), - s"expected forced format-version=2, got $formatVersion") - assert( - table.rows.size == 3, - "table not writable at the forced format-version") - } - - /** The write.metadata.previous-versions-max property requested at creation is honored and reads back as 7. */ - private def ddlPropsPreviousVersionsHonoredCase( - previousVersionsPreparation: TablePreparation[CoreTable.type]): Plan.Case = - previousVersionsPreparation.test("ddl.props.previousVersionsHonored") { table => - val previousVersions = - tableProps(table.spark, table.name).get("write.metadata.previous-versions-max") - - assert( - previousVersions.contains("7"), - s"expected previous-versions-max=7, got $previousVersions") - } - - /** - * The table-property cases. Two of them start from the preparedCoreFormats preparation for the file format, one from - * a table created with format-version=1 requested, and one from an unseeded table created with - * write.metadata.previous-versions-max=7. - */ - val ddlPropertyCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => - val format = preparation.label - val formatVersionPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'format-version'='1')")() - .insert(3)()) - val previousVersionsPreparation = TablePreparation( - format, - TableTest(Core).sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'write.metadata.previous-versions-max'='7')")()) - - List( - ddlPropsUserRoundTripCase(preparation), - ddlPropsReservedOpenhouseCase(preparation), - ddlPropsFormatVersionForcedCase(formatVersionPreparation), - ddlPropsPreviousVersionsHonoredCase(previousVersionsPreparation)) - } - - /** ALTER TABLE WRITE ORDERED BY a single column sets write.distribution-mode to range. */ - private def ddlSortOrderOrderedByCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.sortOrder.orderedBy") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} WRITE ORDERED BY ${Core.long0.columnName}") - - val distributionMode = - tableProps(table.spark, table.name).get("write.distribution-mode") - - assert( - distributionMode.contains("range"), - s"distribution-mode not range: $distributionMode") - } - - /** - * ALTER TABLE WRITE ORDERED BY multiple columns sets range distribution and the table remains writable, growing from - * 3 to 5 rows after a follow-up insert. - */ - private def ddlSortOrderOrderedByMultiCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.sortOrder.orderedByMulti") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} WRITE ORDERED BY " + - s"${Core.string0.columnName} DESC NULLS FIRST, ${Core.long0.columnName}") - - assert( - tableProps(table.spark, table.name).get("write.distribution-mode").contains("range"), - "multi-col ordered-by should set range") - - table.spark.sql( - s"INSERT INTO ${table.name} ${RowGenerator.valuesClause(Core, 2)}") - - assert(table.rows.size == 5, "multi-col ordered write path failed") - } - - /** - * ALTER TABLE RENAME TO moves the table to the new name with its 3 rows intact, and the old name stops resolving. A - * second rename puts the table back under its original name, which teardown drops. - */ - private def ddlRenameTableCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.renameTable") { table => - val renamedTable = s"${table.name}_ren" - - table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $renamedTable") - assert( - table.spark.sql(s"SELECT count(*) FROM $renamedTable").collect()(0).getLong(0) == 3, - "renamed table lost rows") - Check.intercept[Exception]( - table.spark.sql(s"SELECT 1 FROM ${table.name} LIMIT 1")) - table.spark.sql(s"ALTER TABLE $renamedTable RENAME TO ${table.name}") - } - - /** ALTER TABLE RENAME TO a name that already exists is rejected with an error naming the conflict. */ - private def ddlRenameTableConflictCase( - preparation: TablePreparation[CoreTable.type], - format: String): Plan.Case = - preparation.test("ddl.renameTable.conflict") { table => - val conflictingTable = s"${table.name}_other" - - table.spark.sql(s"DROP TABLE IF EXISTS $conflictingTable") - table.spark.sql( - s"CREATE TABLE $conflictingTable ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')") - val exception = Check.intercept[WebClientResponseWithMessageException]( - table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $conflictingTable")) - - assert( - exception.getMessage.contains("already exists"), - s"msg: ${exception.getMessage.take(160)}") - table.spark.sql(s"DROP TABLE IF EXISTS $conflictingTable") - } - - /** - * CREATE NAMESPACE is rejected with an UnsupportedOperationException, since this catalog does not support creating - * namespaces. - */ - private def ddlNsCreateRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.ns.createRejected") { table => - val exception = Check.intercept[UnsupportedOperationException]( - table.spark.sql("CREATE NAMESPACE openhouse.a_new_db")) - - assert( - exception.getMessage.contains("not supported"), - s"msg: ${exception.getMessage.take(160)}") - } - - /** - * DROP NAMESPACE is rejected with an UnsupportedOperationException, since this catalog does not support dropping - * namespaces. - */ - private def ddlNsDropRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.ns.dropRejected") { table => - val exception = Check.intercept[UnsupportedOperationException]( - table.spark.sql("DROP NAMESPACE openhouse.dbMatrix")) - - assert( - exception.getMessage.contains("not supported"), - s"msg: ${exception.getMessage.take(160)}") - } - - /** The remaining DDL cases, on the preparedCoreFormats preparations. */ - val ddlMiscellaneousCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => - val format = preparation.label - - List( - ddlSortOrderOrderedByCase(preparation), - ddlSortOrderOrderedByMultiCase(preparation), - ddlRenameTableCase(preparation), - ddlRenameTableConflictCase(preparation, format), - ddlNsCreateRejectedCase(preparation), - ddlNsDropRejectedCase(preparation)) - } - - /** SET POLICY (SHARING=TRUE) records the sharing policy and the table remains queryable. */ - private def ddlPolicySharingCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.policy.sharing") { - table => - table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") - - val policies = tableProps(table.spark, table.name).getOrElse("policies", "") - - assert( - policies.toLowerCase.contains("true") || - policies.toLowerCase.contains("sharing"), - s"sharing policy not stored: $policies") - assert( - table.rows.size == 3, - "table not queryable after SET POLICY (SHARING)") - } - - /** SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20) records the history policy and the table remains queryable. */ - private def ddlPolicyHistoryCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.policy.history") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20)") - - val policies = tableProps(table.spark, table.name).getOrElse("policies", "") - - assert( - policies.contains("20") || policies.toLowerCase.contains("history"), - s"history policy not stored: $policies") - assert( - table.rows.size == 3, - "table not queryable after SET POLICY (HISTORY)") - } - - /** - * SET POLICY (REPLICATION) followed by UNSET POLICY (REPLICATION) leaves the table queryable with its 3 rows intact. - */ - private def ddlPolicyReplicationCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.policy.replication") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") - table.spark.sql( - s"ALTER TABLE ${table.name} UNSET POLICY (REPLICATION)") - - assert(table.rows.size == 3) - } - - /** - * SET POLICY (RETENTION = 30d ON COLUMN foo_col_date ...) records the retention policy and the table remains - * queryable. - */ - private def ddlPolicyRetentionCase( - retentionPreparation: TablePreparation[CoreTable.type]): Plan.Case = - retentionPreparation.test("ddl.policy.retention") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (" + - s"RETENTION = 30d ON COLUMN ${Core.date0.columnName} WHERE pattern = 'yyyy-MM-dd-HH')") - - val policies = tableProps(table.spark, table.name).getOrElse("policies", "") - - assert( - policies.toLowerCase.contains("retention") || policies.contains("30"), - s"retention policy not stored: $policies") - assert( - table.rows.size == 3, - "table not queryable after SET POLICY (RETENTION)") - } - - /** - * SET POLICY (HISTORY MAX_AGE=5D) exceeds the allowed range and is rejected with a BadRequestException stating the - * 1-to-3-day limit. - */ - private def ddlPolicyNegHistoryMaxAgeCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.policy.neg.historyMaxAge") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=5D)")) - - assert( - exception.getMessage.contains("max age must be between 1 to 3 days"), - s"msg: ${exception.getMessage.take(160)}") - } - - /** - * SET POLICY (HISTORY VERSIONS=200) exceeds the allowed range and is rejected with a BadRequestException stating the - * 2-to-100-version limit. - */ - private def ddlPolicyNegHistoryVersionsCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.policy.neg.historyVersions") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (HISTORY VERSIONS=200)")) - - assert( - exception.getMessage.contains("must be between 2 to 100 versions"), - s"msg: ${exception.getMessage.take(160)}") - } - - /** - * The table-policy cases. They start from the preparedCoreFormats preparation for the file format, except the - * retention case, which starts from three seed rows in a table partitioned by the date column. - */ - val ddlPolicyCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => - val format = preparation.label - val retentionPreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.date0.columnName}) " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - - List( - ddlPolicySharingCase(preparation), - ddlPolicyHistoryCase(preparation), - ddlPolicyReplicationCase(preparation), - ddlPolicyRetentionCase(retentionPreparation), - ddlPolicyNegHistoryMaxAgeCase(preparation), - ddlPolicyNegHistoryVersionsCase(preparation)) - } - - /** - * ALTER TABLE MODIFY COLUMN SET TAG = (PII) tags a column without masking or changing the values that queries return. - */ - private def ddlColTagCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.colTag") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} MODIFY COLUMN " + - s"${Core.string0.columnName} SET TAG = (PII)") - - val values = table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - s"ORDER BY ${Core.long0.columnName}") - .collect() - .toSeq - .map(_.getString(0)) - - assert( - values == Seq("row-1", "row-2", "row-3"), - s"SET TAG changed query results (should not mask): $values") - } - - /** - * GRANT SELECT on a table that is not marked shared is rejected with an IllegalArgumentException stating the table is - * not shared. - */ - private def ddlAclGrantUnsharedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.acl.grantUnshared") { table => - val exception = Check.intercept[IllegalArgumentException]( - table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC")) - - assert( - exception.getMessage.contains("is not a shared table"), - s"msg: ${exception.getMessage.take(160)}") - } - - /** - * On a shared table, GRANT SELECT TO PUBLIC makes SHOW GRANTS list SELECT for PUBLIC and the table stays queryable; - * REVOKE SELECT then removes that grant from SHOW GRANTS. - */ - private def ddlAclGrantSharedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation - .test("ddl.acl.grantShared") { table => - table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") - table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC") - - val grantsAfterGrant = table.spark - .sql(s"SHOW GRANTS ON TABLE ${table.name}") - .collect() - .map(row => (row.getString(0), row.getString(1))) - .toSet - assert( - grantsAfterGrant.contains(("SELECT", "PUBLIC")), - s"SHOW GRANTS did not include SELECT for PUBLIC: $grantsAfterGrant") - assert(table.rows.size == 3, "shared/granted table not queryable") - - table.spark.sql(s"REVOKE SELECT ON TABLE ${table.name} FROM PUBLIC") - val grantsAfterRevoke = table.spark - .sql(s"SHOW GRANTS ON TABLE ${table.name}") - .collect() - .map(row => (row.getString(0), row.getString(1))) - .toSet - assert( - !grantsAfterRevoke.contains(("SELECT", "PUBLIC")), - s"SHOW GRANTS retained SELECT for PUBLIC: $grantsAfterRevoke") - } - .copy(embeddedSkipReason = Some( - "The embedded test server has no OPA endpoint configured, so grantRole and " + - "listAclPolicies are no-ops that always report an empty ACL list. GRANT and REVOKE " + - "succeed without error, while SHOW GRANTS always returns an empty ACL list. The " + - "li-openhouse acceptance environment runs the assertions against its configured " + - "authorization service.")) - - /** - * The write.distribution-mode=none property requested at creation is honored and the table remains writable under it. - */ - private def ddlFeatureFlagDistributionModeCase( - distributionModePreparation: TablePreparation[CoreTable.type]): Plan.Case = - distributionModePreparation.test("ddl.featureFlag.distributionMode") { table => - val distributionMode = - tableProps(table.spark, table.name).get("write.distribution-mode") - - assert( - distributionMode.contains("none"), - s"distribution-mode not honored: $distributionMode") - assert( - table.rows.size == 3, - "table not writable under distribution-mode=none") - } - - /** - * ALTER TABLE SET TBLPROPERTIES ('openhouse.tableType'='REPLICA_TABLE') is rejected with a BadRequestException, since - * table type cannot be changed after creation. - */ - private def ddlReplTableTypeImmutableCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("ddl.repl.tableTypeImmutable") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + - "'openhouse.tableType'='REPLICA_TABLE')")) - - assert( - exception.getMessage.contains("restriction"), - s"msg: ${exception.getMessage.take(160)}") - } - - /** - * The column-tag, ACL and feature-flag cases. They start from the preparedCoreFormats preparation for the file - * format, except the distribution-mode case, which starts from three seed rows in a table created with - * write.distribution-mode=none. - */ - val ddlTagAclFeatureCases: List[Plan.Case] = preparedCoreFormats.flatMap { preparation => - val format = preparation.label - val distributionModePreparation = TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'write.distribution-mode'='none')")() - .insert(3)()) - - List( - ddlColTagCase(preparation), - ddlAclGrantUnsharedCase(preparation), - ddlAclGrantSharedCase(preparation), - ddlFeatureFlagDistributionModeCase(distributionModePreparation), - ddlReplTableTypeImmutableCase(preparation)) - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypeScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypeScenarios.scala new file mode 100644 index 000000000..9d8a46711 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypeScenarios.scala @@ -0,0 +1,260 @@ +package harness + +/** + * Nested and complex types: struct, array, map and struct-in-struct columns, the reads and writes that address their + * fields, and the schema changes the catalog allows inside a struct. + * + * Operations: a full round trip of every nested column, projection of a struct field, a filter on a struct field, an + * UPDATE of a struct field, a MERGE that inserts a fully nested row, a DELETE filtered on a struct field, an INSERT of + * null and empty nested values, ADD COLUMN of a new struct field, and the rejected DROP COLUMN of an existing struct + * field. + * + * Preparation axes: one unpartitioned NestedTable layout per file format, each seeded with three rows carrying struct, + * array, map and doubly-nested struct values; plus the standard seeded core table in Parquet and ORC for the two + * struct-evolution families, which build and drop their own side table. + * + * Case families: nine families contributing 25 cases, 21 on the nested layouts and 4 on the standard formats. + */ +trait NestedTypeScenarios extends ScenarioKit { + + /** Every nested-type case: the reads and writes on the nested layouts, then the struct-evolution cases. */ + lazy val nestedTypeCases: List[Plan.Case] = + preparedNestedTables.flatMap(preparation => + List( + roundtripCase(preparation), + projectFieldCase(preparation), + filterNestedFieldCase(preparation), + updateStructFieldCase(preparation), + mergeInsertCase(preparation), + deleteByNestedFieldCase(preparation), + nullValuesCase(preparation))) ++ + preparedCoreFormats.flatMap(preparation => + List( + addStructFieldCase(preparation), + dropStructFieldRejectedCase(preparation))) + + /** One unpartitioned nested-column table per file format. */ + lazy val nestedLayouts: List[Layout] = + fileFormats.map(format => + Layout( + s"nested-unpartitioned/$format", + table => + s"CREATE TABLE $table (${NestedTable.columnDefinitions}) USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")) + + /** One preparation per nested layout: the table is created, then seeded with three nested rows. */ + lazy val preparedNestedTables: List[TablePreparation[NestedTable.type]] = + nestedLayouts.map(layout => + TablePreparation( + layout.label, + TableTest(NestedTable).sql("create")(layout.create)().insert(standardSeedRowCount)())) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * Selecting the top-level id alongside struct, array, map and nested-struct fields reads back exactly the seeded + * values for all 3 rows. + */ + private def roundtripCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.roundtrip") { table => + val actual = table.spark + .sql( + s"SELECT id, s.x, s.y, arr, m['k'], nested.inner.z " + + s"FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(row => + ( + row.getLong(0), + row.getInt(1), + row.getString(2), + row.getSeq[Int](3), + row.getInt(4), + row.getInt(5))) + val expected = (1 to standardSeedRowCount).map { value => + ( + value.toLong, + value, + s"row-$value", + Seq(value, value + 1), + value, + value) + } + + assert(actual == expected) + } + + /** Selecting only a nested struct field (s.x) returns just that field's values for all 3 rows, in id order. */ + private def projectFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.projectField") { table => + val actual = table.spark + .sql(s"SELECT s.x FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(_.getInt(0)) + + assert(actual == Seq(1, 2, 3)) + } + + /** Filtering WHERE s.x = 2 on a nested struct field returns only the matching row's id. */ + private def filterNestedFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.filterNestedField") { table => + val actual = table.spark + .sql(s"SELECT id FROM ${table.name} WHERE s.x = 2 ORDER BY id") + .collect() + .toSeq + .map(_.getLong(0)) + + assert(actual == Seq(2L)) + } + + /** UPDATE SET s.x = 99 WHERE id = 2 changes only that row's nested field and leaves every other row unchanged. */ + private def updateStructFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.updateStructField") { table => + table.spark.sql( + s"UPDATE ${table.name} SET s.x = 99 WHERE id = 2") + + assert( + table.spark + .sql(s"SELECT s.x FROM ${table.name} WHERE id = 2") + .collect()(0) + .getInt(0) == 99) + assert( + table.spark + .sql(s"SELECT s.x FROM ${table.name} WHERE id = 1") + .collect()(0) + .getInt(0) == 1) + } + + /** + * MERGE WHEN NOT MATCHED THEN INSERT with a fully nested source row adds a 4th row whose nested struct field reads + * back as inserted. + */ + private def mergeInsertCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.mergeInsert") { table => + table.spark.sql( + s"""MERGE INTO ${table.name} target USING ( + SELECT * FROM VALUES + ( + CAST(4 AS BIGINT), + named_struct('x', 4, 'y', 'row-4'), + array(4, 5), + map('k', 4), + named_struct('inner', named_struct('z', 4))) + AS source(id, s, arr, m, nested) + ) source ON target.id = source.id + WHEN NOT MATCHED THEN INSERT *""") + + val ids = table.spark + .sql(s"SELECT id FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(_.getLong(0)) + + assert(ids == Seq(1L, 2L, 3L, 4L)) + assert( + table.spark + .sql(s"SELECT s.x FROM ${table.name} WHERE id = 4") + .collect()(0) + .getInt(0) == 4) + } + + /** DELETE WHERE s.x = 2 filtering on a nested struct field removes only the matching row, leaving ids 1 and 3. */ + private def deleteByNestedFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation + .test("nested.deleteByNestedField") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE s.x = 2") + + val ids = table.spark + .sql(s"SELECT id FROM ${table.name} ORDER BY id") + .collect() + .toSeq + .map(_.getLong(0)) + + assert(ids == Seq(1L, 3L)) + } + .copy(knownBugReason = Some( + "DELETE on a nested struct field crashes in the Spark and Iceberg row-level " + + "rewrite.")) + + /** + * Inserting a row with NULL struct, empty array and empty map reads back a null struct and an empty array for that + * row. + */ + private def nullValuesCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + preparation.test("nested.nullValues") { table => + table.spark.sql( + s"INSERT INTO ${table.name} VALUES (" + + "CAST(4 AS BIGINT), " + + "CAST(NULL AS struct), " + + "CAST(array() AS array), " + + "CAST(map() AS map), " + + "CAST(NULL AS struct>))") + + val insertedRow = table.spark + .sql(s"SELECT id, s, arr FROM ${table.name} WHERE id = 4") + .collect()(0) + + assert(insertedRow.isNullAt(1)) + assert(insertedRow.getSeq[Int](2).isEmpty) + } + + /** + * On a side table, ADD COLUMN of a new nested struct field null-fills it for the existing row and accepts a new row + * that sets the field. + */ + private def addStructFieldCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("nested.addStructField") { table => + val sideTable = s"${table.name}_nst" + withOwnedTable(table.spark.sql(_), sideTable)( + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, s STRUCT) USING $dataSource")) { + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") + table.spark.sql( + s"ALTER TABLE $sideTable ADD COLUMN s.w INT") + assert( + countOf(table.spark, s"SELECT count(*) FROM $sideTable WHERE s.w IS NULL") == "1", + "new nested field should null-fill the existing row") + + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(2 AS BIGINT), " + + "named_struct('x', 2, 'y', 'b', 'w', 9))") + assert( + countOf(table.spark, s"SELECT count(*) FROM $sideTable WHERE s.w = 9") == "1", + "new nested field should be writable") + } + } + + /** + * On a side table, ALTER TABLE DROP COLUMN of a nested struct field is rejected with an exception, and the field + * remains readable afterward. + */ + private def dropStructFieldRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("nested.dropStructField.rejected") { table => + val sideTable = s"${table.name}_nsd" + withOwnedTable(table.spark.sql(_), sideTable)( + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, s STRUCT) USING $dataSource")) { + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") + Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE $sideTable DROP COLUMN s.x")) + + assert( + table.spark + .sql(s"SELECT s.x FROM $sideTable") + .collect()(0) + .getInt(0) == 1, + "rejected nested drop should leave the field readable") + } + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala deleted file mode 100644 index 1ff8e6e13..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypesScenarios.scala +++ /dev/null @@ -1,505 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -trait NestedTypesScenarios extends ScenarioKit { - import Rows._ - - // Nested and complex types (NestedTable). - - /** One unpartitioned nested-column table per file format. */ - val nestedLayouts: List[Layout] = - List("parquet", "orc", "avro").map(format => Layout(s"nested-unpartitioned/$format", table => - s"CREATE TABLE $table (${NestedTable.columnDefinitions}) USING $dataSource TBLPROPERTIES ('write.format.default'='$format')")) - - /** Creates the nested-column table under `layout`, then seeds `numberOfRows` rows. */ - def createAndSeedNested(layout: Layout, numberOfRows: Int): TableTest[NestedTable.type] = - TableTest(NestedTable).sql("create")(layout.create)().insert(numberOfRows)() - - /** - * Selecting the top-level id alongside struct, array, map and nested-struct fields reads back exactly the seeded - * values for all 3 rows. - */ - private def nestedRoundtripCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = - preparation.test("nested.roundtrip") { table => - val actual = table.spark - .sql( - s"SELECT id, s.x, s.y, arr, m['k'], nested.inner.z " + - s"FROM ${table.name} ORDER BY id") - .collect() - .toSeq - .map(row => - ( - row.getLong(0), - row.getInt(1), - row.getString(2), - row.getSeq[Int](3), - row.getInt(4), - row.getInt(5))) - val expected = (1 to 3).map { value => - ( - value.toLong, - value, - s"row-$value", - Seq(value, value + 1), - value, - value) - } - - assert(actual == expected) - } - - /** Selecting only a nested struct field (s.x) returns just that field's values for all 3 rows, in id order. */ - private def nestedProjectFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = - preparation.test("nested.projectField") { table => - val actual = table.spark - .sql(s"SELECT s.x FROM ${table.name} ORDER BY id") - .collect() - .toSeq - .map(_.getInt(0)) - - assert(actual == Seq(1, 2, 3)) - } - - /** Filtering WHERE s.x = 2 on a nested struct field returns only the matching row's id. */ - private def nestedFilterNestedFieldCase( - preparation: TablePreparation[NestedTable.type]): Plan.Case = - preparation.test("nested.filterNestedField") { table => - val actual = table.spark - .sql(s"SELECT id FROM ${table.name} WHERE s.x = 2 ORDER BY id") - .collect() - .toSeq - .map(_.getLong(0)) - - assert(actual == Seq(2L)) - } - - /** UPDATE SET s.x = 99 WHERE id = 2 changes only that row's nested field and leaves every other row unchanged. */ - private def nestedUpdateStructFieldCase( - preparation: TablePreparation[NestedTable.type]): Plan.Case = - preparation.test("nested.updateStructField") { table => - table.spark.sql( - s"UPDATE ${table.name} SET s.x = 99 WHERE id = 2") - - assert( - table.spark - .sql(s"SELECT s.x FROM ${table.name} WHERE id = 2") - .collect()(0) - .getInt(0) == 99) - assert( - table.spark - .sql(s"SELECT s.x FROM ${table.name} WHERE id = 1") - .collect()(0) - .getInt(0) == 1) - } - - /** - * MERGE WHEN NOT MATCHED THEN INSERT with a fully nested source row adds a 4th row whose nested struct field reads - * back as inserted. - */ - private def nestedMergeInsertCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = - preparation.test("nested.mergeInsert") { table => - table.spark.sql( - s"""MERGE INTO ${table.name} target USING ( - SELECT * FROM VALUES - ( - CAST(4 AS BIGINT), - named_struct('x', 4, 'y', 'row-4'), - array(4, 5), - map('k', 4), - named_struct('inner', named_struct('z', 4))) - AS source(id, s, arr, m, nested) - ) source ON target.id = source.id - WHEN NOT MATCHED THEN INSERT *""") - - val ids = table.spark - .sql(s"SELECT id FROM ${table.name} ORDER BY id") - .collect() - .toSeq - .map(_.getLong(0)) - - assert(ids == Seq(1L, 2L, 3L, 4L)) - assert( - table.spark - .sql(s"SELECT s.x FROM ${table.name} WHERE id = 4") - .collect()(0) - .getInt(0) == 4) - } - - /** DELETE WHERE s.x = 2 filtering on a nested struct field removes only the matching row, leaving ids 1 and 3. */ - private def nestedDeleteByNestedFieldCase( - preparation: TablePreparation[NestedTable.type]): Plan.Case = - preparation - .test("nested.deleteByNestedField") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE s.x = 2") - - val ids = table.spark - .sql(s"SELECT id FROM ${table.name} ORDER BY id") - .collect() - .toSeq - .map(_.getLong(0)) - - assert(ids == Seq(1L, 3L)) - } - .copy(knownBugReason = Some( - "DELETE on a nested struct field crashes in the Spark and Iceberg row-level " + - "rewrite.")) - - /** - * Inserting a row with NULL struct, empty array and empty map reads back a null struct and an empty array for that - * row. - */ - private def nestedNullValuesCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = - preparation.test("nested.nullValues") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES (" + - "CAST(4 AS BIGINT), " + - "CAST(NULL AS struct), " + - "CAST(array() AS array), " + - "CAST(map() AS map), " + - "CAST(NULL AS struct>))") - - val insertedRow = table.spark - .sql(s"SELECT id, s, arr FROM ${table.name} WHERE id = 4") - .collect()(0) - - assert(insertedRow.isNullAt(1)) - assert(insertedRow.getSeq[Int](2).isEmpty) - } - - /** - * The nested-type cases. Each preparation holds three seed rows with struct, array, map and doubly-nested struct - * fields in one unpartitioned nested layout. - */ - val nestedCases: List[Plan.Case] = - nestedLayouts - .map(layout => - TablePreparation( - layout.label, - createAndSeedNested(layout, 3))) - .flatMap { preparation => - List( - nestedRoundtripCase(preparation), - nestedProjectFieldCase(preparation), - nestedFilterNestedFieldCase(preparation), - nestedUpdateStructFieldCase(preparation), - nestedMergeInsertCase(preparation), - nestedDeleteByNestedFieldCase(preparation), - nestedNullValuesCase(preparation)) - } - - // Type-edge coverage (TypesTable). - - /** One unpartitioned scalar-type table per file format. */ - val typesLayouts: List[Layout] = - List("parquet", "orc", "avro").map(format => Layout(s"types-unpartitioned/$format", table => - s"CREATE TABLE $table (${TypesTable.columnDefinitions}) USING $dataSource TBLPROPERTIES ('write.format.default'='$format')")) - - /** Creates the scalar-type table under `layout`, then seeds `numberOfRows` rows. */ - def createAndSeedTypes(layout: Layout, numberOfRows: Int): TableTest[TypesTable.type] = - TableTest(TypesTable).sql("create")(layout.create)().insert(numberOfRows)() - - // A full valued row for TypesTable with the given id; individual tests override specific columns. - private def typesRow(id: Long, n: String, x: String, dec: String, str: String): String = - s"(CAST($id AS BIGINT), $n, $x, $dec, $str, CAST('b' AS binary), DATE '2024-01-01', " + - s"TIMESTAMP '2024-01-01 00:00:00', TIMESTAMP_NTZ '2024-01-01 00:00:00')" - - private def partitionRow(id: Long, str: String, timestamp: String): String = - s"(CAST($id AS BIGINT), ${id.toInt}, ${id}.5, " + - s"CAST(${id}.50 AS decimal(10,2)), '$str', CAST('bin-$id' AS binary), " + - s"DATE '${timestamp.take(10)}', TIMESTAMP '$timestamp', TIMESTAMP_NTZ '$timestamp')" - - /** - * Selecting id, n, x, dec and str for the first seeded row reads back the exact long, int, double, decimal and string - * values that were seeded. - */ - private def typesRoundtripCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = - preparation.test("types.roundtrip") { table => - val row = table.spark - .sql( - s"SELECT id, n, x, dec, str FROM ${table.name} WHERE id = 1") - .collect()(0) - - assert( - row.getLong(0) == 1L && - row.getInt(1) == 1 && - row.getDouble(2) == 1.5) - assert( - row.getDecimal(3).compareTo( - new java.math.BigDecimal("1.50")) == 0) - assert(row.getString(4) == "row-1") - } - - /** - * Inserting a row with every non-key column NULL reads back as null for the int, double, string, timestamp and - * timestamp_ntz columns. - */ - private def typesNullsCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = - preparation.test("types.nulls") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES (" + - "CAST(10 AS BIGINT), NULL, NULL, NULL, NULL, " + - "NULL, NULL, NULL, NULL)") - - val row = table.spark - .sql( - s"SELECT n, x, str, ts, tsntz FROM ${table.name} WHERE id = 10") - .collect()(0) - - assert((0 to 4).forall(row.isNullAt)) - } - - /** Inserting rows with double('NaN') and double('Infinity') reads back as NaN and positive infinity respectively. */ - private def typesSpecialFloatsCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = - preparation.test("types.specialFloats") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - s"${typesRow(11, "0", "double('NaN')", "CAST(0 AS decimal(10,2))", "'x'")}, " + - s"${typesRow(12, "0", "double('Infinity')", "CAST(0 AS decimal(10,2))", "'y'")}") - - assert( - table.spark - .sql(s"SELECT x FROM ${table.name} WHERE id = 11") - .collect()(0) - .getDouble(0) - .isNaN) - assert( - table.spark - .sql(s"SELECT x FROM ${table.name} WHERE id = 12") - .collect()(0) - .getDouble(0) - .isInfinite) - } - - /** - * Inserting a row at Long.MaxValue, Int.MaxValue and a max-precision decimal reads those boundary values back - * unchanged. - */ - private def typesBoundariesCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = - preparation.test("types.boundaries") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - typesRow( - Long.MaxValue, - Int.MaxValue.toString, - "0.0", - "CAST(99999999.99 AS decimal(10,2))", - "'max'")) - - val row = table.spark - .sql( - s"SELECT id, n, dec FROM ${table.name} WHERE str = 'max'") - .collect()(0) - - assert( - row.getLong(0) == Long.MaxValue && - row.getInt(1) == Int.MaxValue) - assert( - row.getDecimal(2).compareTo( - new java.math.BigDecimal("99999999.99")) == 0) - } - - /** Inserting rows with a unicode string and an empty string reads each back unchanged. */ - private def typesUnicodeAndEmptyCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = - preparation.test("types.unicodeAndEmpty") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - s"${typesRow(13, "0", "0.0", "CAST(0 AS decimal(10,2))", "'\u65e5\u672c\u8a9e \uD83C\uDF89'")}, " + - s"${typesRow(14, "0", "0.0", "CAST(0 AS decimal(10,2))", "''")}") - - assert( - table.spark - .sql(s"SELECT str FROM ${table.name} WHERE id = 13") - .collect()(0) - .getString(0) == "\u65e5\u672c\u8a9e \uD83C\uDF89") - assert( - table.spark - .sql(s"SELECT str FROM ${table.name} WHERE id = 14") - .collect()(0) - .getString(0) == "") - } - - /** - * The type-edge cases. Each preparation holds three seed rows covering the int, double, decimal, string, binary, - * date, timestamp and timestamp_ntz columns in one unpartitioned types layout. - */ - val typesCases: List[Plan.Case] = - typesLayouts - .map(layout => - TablePreparation( - layout.label, - createAndSeedTypes(layout, 3))) - .flatMap { preparation => - List( - typesRoundtripCase(preparation), - typesNullsCase(preparation), - typesSpecialFloatsCase(preparation), - typesBoundariesCase(preparation), - typesUnicodeAndEmptyCase(preparation)) - } - - // Partition transforms and evolution. - - /** - * One supported partition transform: a table PARTITIONED BY that transform reports a single partition field with the - * expected name in its partitions metadata table, and the seeded rows land in the expected number of distinct - * partitions. The transform, its partition field name, and that partition count are the parameters. - */ - private def supportedPartitionTransformCase( - format: String, - caseName: String, - transform: String, - partitionField: String, - expectedPartitionCount: Int): Plan.Case = - TablePreparation( - format, - TableTest(TypesTable) - .sql("create")(table => - s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + - s"USING $dataSource PARTITIONED BY ($transform) " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .sql("insertPartitionRows")(table => - s"INSERT INTO $table VALUES " + - partitionRow(1, "aa-1", "2023-12-31 23:00:00") + ", " + - partitionRow(2, "bb-2", "2024-01-01 00:00:00") + ", " + - partitionRow(3, "cc-3", "2024-02-01 01:00:00"))(view => - assert( - view.after.size == view.before.size + 3, - s"expected three partition test rows, got ${view.after.size}"))) - .test(caseName) { table => - val partitionTable = - table.spark.table(s"${table.name}.partitions") - val partitionFields = partitionTable.schema("partition").dataType - .asInstanceOf[org.apache.spark.sql.types.StructType] - .fieldNames - .toSeq - - assert( - partitionFields == Seq(partitionField), - s"expected partition field $partitionField, got ${partitionFields.mkString(", ")}") - assert( - partitionTable.count() == expectedPartitionCount, - s"expected $expectedPartitionCount partitions for $transform") - } - - /** - * One rejected partition transform: CREATE TABLE PARTITIONED BY that transform fails with a RuntimeException carrying - * the expected message, and no scratch table is left behind. The transform and the expected message are the - * parameters. - */ - private def rejectedPartitionTransformCase( - format: String, - caseName: String, - transform: String, - expectedMessage: String): Plan.Case = - TablePreparation( - format, - TableTest(TypesTable) - .sql("create")(table => - s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + - s"USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")()) - .test(caseName) { table => - val scratchTable = table.name + "_x" - val exception = Check.intercept[RuntimeException]( - table.spark.sql( - s"CREATE TABLE $scratchTable " + - s"(${TypesTable.columnDefinitions}) " + - s"USING $dataSource PARTITIONED BY ($transform) " + - s"TBLPROPERTIES ('write.format.default'='$format')")) - - table.spark.sql(s"DROP TABLE IF EXISTS $scratchTable") - assert(exception.getMessage.contains(expectedMessage)) - } - - /** The supported and the rejected partition transforms, in parquet and in orc. */ - val partitionTransformCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - val supported = List( - ("partition.identity", "id", "id", 3), - ("partition.bucket", "bucket(4, id)", "id_bucket", 2), - ("partition.truncate", "truncate(2, str)", "str_trunc", 3), - ("partition.years", "years(ts)", "ts_year", 2), - ("partition.months", "months(ts)", "ts_month", 3), - ("partition.days", "days(ts)", "ts_day", 3), - ("partition.hours", "hours(ts)", "ts_hour", 3)) - .map { - case (caseName, transform, partitionField, expectedPartitionCount) => - supportedPartitionTransformCase(format, caseName, transform, partitionField, expectedPartitionCount) - } - val rejected = List( - ("partition.void.rejected", "void(n)", "not supported"), - ( - "partition.dateDay.rejected", - "days(dt)", - "Unsupported column")) - .map { - case (caseName, transform, expectedMessage) => - rejectedPartitionTransformCase(format, caseName, transform, expectedMessage) - } - - supported ++ rejected - } - - /** - * On three seed rows in an unpartitioned table in the given file format, ALTER TABLE ADD PARTITION FIELD is rejected - * with an exception stating that evolution of table partitioning is unsupported, which leaves recreating the table as - * the way to change partitioning. - */ - private def partitionEvolutionAddRejectedCase(format: String): Plan.Case = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - .test("partition.evolutionAdd.rejected") { table => - val exception = Check.intercept[Exception]( - table.spark.sql( - s"ALTER TABLE ${table.name} ADD PARTITION FIELD ${Core.date0.columnName}")) - - assert( - exception.getMessage.contains("Evolution of table partitioning")) - } - - /** - * On three seed rows in a table partitioned by the date column in the given file format, ALTER TABLE DROP PARTITION - * FIELD is rejected with an exception stating that evolution of table partitioning is unsupported. - */ - private def partitionEvolutionDropRejectedCase(format: String): Plan.Case = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.date0.columnName}) " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - .test("partition.evolutionDrop.rejected") { table => - val exception = Check.intercept[Exception]( - table.spark.sql( - s"ALTER TABLE ${table.name} DROP PARTITION FIELD ${Core.date0.columnName}")) - - assert( - exception.getMessage.contains("Evolution of table partitioning")) - } - - /** The rejected partition-evolution statements, in parquet and in orc. */ - val partitionEvolutionCases: List[Plan.Case] = - List("parquet", "orc").flatMap { format => - List( - partitionEvolutionAddRejectedCase(format), - partitionEvolutionDropRejectedCase(format)) - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala index 12e3a1659..5fdd0c0b3 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala @@ -1,13 +1,35 @@ package harness -/** Mixes the standard scenario-owned case lists into one catalog source. */ +/** Mixes every standard capability trait into one catalog source. */ object Scenarios - extends DmlScenarios - with NestedTypesScenarios - with MaintControlScenarios - with ForkScenarios - with NegativeDdlScenarios - with InteractionScenarios - with SurfaceScenarios - with HazardReaderWriterScenarios - with ImplementationPinScenarios + extends AccessControlScenarios + with ChangelogScenarios + with ColumnTagScenarios + with CompactionPlanningScenarios + with ConcurrencyScenarios + with DataTypeScenarios + with DmlScenarios + with DmlValidationScenarios + with EncryptionScenarios + with FileFormatScenarios + with FileReplicationScenarios + with IncrementalReadScenarios + with LockingScenarios + with MaintenanceScenarios + with MetadataTableScenarios + with NamespaceScenarios + with NestedTypeScenarios + with PartitionEvolutionScenarios + with PartitionTransformScenarios + with ProcedureScenarios + with RenameScenarios + with ScanPlanningScenarios + with SchemaEvolutionScenarios + with SnapshotRestoreScenarios + with SortOrderScenarios + with StreamingScenarios + with TableEvolutionCompatibilityScenarios + with TablePropertyScenarios + with TimeTravelScenarios + with WriteDistributionScenarios + with WriterCompatibilityScenarios diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionEvolutionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionEvolutionScenarios.scala new file mode 100644 index 000000000..5bd6b09a6 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionEvolutionScenarios.scala @@ -0,0 +1,60 @@ +package harness + +/** + * Partition evolution: changing the partition specification of an existing table. + * + * Operations: ALTER TABLE ADD PARTITION FIELD on an unpartitioned table and ALTER TABLE DROP PARTITION FIELD on a + * date-partitioned table. The catalog rejects both, so recreating the table is the way to change its partitioning. + * + * Preparation axes: in each of the two columnar formats, the standard seeded core table for the add case and a + * date-partitioned core table seeded with the standard rows for the drop case. + * + * Case families: two families contributing 4 cases. + */ +trait PartitionEvolutionScenarios extends ScenarioKit { + + /** The rejected partition-evolution statements, one file format at a time. */ + lazy val partitionEvolutionCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + addPartitionFieldRejectedCase(format), + dropPartitionFieldRejectedCase(format)) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * ALTER TABLE ADD PARTITION FIELD on an unpartitioned table is rejected with an exception stating that evolution of + * table partitioning is unsupported. + */ + private def addPartitionFieldRejectedCase(format: String): Plan.Case = + preparedStandardTable(format).test("partitionEvolution.add.rejected") { table => + val exception = Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} ADD PARTITION FIELD ${Core.date0.columnName}")) + + assert(exception.getMessage.contains("Evolution of table partitioning")) + } + + /** + * ALTER TABLE DROP PARTITION FIELD on a date-partitioned table is rejected with an exception stating that evolution + * of table partitioning is unsupported. + */ + private def dropPartitionFieldRejectedCase(format: String): Plan.Case = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"PARTITIONED BY (${Core.date0.columnName}) " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .insert(standardSeedRowCount)()) + .test("partitionEvolution.drop.rejected") { table => + val exception = Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} DROP PARTITION FIELD ${Core.date0.columnName}")) + + assert(exception.getMessage.contains("Evolution of table partitioning")) + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionTransformScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionTransformScenarios.scala new file mode 100644 index 000000000..88ef88e86 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionTransformScenarios.scala @@ -0,0 +1,155 @@ +package harness + +import org.apache.spark.sql.AnalysisException +import org.apache.spark.sql.types.StructType + +/** + * Partition transforms: which PARTITIONED BY transforms the catalog accepts at table creation, the partition field + * each accepted transform produces, and the partition specifications it rejects. + * + * Operations: CREATE TABLE PARTITIONED BY each of identity, bucket, truncate, years, months, days and hours followed + * by a read of the partitions metadata table; CREATE TABLE PARTITIONED BY the rejected void transform, the rejected + * days transform over a date column, and a column the table does not declare. + * + * Preparation axes: for the accepted and rejected transforms, a TypesTable in each of the two columnar formats seeded + * with three rows whose timestamps fall in three distinct hours, days, months and years; for the rejected partition + * column, the standard seeded core table in the same two formats. + * + * Case families: ten families contributing 20 cases, 14 accepted transforms and 6 rejections. + */ +trait PartitionTransformScenarios extends ScenarioKit { + + /** Every partition-transform case, one file format at a time. */ + lazy val partitionTransformCases: List[Plan.Case] = + standardFormats.flatMap { format => + acceptedTransforms.map { + case (caseName, transform, partitionField, expectedPartitionCount) => + acceptedTransformCase(format, caseName, transform, partitionField, expectedPartitionCount) + } ++ + rejectedTransforms.map { + case (caseName, transform, expectedMessage) => + rejectedTransformCase(format, caseName, transform, expectedMessage) + } + } ++ preparedCoreFormats.map(partitionByNonExistentColumnCase) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + // A fully valued TypesTable row whose date and timestamp columns both come from `timestamp`, so one row lands in one + // partition of every time-based transform. + private def partitionRow(id: Long, str: String, timestamp: String): String = + s"(CAST($id AS BIGINT), ${id.toInt}, ${id}.5, " + + s"CAST(${id}.50 AS decimal(10,2)), '$str', CAST('bin-$id' AS binary), " + + s"DATE '${timestamp.take(10)}', TIMESTAMP '$timestamp', TIMESTAMP_NTZ '$timestamp')" + + /** + * One accepted partition transform: a table PARTITIONED BY that transform reports a single partition field with the + * expected name in its partitions metadata table, and the three seeded rows land in the expected number of distinct + * partitions. The transform, its partition field name, and that partition count are the parameters. + */ + private def acceptedTransformCase( + format: String, + caseName: String, + transform: String, + partitionField: String, + expectedPartitionCount: Int): Plan.Case = + TablePreparation( + format, + TableTest(TypesTable) + .sql("create")(table => + s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + + s"USING $dataSource PARTITIONED BY ($transform) " + + s"TBLPROPERTIES ('write.format.default'='$format')")() + .sql("insertPartitionRows")(table => + s"INSERT INTO $table VALUES " + + partitionRow(1, "aa-1", "2023-12-31 23:00:00") + ", " + + partitionRow(2, "bb-2", "2024-01-01 00:00:00") + ", " + + partitionRow(3, "cc-3", "2024-02-01 01:00:00"))(view => + assert( + view.after.size == view.before.size + 3, + s"expected three partition test rows, got ${view.after.size}"))) + .test(caseName) { table => + val partitionTable = table.spark.table(s"${table.name}.partitions") + val partitionFields = partitionTable.schema("partition").dataType + .asInstanceOf[StructType] + .fieldNames + .toSeq + + assert( + partitionFields == Seq(partitionField), + s"expected partition field $partitionField, got ${partitionFields.mkString(", ")}") + assert( + partitionTable.count() == expectedPartitionCount, + s"expected $expectedPartitionCount partitions for $transform") + } + + /** + * One rejected partition transform: CREATE TABLE PARTITIONED BY that transform fails with a RuntimeException + * carrying the expected message, and the scratch table it would have created is gone. The transform and the expected + * message are the parameters. + */ + private def rejectedTransformCase( + format: String, + caseName: String, + transform: String, + expectedMessage: String): Plan.Case = + TablePreparation( + format, + TableTest(TypesTable) + .sql("create")(table => + s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + + s"USING $dataSource " + + s"TBLPROPERTIES ('write.format.default'='$format')")()) + .test(caseName) { table => + val scratchTable = table.name + "_x" + + withCleanupStatement(table.spark.sql(_), s"DROP TABLE IF EXISTS $scratchTable") { + val exception = Check.intercept[RuntimeException]( + table.spark.sql( + s"CREATE TABLE $scratchTable " + + s"(${TypesTable.columnDefinitions}) " + + s"USING $dataSource PARTITIONED BY ($transform) " + + s"TBLPROPERTIES ('write.format.default'='$format')")) + + assert(exception.getMessage.contains(expectedMessage)) + } + } + + /** + * CREATE TABLE PARTITIONED BY a column the table does not declare is rejected with an AnalysisException naming that + * column, and the scratch table it would have created is gone. + */ + private def partitionByNonExistentColumnCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("partition.byNonExistentColumn.rejected") { table => + val scratchTable = table.name + "_x" + + withCleanupStatement(table.spark.sql(_), s"DROP TABLE IF EXISTS $scratchTable") { + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"CREATE TABLE $scratchTable ($columnDefinitions) " + + s"USING $dataSource PARTITIONED BY (no_such_column) " + + s"TBLPROPERTIES ('write.format.default'='${preparation.label}')")) + + assert(exception.getMessage.contains("no_such_column")) + } + } + + // The accepted transforms: the case name, the PARTITIONED BY clause, the partition field the catalog derives, and + // the number of distinct partitions the three seeded rows land in. + private val acceptedTransforms: List[(String, String, String, Int)] = + List( + ("partition.identity", "id", "id", 3), + ("partition.bucket", "bucket(4, id)", "id_bucket", 2), + ("partition.truncate", "truncate(2, str)", "str_trunc", 3), + ("partition.years", "years(ts)", "ts_year", 2), + ("partition.months", "months(ts)", "ts_month", 3), + ("partition.days", "days(ts)", "ts_day", 3), + ("partition.hours", "hours(ts)", "ts_hour", 3)) + + // The rejected transforms: the case name, the PARTITIONED BY clause, and the message the rejection carries. + private val rejectedTransforms: List[(String, String, String)] = + List( + ("partition.void.rejected", "void(n)", "not supported"), + ("partition.dateDay.rejected", "days(dt)", "Unsupported column")) + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala index 58aa8ae1a..3d8096f9b 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala @@ -1,6 +1,12 @@ package harness -/** Defines the ordered catalog of scenario-owned test cases. */ +/** + * The ordered catalog of scenario-owned test cases. + * + * Every capability trait contributes exactly one case list. Plan names each contribution once, in alphabetical order + * by contribution name, and concatenates them. Composition is all Plan does: a scenario body, a preparation and a case + * ID all belong to the capability that owns them. + */ object Plan { final case class Case( id: String, @@ -15,96 +21,40 @@ object Plan { def bugReason(testCase: Case): Option[String] = testCase.knownBugReason.map(reason => s"bug: $reason") - // The interaction, surface, reader/writer and hazard families are crossed with these two file formats. Each family - // runs on one format before the next format starts, so the format loop is the outer one and every contribution below - // keeps the catalog position it holds today. - private val crossedFormats: List[String] = List("parquet", "orc") - - private def interactionContributions: List[Case] = - crossedFormats.flatMap { format => - List( - Scenarios.interactionDdlCases(format), - Scenarios.interactionMiscellaneousCases(format) - ).flatten - } - - private def surfaceContributions: List[Case] = - crossedFormats.flatMap { format => - List( - Scenarios.surfaceReaderCases(format), - Scenarios.surfaceRewriteProcedureCases(format), - Scenarios.surfaceSnapshotProcedureCases(format), - Scenarios.surfaceMetadataCases(format), - Scenarios.surfaceConcurrencyCases(format), - Scenarios.surfaceSchemaCases(format), - Scenarios.surfaceWriteCases(format), - Scenarios.surfacePinCases(format) - ).flatten - } - - private def hazardContributions: List[Case] = - crossedFormats.flatMap { format => - List( - Scenarios.hazardReaderCases(format), - Scenarios.hazardWriterCases(format) - ).flatten - } - - private def readerWriterContributions: List[Case] = - crossedFormats.flatMap { format => - List( - Scenarios.readerWriterChangelogAppendCases(format), - Scenarios.readerWriterChangelogOverwriteCases(format), - Scenarios.readerWriterChangelogDeleteCases(format), - Scenarios.readerWriterChangelogUpdateCases(format), - Scenarios.readerWriterChangelogMergeCases(format), - Scenarios.readerWriterIncrementalAndStreamCases(format) - ).flatten - } - - // Every DDL-consumer family runs against one evolved preparation before the next preparation starts, so the - // preparation loop is the outer one here. - private def ddlConsumerContributions: List[Case] = - Scenarios.ddlConsumerPreparations.flatMap { preparation => - List( - Scenarios.ddlConsumerDataCases(preparation), - Scenarios.ddlConsumerCompactionCases(preparation) - ).flatten - } - - def cases: List[Case] = + /** Every capability contribution, named once, in the order Plan integrates them. */ + def contributions: List[(String, List[Case])] = List( - Scenarios.coreDmlCases, - Scenarios.partitionedDmlCases, - Scenarios.nestedCases, - Scenarios.typesCases, - Scenarios.partitionTransformCases, - Scenarios.partitionEvolutionCases, - Scenarios.timeTravelCases, - Scenarios.restoreRollbackCases, - Scenarios.negativeCases, - Scenarios.createSchemaCases, - Scenarios.layoutFormatCases, - Scenarios.ddlSchemaCases, - Scenarios.ddlNegativeCases, - Scenarios.ddlPropertyCases, - Scenarios.ddlMiscellaneousCases, - Scenarios.ddlPolicyCases, - Scenarios.ddlTagAclFeatureCases, - Scenarios.maintenanceCases, - Scenarios.controlPlaneCases - ).flatten ++ - interactionContributions ++ - surfaceContributions ++ - hazardContributions ++ - Scenarios.hazardContextCases ++ - ddlConsumerContributions ++ - readerWriterContributions ++ - List( - Scenarios.orderedDmlCases, - Scenarios.evolvedDmlCases, - Scenarios.encryptionPinCases, - Scenarios.forkColumnDefaultAndDistributionCases, - Scenarios.forkFileAndCompactionCases - ).flatten + "accessControlCases" -> Scenarios.accessControlCases, + "changelogCases" -> Scenarios.changelogCases, + "columnTagCases" -> Scenarios.columnTagCases, + "compactionPlanningCases" -> Scenarios.compactionPlanningCases, + "concurrencyCases" -> Scenarios.concurrencyCases, + "dataTypeCases" -> Scenarios.dataTypeCases, + "dmlCases" -> Scenarios.dmlCases, + "dmlValidationCases" -> Scenarios.dmlValidationCases, + "encryptionCases" -> Scenarios.encryptionCases, + "fileFormatCases" -> Scenarios.fileFormatCases, + "fileReplicationCases" -> Scenarios.fileReplicationCases, + "incrementalReadCases" -> Scenarios.incrementalReadCases, + "lockingCases" -> Scenarios.lockingCases, + "maintenanceCases" -> Scenarios.maintenanceCases, + "metadataTableCases" -> Scenarios.metadataTableCases, + "namespaceCases" -> Scenarios.namespaceCases, + "nestedTypeCases" -> Scenarios.nestedTypeCases, + "partitionEvolutionCases" -> Scenarios.partitionEvolutionCases, + "partitionTransformCases" -> Scenarios.partitionTransformCases, + "procedureCases" -> Scenarios.procedureCases, + "renameCases" -> Scenarios.renameCases, + "scanPlanningCases" -> Scenarios.scanPlanningCases, + "schemaEvolutionCases" -> Scenarios.schemaEvolutionCases, + "snapshotRestoreCases" -> Scenarios.snapshotRestoreCases, + "sortOrderCases" -> Scenarios.sortOrderCases, + "streamingCases" -> Scenarios.streamingCases, + "tableEvolutionCompatibilityCases" -> Scenarios.tableEvolutionCompatibilityCases, + "tablePropertyCases" -> Scenarios.tablePropertyCases, + "timeTravelCases" -> Scenarios.timeTravelCases, + "writeDistributionCases" -> Scenarios.writeDistributionCases, + "writerCompatibilityCases" -> Scenarios.writerCompatibilityCases) + + def cases: List[Case] = contributions.flatMap { case (_, contribution) => contribution } } diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ProcedureScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ProcedureScenarios.scala new file mode 100644 index 000000000..0ccff3c48 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ProcedureScenarios.scala @@ -0,0 +1,113 @@ +package harness + +/** + * Catalog procedures and catalog-level statements: which of them this catalog implements, and which it rejects. + * + * Operations: ancestors_of over a two-snapshot history; register_table onto a new name from an existing metadata file, + * followed by the rejected system.snapshot and system.add_files import procedures; and the rejected CREATE VIEW and + * ANALYZE TABLE COMPUTE STATISTICS statements. + * + * Preparation axes: in each of the two columnar formats, the two-snapshot core table for the ancestry family and the + * standard seeded core table for the import and statement families. + * + * Case families: three families contributing 6 cases. + */ +trait ProcedureScenarios extends ScenarioKit { + + /** Every catalog-procedure case, one file format at a time. */ + lazy val procedureCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + ancestorsOfCase(preparedTwoSnapshotTable(format)), + registerTableCase(preparedStandardTable(format)), + viewAndAnalyzeRejectedCase(preparedStandardTable(format))) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** ancestors_of lists both snapshots of the table's two-snapshot history. */ + private def ancestorsOfCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("procedure.ancestorsOf") { table => + val ancestorCount = table.spark + .sql( + "CALL openhouse.system.ancestors_of(" + + s"table => '${catalogRelative(table.name)}')") + .collect() + .length + + assert( + ancestorCount == 2, + s"ancestors_of should list two snapshots, got $ancestorCount") + } + + /** + * register_table onto a new name makes the source table's snapshot readable there (3 rows) and leaves the source + * unchanged, and dropping the registered table leaves the source unchanged. The system.snapshot and system.add_files + * procedures each reject their unsupported inputs with an exception. + * + * The drop of the registered table is both its cleanup and the operation the source assertion after the ownership + * boundary depends on, so it runs as that boundary's cleanup. A drop the catalog refuses fails the case, so the case + * cannot pass while leaving the registration behind. The snapshot target extends the prepared table's generated + * name, and its own boundary removes it whether the procedure was rejected as expected, threw something else, or + * unexpectedly succeeded. + */ + private def registerTableCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("procedure.registerTable") { table => + val registeredTable = s"${table.name}_registered" + val snapshotTarget = s"${table.name}_snapshotTarget" + val absentSourceDirectory = s"/tmp/${table.name.split('.').last}_absentSource" + val metadataFile = table.spark + .sql( + s"SELECT file FROM ${table.name}.metadata_log_entries " + + "ORDER BY timestamp DESC LIMIT 1") + .collect()(0) + .getString(0) + + withOwnedTable(table.spark.sql(_), registeredTable)( + table.spark.sql( + "CALL openhouse.system.register_table(" + + s"table => '${catalogRelative(registeredTable)}', " + + s"metadata_file => '$metadataFile')")) { + assert( + countOf(table.spark, s"SELECT count(*) FROM $registeredTable") == "3", + "register_table should make all source rows readable") + } + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", + "dropping the registered table should leave the source rows in place") + + withCleanupStatement(table.spark.sql(_), s"DROP TABLE IF EXISTS $snapshotTarget") { + Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.snapshot(" + + s"source_table => '${catalogRelative(table.name)}', " + + s"table => '${catalogRelative(snapshotTarget)}')")) + } + + Check.intercept[Exception]( + table.spark.sql( + "CALL openhouse.system.add_files(" + + s"table => '${catalogRelative(table.name)}', " + + s"source_table => '`parquet`.`$absentSourceDirectory`')")) + } + + /** + * CREATE VIEW and ANALYZE TABLE COMPUTE STATISTICS are each rejected with an exception. The view name extends the + * prepared table's generated name, and its boundary removes the view whether the statement was rejected as expected, + * threw something else, or unexpectedly succeeded. + */ + private def viewAndAnalyzeRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("procedure.viewAndAnalyze.rejected") { table => + val viewName = s"${table.name}_view" + + withCleanupStatement(table.spark.sql(_), s"DROP VIEW IF EXISTS $viewName") { + Check.intercept[Exception]( + table.spark.sql(s"CREATE VIEW $viewName AS SELECT 1 AS one")) + } + + Check.intercept[Exception]( + table.spark.sql( + s"ANALYZE TABLE ${table.name} COMPUTE STATISTICS")) + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RenameScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RenameScenarios.scala new file mode 100644 index 000000000..d840c60e3 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RenameScenarios.scala @@ -0,0 +1,68 @@ +package harness + +import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException + +/** + * Table rename: ALTER TABLE RENAME TO moves a table to a new name with its rows, and the catalog refuses a rename onto + * a name that is already taken. + * + * Operations: RENAME TO a free name followed by a read of both the new and the old name, then a rename back; and + * RENAME TO the name of a table that already exists. + * + * Preparation axes: the standard seeded core table in each of the two columnar formats. The conflict family creates + * and drops the table it collides with. + * + * Case families: two families contributing 4 cases. + */ +trait RenameScenarios extends ScenarioKit { + + /** Every rename case, one file format at a time. */ + lazy val renameCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + renameTableCase(preparedStandardTable(format)), + renameTableConflictCase(preparedStandardTable(format), format)) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * ALTER TABLE RENAME TO moves the table to the new name with its 3 rows intact, and the old name stops resolving. A + * second rename puts the table back under its original name, which teardown drops. The rename boundary records the + * live name after each accepted rename, so a failure between the two renames drops the table under the name it + * currently answers to. + */ + private def renameTableCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("rename.table") { table => + val renamedTable = s"${table.name}_ren" + + withTrackedRename(table.spark.sql(_), table.name) { renameTo => + renameTo(renamedTable) + assert( + countOf(table.spark, s"SELECT count(*) FROM $renamedTable") == "3", + "the renamed table should keep its rows") + Check.intercept[Exception]( + table.spark.sql(s"SELECT 1 FROM ${table.name} LIMIT 1")) + renameTo(table.name) + } + } + + /** ALTER TABLE RENAME TO a name that already exists is rejected with an error naming the conflict. */ + private def renameTableConflictCase( + preparation: TablePreparation[CoreTable.type], + format: String): Plan.Case = + preparation.test("rename.table.conflict") { table => + val conflictingTable = s"${table.name}_other" + + withOwnedTable(table.spark.sql(_), conflictingTable)( + table.spark.sql(coreCreate(conflictingTable, format))) { + val exception = Check.intercept[WebClientResponseWithMessageException]( + table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $conflictingTable")) + + assert( + exception.getMessage.contains("already exists"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScanPlanningScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScanPlanningScenarios.scala new file mode 100644 index 000000000..e1e97fb5d --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScanPlanningScenarios.scala @@ -0,0 +1,115 @@ +package harness + +import org.apache.iceberg.TableProperties +import org.apache.iceberg.spark.{Spark3Util, SparkSQLProperties} +import scala.collection.JavaConverters._ + +/** + * Scan planning: the split size decides how the read path combines data files into read tasks, and every split size + * returns the same rows. + * + * Operations: read a six-file table under a large and a tiny spark.sql.iceberg.split-size, comparing the row set and + * the read RDD partition count; then plan the same table directly through the Iceberg scan API under a split size + * above the whole table and one below a single file, comparing the task-group counts. + * + * Preparation axes: one table per file format, built inside the case with write.distribution-mode=none and + * read.split.open-file-cost=1 and filled by six separate inserts, so it holds six separately weighted data files. + * + * Case families: one family contributing 2 cases. + */ +trait ScanPlanningScenarios extends ScenarioKit { + + /** The split-size case, one file format at a time. */ + lazy val scanPlanningCases: List[Plan.Case] = + standardFormats.map(format => + Plan.Case(s"scanPlanning.splitSize @ $format", splitSizeCase(format))) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * Over several small files, a large split size combines them into fewer read tasks and a tiny split size splits them + * into more, visible through rdd.getNumPartitions, and both reads return the same rows. The planner shows the same + * effect directly: a split size above the whole table plans one task group, and a split size below one file plans + * one group per file. + */ + private def splitSizeCase(format: String)(ctx: Ctx): Unit = { + val spark = ctx.spark + val table = TableTest.nextQualifiedTableName(ctx.namespace) + + // distribution=none plus several separate inserts produces several distinct data files. An open-file-cost of 1 + // sets each file's planning weight to its byte length, making split-size the knob that governs task-group count. + withOwnedTable(spark.sql(_), table)( + spark.sql( + s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'write.distribution-mode'='none', " + + "'read.split.open-file-cost'='1')")) { + val numberOfFiles = 6 + (0 until numberOfFiles).foreach { fileIndex => + spark.sql(s"INSERT INTO $table SELECT ${fileIndex}L, repeat('r$fileIndex', 4000)") + } + val fileCount = spark.sql(s"SELECT count(*) FROM $table.data_files").collect()(0).getLong(0) + assert( + fileCount >= 2, + s"[$format] expected multiple data files for a split test, got $fileCount") + + val splitSizeKey = SparkSQLProperties.SPLIT_SIZE // "spark.sql.iceberg.split-size" + val savedSplitSize = spark.conf.getOption(splitSizeKey) + def keys(): Seq[Long] = + spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) + def readPartitionCount(): Int = spark.sql(s"SELECT * FROM $table").rdd.getNumPartitions + val expectedKeys = (0 until numberOfFiles).map(_.toLong) + try { + // The row set is invariant under the split size, while the read RDD partition count follows it. + spark.conf.set(splitSizeKey, (512L * 1024 * 1024).toString) + val keysUnderLargeSplit = keys() + val partitionsUnderLargeSplit = readPartitionCount() + spark.conf.set(splitSizeKey, "1") + val keysUnderTinySplit = keys() + val partitionsUnderTinySplit = readPartitionCount() + assert( + keysUnderLargeSplit == expectedKeys && keysUnderTinySplit == expectedKeys, + s"[$format] split-size must leave the row set alone: large=$keysUnderLargeSplit " + + s"tiny=$keysUnderTinySplit expected=$expectedKeys") + assert( + partitionsUnderTinySplit >= partitionsUnderLargeSplit, + s"[$format] a smaller split-size must keep or raise the read RDD partition count: " + + s"tiny=$partitionsUnderTinySplit large=$partitionsUnderLargeSplit") + + // The same knob checked directly at the planner: with open-file-cost=1 each file's planning weight is its + // byte length, so a split-size below one file combines nothing (one task group per file) while a split-size + // above the whole table combines everything into one group. + val icebergTable = Spark3Util.loadIcebergTable(spark, table) + val targetSizeKey = TableProperties.SPLIT_SIZE // "read.split.target-size" + def plannedTaskGroups(splitBytes: Long): Int = + icebergTable + .newScan() + .option(targetSizeKey, splitBytes.toString) + .planTasks() + .asScala + .size + val groupsUnderLargeSplit = plannedTaskGroups(512L * 1024 * 1024) + val groupsUnderTinySplit = plannedTaskGroups(1L) + assert( + groupsUnderLargeSplit == 1, + s"[$format] a split-size above the whole table should plan 1 task group, " + + s"got $groupsUnderLargeSplit") + assert( + groupsUnderTinySplit == fileCount, + s"[$format] a split-size below one file should plan one task group per file ($fileCount), " + + s"got $groupsUnderTinySplit") + + println( + s"DIAG scanPlanning.splitSize[$format]: key='$splitSizeKey' files=$fileCount " + + s"readPartitions(large=$partitionsUnderLargeSplit,tiny=$partitionsUnderTinySplit) " + + s"taskGroups(large=$groupsUnderLargeSplit,tiny=$groupsUnderTinySplit)") + } finally { + // The split size is session state, not a table, so the case restores whatever the session held before it. + savedSplitSize match { + case Some(value) => spark.conf.set(splitSizeKey, value) + case None => spark.conf.unset(splitSizeKey) + } + } + } + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala index d0b358a2f..10da0fad8 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala @@ -1,25 +1,21 @@ package harness -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// Shared foundation for every Scenario trait: the standard table/layout/prep "kit". All domain traits (DmlScenarios, -// ForkScenarios, ...) extend this, so mixing them into `object Scenarios` puts ScenarioKit first in the linearization, -// so its vals initialize before any domain's, exactly as in the original single object. It holds the copy-on-write -// layouts and preparations only; each feature layer carries its own kit that extends this one. `protected` members are -// the shared kit; `public` ones are also consumed by `object Plan`. +import org.apache.spark.sql.{Row, SparkSession} +import java.util.concurrent.TimeUnit + +/** + * The shared starting-state kit every capability trait builds on: the core table shape, the layout cross-product, the + * standard seed, the standard preparations, and the small query helpers a case needs. + * + * Every capability trait extends this kit, so mixing them into `object Scenarios` puts ScenarioKit first in the + * linearization and its vals initialize before any capability's. It holds copy-on-write layouts and preparations only; + * each feature layer carries its own kit that extends this one. `protected` members are the shared kit; `public` ones + * are also consumed by `object Plan` and by the catalog tests. + */ trait ScenarioKit { - import Rows._ protected val Core = CoreTable // brevity in the typed column references below - protected val cols = Core.columnNames.mkString(", ") // source column list, so renames propagate + protected val columnNameList = Core.columnNames.mkString(", ") // source column list, so renames propagate // The rows a case reads back are ordered by the long column and carry exactly the core columns in their declared // order, so an expected row set is written as the rows the case started from, filtered, mapped through @@ -35,9 +31,9 @@ trait ScenarioKit { // --- layouts: one file format and one partitioning per starting table shape --- // A layout is one starting table shape. Each layout is a plain literal CREATE statement: the column list is one - // shared literal `columnDefinitions`, and format and partitioning are literal fragments. createSchema cross-checks - // the literal against CoreTable's declared columns, so the two stay in step. A layout belongs to the preparation, so - // one test case is written once and runs on every layout. + // shared literal `columnDefinitions`, and format and partitioning are literal fragments. The schema-creation case + // cross-checks the literal against CoreTable's declared columns, so the two stay in step. A layout belongs to the + // preparation, so one test case is written once and runs on every layout. protected val columnDefinitions = "foo_col_long bigint, foo_col_int int, foo_col_string string, foo_col_double double, " + "foo_col_boolean boolean, foo_col_date string" @@ -57,7 +53,14 @@ trait ScenarioKit { protected val partitionings: List[Partitioning] = List(unpartitioned, partitionedByDate) - protected val fileFormats: List[String] = List("parquet", "orc", "avro") + /** Every file format the catalog writes. This is the single source for a format list anywhere in the harness. */ + val fileFormats: List[String] = List("parquet", "orc", "avro") + + /** + * The two columnar formats every capability family runs on when it is crossed by format. The file-format capability + * itself covers the whole of `fileFormats`; every other family covers these two. + */ + val standardFormats: List[String] = List("parquet", "orc") /** One copy-on-write table in `format`, shaped by `partitioning`, labelled for its case IDs. */ private def coreLayout(partitioning: Partitioning, format: String): Layout = @@ -78,85 +81,111 @@ trait ScenarioKit { val partitionedLayouts: List[Layout] = fileFormats.map(format => coreLayout(partitionedByDate, format)) - /** - * The Parquet and ORC core layouts, each crossed with both partitionings, for the bespoke DDL cases that do not need - * the full file-format cross. - */ + /** The Parquet and ORC core layouts, each crossed with both partitionings. */ val parquetAndOrcLayouts: List[Layout] = for { - format <- List("parquet", "orc") + format <- standardFormats partitioning <- partitionings } yield coreLayout(partitioning, format) - /** Creates the table under `layout`, then seeds `numberOfRows` deterministic rows. */ - def createAndSeed(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = - TableTest(Core).sql("create")(layout.create)().insert(numberOfRows)() + /** + * The standard seed writes three deterministic rows with keys 1, 2 and 3. Row `n` holds key `n` in the long column, + * `n` in the int column, `row-n` in the string column, `n.5` in the double column, `n % 2 == 0` in the boolean + * column, and the date value `n - 1` hours after `2024-01-01-00`. `RowGenerator` builds those literals from + * `CoreTable`, so the seed follows a column rename. Every preparation that seeds a core table writes exactly these + * rows, so a case that starts from a seeded table knows its three starting keys. + */ + val standardSeedRowCount: Int = 3 - /** One preparation per core layout: three seed rows with keys 1, 2 and 3. */ + /** Creates the table under `layout` and leaves it empty. The caller adds the seed step it wants. */ + def create(layout: Layout): TableTest[CoreTable.type] = + TableTest(Core).sql("create")(layout.create)() + + /** One preparation per core layout: the table is created, then seeded with the standard rows. */ val preparedCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation( layout.label, - createAndSeed(layout, 3))) + create(layout).insert(standardSeedRowCount)())) - /** One preparation per date-partitioned core layout: three seed rows with keys 1, 2 and 3, one row per date value. */ + /** + * One preparation per date-partitioned core layout: the table is created, then seeded with the standard rows, whose + * date values put one row in each of three partitions. + */ val preparedPartitionedCoreTables: List[TablePreparation[CoreTable.type]] = partitionedLayouts.map(layout => TablePreparation( layout.label, - createAndSeed(layout, 3))) + create(layout).insert(standardSeedRowCount)())) /** - * One preparation per core layout: three seed rows, then ALTER TABLE WRITE ORDERED BY the long key, so the table - * carries that write sort order. + * One preparation per core layout: the table is created, seeded, then given a write sort order on the long key by + * ALTER TABLE WRITE ORDERED BY. The column list stays as seeded, so every DML case runs on the result. */ val preparedOrderedCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation( layout.label, - createAndSeedOrdered(layout, 3), + create(layout) + .insert(standardSeedRowCount)() + .sql("writeOrderedByLongKey")(table => + s"ALTER TABLE $table WRITE ORDERED BY ${Core.long0.columnName}")(), "prep.ordered:")) /** - * One preparation per core layout: three seed rows, then ADD COLUMN prep_extra int, so the table carries one column - * beyond the seed row shape and the seeded rows read null for it. + * One preparation per core layout: the table is created, seeded, then given a prep_extra int column by ALTER TABLE + * ADD COLUMN. The column list grows past the seed row shape and the seeded rows read null for the new column, so the + * cases that address columns by name run on the result: the reads, the deletes and the updates. */ val preparedEvolvedCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation( layout.label, - createAndSeedEvolved(layout, 3), + create(layout) + .insert(standardSeedRowCount)() + .sql("addPrepExtraColumn")(table => s"ALTER TABLE $table ADD COLUMN prep_extra int")(), "prep.evolved:")) /** One preparation per core layout: the table is created and left unseeded, so it holds no rows. */ val preparedEmptyCoreTables: List[TablePreparation[CoreTable.type]] = - layouts.map(layout => - TablePreparation( - layout.label, - TableTest(Core).sql("create")(layout.create)())) + layouts.map(layout => TablePreparation(layout.label, create(layout))) - /** One preparation per Parquet and ORC unpartitioned layout: three seed rows with keys 1, 2 and 3. */ - val preparedCoreFormats: List[TablePreparation[CoreTable.type]] = - List("parquet", "orc").map { format => - val layout = coreLayout(unpartitioned, format) - TablePreparation( - format, - createAndSeed(layout, 3)) - } + /** The CREATE statement for an unpartitioned core table in `format`. */ + protected def coreCreate(table: String, format: String): String = + coreLayout(unpartitioned, format).create(table) + + /** An unseeded, unpartitioned core table in `format`, so the case owns every row the table holds. */ + protected def preparedEmptyStandardTable(format: String): TablePreparation[CoreTable.type] = + TablePreparation(format, create(coreLayout(unpartitioned, format))) /** - * Creates and seeds the table under `layout`, then gives it a write sort order on the long key. The column list stays - * as seeded, so every DML case runs on the result. + * An unpartitioned core table in `format`, created and then seeded with the standard rows. This is the plainest + * starting state in the harness, so most capability families build on it. */ - def createAndSeedOrdered(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = - createAndSeed(layout, numberOfRows).sql("prep.ordered")(t => s"ALTER TABLE $t WRITE ORDERED BY ${CoreTable.long0.columnName}")() + protected def preparedStandardTable(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + create(coreLayout(unpartitioned, format)).insert(standardSeedRowCount)()) + + /** The standard seeded table in each of the two columnar formats. */ + val preparedCoreFormats: List[TablePreparation[CoreTable.type]] = + standardFormats.map(preparedStandardTable) /** - * Creates and seeds the table under `layout`, then adds the prep_extra column. The column list grows past the seed - * row shape, so the cases that address columns by name run on the result: the reads, the deletes and the updates. + * An unpartitioned core table in `format` holding five rows across two snapshots: the standard seed, then rows 4 and + * 5. The step between the two commits holds until the wall clock passes the seed commit's timestamp, so the two + * snapshots carry distinct commit times and a timestamp-bounded read resolves to exactly one of them. */ - def createAndSeedEvolved(layout: Layout, numberOfRows: Int): TableTest[CoreTable.type] = - createAndSeed(layout, numberOfRows).sql("prep.evolved")(t => s"ALTER TABLE $t ADD COLUMN prep_extra int")() + protected def preparedTwoSnapshotTable(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + create(coreLayout(unpartitioned, format)) + .insert(standardSeedRowCount)() + .step("waitForNextSnapshotTimestamp")(waitForNextSnapshotTimestamp)() + .sql("insertRowsFourAndFive")(table => + s"INSERT INTO $table VALUES " + + "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + + "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) /** * The same starting state with a fourth row whose key is 99 and whose string column is null, so exactly one row of @@ -177,48 +206,8 @@ trait ScenarioKit { val preparedNullStringOrderedCoreTables: List[TablePreparation[CoreTable.type]] = preparedOrderedCoreTables.map(withNullStringRow) - /** - * Every data file the preparation wrote carries the extension of the table's declared write.format.default, and - * listing the files leaves the table state unchanged. - */ - private def formatMaterializationCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("format.materialization") { table => - val before = table.state - val declaredFormat = table.spark - .sql(s"SHOW TBLPROPERTIES ${table.name} ('write.format.default')") - .collect()(0) - .getString(1) - val filePaths = table.spark - .sql(s"SELECT file_path FROM ${table.name}.files") - .collect() - .toSeq - .map(_.getString(0)) - val after = table.state - - assert( - filePaths.nonEmpty && filePaths.forall(_.toLowerCase.endsWith(s".$declaredFormat")), - s"data files are not all .$declaredFormat: $filePaths") - assert(after == before, "listing files leaves the rows and the snapshot count unchanged") - } - - /** - * The format-materialization case for each preparation given. It applies to any preparation that leaves data files - * behind, so each feature layer passes the list its own preparations produce. - */ - def layoutFormatCasesFor( - preparations: List[TablePreparation[CoreTable.type]] - ): List[Plan.Case] = - preparations.map { preparation => - formatMaterializationCase(preparation) - } - - /** The standard preparations that leave data files behind: the core and write-ordered ones. */ - val layoutFormatPreparations: List[TablePreparation[CoreTable.type]] = - preparedCoreTables ++ preparedOrderedCoreTables - - /** The format-materialization case on every standard preparation that writes data files. */ - def layoutFormatCases: List[Plan.Case] = layoutFormatCasesFor(layoutFormatPreparations) - + // Waits until the wall clock passes the newest snapshot's commit timestamp, so the next commit lands on a strictly + // later millisecond and a timestamp-bounded read separates the two snapshots. private def waitForNextSnapshotTimestamp(spark: SparkSession, table: String): Unit = { val previousTimestamp = spark .sql( @@ -227,7 +216,7 @@ trait ScenarioKit { .collect()(0) .getTimestamp(0) .getTime - val deadline = System.nanoTime() + java.util.concurrent.TimeUnit.SECONDS.toNanos(5) + val deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5) while ( System.currentTimeMillis() <= previousTimestamp && @@ -240,34 +229,100 @@ trait ScenarioKit { s"clock did not advance beyond snapshot timestamp $previousTimestamp") } - // Shared helpers used across domain traits. + // --- table, rename and lock lifecycle boundaries used by cases that build artifacts for themselves --- + // Each boundary takes the catalog statement executor, so a test drives the same code with a recorder in place of + // Spark. Call sites pass `spark.sql(_)`. /** - * Creates a table in the given file format, seeds three rows as the first snapshot, then inserts rows 4 and 5 as a - * second snapshot committed at a later timestamp. + * Runs `use` against `table`, a table the case builds for itself. `create` issues the CREATE; ownership starts the + * moment it returns, so a name that is already taken leaves the pre-existing table intact and the drop afterwards + * removes only the table this call created. A failure in `create` or `use` stays the + * primary failure, and a cleanup failure is attached to it as a suppressed exception. Callers name the table with + * `TableTest.nextQualifiedTableName` or by extending the generated name of a prepared table, so two runs of the same + * case can never address the same table. */ - protected def coreTwoSnapshots(fmt: String): TableTest[CoreTable.type] = - TableTest(Core) - .sql("create")(table => s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='$fmt')")() - .insert(3)() - .step("waitForNextSnapshotTimestamp")(waitForNextSnapshotTimestamp)() - .sql("insertMore")(table => s"INSERT INTO $table VALUES " + - s"(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), (CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")() - - /** The two-snapshot table in parquet. */ - protected def coreTwoSnapshots: TableTest[CoreTable.type] = coreTwoSnapshots("parquet") - - // Snapshots in ancestry order (root first), following the parent_id chain. This is deterministic even if two commits - // happen to share a committed_at millisecond (which `ORDER BY committed_at` is not). + private[harness] def withOwnedTable(runStatement: String => Unit, table: String)( + create: => Unit)(use: => Unit): Unit = + OwnedTableLifecycle.withOwnership(runStatement(s"DROP TABLE IF EXISTS $table")) { + markTableCreated => + create + markTableCreated() + use + } + + /** + * Runs `use`, then runs `cleanupStatement` on every outcome. A case uses this for an artifact whose creation is the + * rejection under test: the statement that would create it is expected to fail, so there is no successful create to + * take ownership of, yet a rejection that partly applied, threw the wrong type, or unexpectedly succeeded must still + * leave nothing behind. The failure `use` raises stays primary and a cleanup failure rides along suppressed. + */ + private[harness] def withCleanupStatement(runStatement: String => Unit, cleanupStatement: String)( + use: => Unit): Unit = + OwnedTableLifecycle.withCleanup(runStatement(cleanupStatement))(use) + + /** + * Runs `use` while tracking which name a table answers to. `use` receives a rename function that issues one ALTER + * TABLE RENAME TO and records the new name only once the catalog accepted it, so the boundary always knows the live + * name. If `use` leaves the table under any name other than `originalTable`, the boundary drops that live name, so + * a failed assertion or a failed rename back still ends with the table gone. A failure in `use` stays primary and a + * cleanup failure rides along suppressed. + */ + private[harness] def withTrackedRename(runStatement: String => Unit, originalTable: String)( + use: (String => Unit) => Unit): Unit = { + var liveTable = originalTable + OwnedTableLifecycle.withCleanup( + if (liveTable != originalTable) runStatement(s"DROP TABLE IF EXISTS $liveTable")) { + use { newTable => + runStatement(s"ALTER TABLE $liveTable RENAME TO $newTable") + liveTable = newTable + } + } + } + + /** + * Runs `use` while the case holds a table lock. `lock` is taken first and its response is checked; `use` receives a + * release function so a case that reads behavior after the lock is gone releases it itself. The boundary releases + * the lock afterwards only while the case still holds it, so exactly one release is attempted. Every release checks + * its response, so a rejected release fails the case, and a release failure that follows a failure inside `use` + * rides along as a suppressed exception. + */ + private[harness] def withTableLock( + lock: () => (Int, String), + unlock: () => (Int, String))(use: (() => Unit) => Unit): Unit = { + val (lockStatus, lockBody) = lock() + assert(lockStatus >= 200 && lockStatus < 300, s"lock request failed: $lockStatus $lockBody") + + var lockHeld = true + def releaseLock(): Unit = { + val (unlockStatus, unlockBody) = unlock() + lockHeld = false + assert( + unlockStatus >= 200 && unlockStatus < 300, + s"unlock request failed: $unlockStatus $unlockBody") + } + + OwnedTableLifecycle.withCleanup(if (lockHeld) releaseLock())(use(() => releaseLock())) + } + + // --- shared query helpers used across capability traits --- + + // Snapshots in ancestry order (root first), following the parent_id chain. The chain orders commits deterministically + // even when two of them share a committed_at millisecond. protected def snapshotIds(spark: SparkSession, table: String): Seq[Long] = { val rows = spark.sql(s"SELECT snapshot_id, parent_id FROM $table.snapshots").collect().toSeq - val ids = rows.map(_.getLong(0)).toSet - val childByParent = rows.collect { case r if !r.isNullAt(1) => r.getLong(1) -> r.getLong(0) }.toMap - val root = rows.collectFirst { case r if r.isNullAt(1) || !ids.contains(r.getLong(1)) => r.getLong(0) }.get - val order = scala.collection.mutable.ListBuffer(root) - var cur = root - while (childByParent.contains(cur)) { cur = childByParent(cur); order += cur } - order.toList + val snapshotIdSet = rows.map(_.getLong(0)).toSet + val childByParent = rows.collect { + case row if !row.isNullAt(1) => row.getLong(1) -> row.getLong(0) + }.toMap + val root = rows.collectFirst { + case row if row.isNullAt(1) || !snapshotIdSet.contains(row.getLong(1)) => row.getLong(0) + }.get + + Iterator + .iterate(Option(root))(parent => parent.flatMap(childByParent.get)) + .takeWhile(_.isDefined) + .flatten + .toList } protected def catalogRelative(table: String): String = table.stripPrefix("openhouse.") @@ -275,15 +330,10 @@ trait ScenarioKit { protected def coreRow(long: Long, tag: String): String = s"(CAST($long AS BIGINT), ${long.toInt}, '$tag', ${long}.5, false, '2024-01-01-00')" - protected val L = CoreTable.long0.columnName - // The Spark data source used by CREATE TABLE statements. The LinkedIn adapter overrides this before building // Plan.cases. Catalog procedure calls still use the catalog name "openhouse". var dataSource: String = "iceberg" - protected def coreCreateParquet(table: String): String = - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES ('write.format.default'='parquet')" - protected def tableProps(spark: SparkSession, table: String): Map[String, String] = spark.sql(s"SHOW TBLPROPERTIES $table").collect().toSeq.map(r => r.getString(0) -> r.getString(1)).toMap diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SchemaEvolutionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SchemaEvolutionScenarios.scala new file mode 100644 index 000000000..4cc596273 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SchemaEvolutionScenarios.scala @@ -0,0 +1,324 @@ +package harness + +import org.apache.spark.sql.AnalysisException +import org.apache.iceberg.exceptions.BadRequestException + +/** + * Schema evolution on the core table: the shape a CREATE TABLE statement produces, and the column additions, type + * changes, reorderings and nullability changes the catalog accepts or rejects afterwards. + * + * Operations: read the created schema, ADD COLUMN in its single, multiple, commented and positioned forms, ALTER + * COLUMN TYPE to widen an int and to widen a decimal, ALTER COLUMN FIRST to reorder, ALTER COLUMN DROP NOT NULL, + * RENAME COLUMN, and the rejected forms DROP COLUMN, DROP COLUMN over written data, ALTER COLUMN TYPE to a narrower + * type and ALTER COLUMN SET NOT NULL. + * + * Preparation axes: the six unseeded core layouts for the created-schema family; the six seeded core layouts for the + * evolution families; the standard seeded table in Parquet and ORC for the rejection families and for the families + * that build their own side table. + * + * Case families: 14 families contributing 56 cases, 6 created-schema, 36 evolution, and 14 rejection or side-table + * cases. + */ +trait SchemaEvolutionScenarios extends ScenarioKit { + + /** Every schema-evolution case: the created schema, then the accepted changes, then the boundaries. */ + lazy val schemaEvolutionCases: List[Plan.Case] = + createdSchemaCases ++ schemaChangeCases ++ schemaBoundaryCases + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * The created table's schema is exactly CoreTable's columns, in declaration order and with their declared types, and + * the table holds no rows. + */ + private def createdSchemaCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.create") { table => + val actual = table.spark + .table(table.name) + .schema + .fields + .toList + .map(field => field.name -> field.dataType.simpleString) + val expected = Core.tableColumns.toList.map(column => (column.columnName, column.sqlType)) + + assert(actual == expected, s"schema is $actual") + assert(table.rows.isEmpty, "a table that was never seeded holds no rows") + } + + /** ADD COLUMN adds the column to the schema, the existing rows read null for it, and the row count is unchanged. */ + private def addColumnSingleCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.addColumn.single") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_int int") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + val nullCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name} WHERE added_int IS NULL") + .collect()(0) + .getLong(0) + + assert(columnNames.contains("added_int"), s"added_int missing: $columnNames") + assert( + nullCount == table.preparedRows.size, + s"existing rows should read null for added_int: $nullCount != ${table.preparedRows.size}") + assert(table.rows.size == table.preparedRows.size, "ADD COLUMN changed the row count") + } + + /** ADD COLUMNS with two columns in one statement adds both to the schema and leaves the row count unchanged. */ + private def addColumnMultipleCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.addColumn.multiple") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMNS (added_a int, added_b string)") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + + assert( + columnNames.contains("added_a") && columnNames.contains("added_b"), + s"added columns missing: $columnNames") + assert(table.rows.size == table.preparedRows.size, "ADD COLUMNS changed the row count") + } + + /** ADD COLUMN ... COMMENT stores the comment on the added column and the reader sees it. */ + private def addColumnCommentCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.addColumn.comment") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_c int COMMENT 'a note'") + + val addedColumn = table.spark + .table(table.name) + .schema + .fields + .find(_.name == "added_c") + .getOrElse(throw new AssertionError("added_c missing")) + + assert( + addedColumn.getComment().contains("a note"), + s"comment not stored: ${addedColumn.getComment()}") + } + + /** ADD COLUMN ... AFTER foo_col_long places the added column directly after that column in the schema. */ + private def addColumnPositionCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.addColumn.position") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN added_after int AFTER ${Core.long0.columnName}") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + + assert( + columnNames.indexOf("added_after") == columnNames.indexOf(Core.long0.columnName) + 1, + s"added_after not after long0: $columnNames") + } + + /** + * ALTER COLUMN foo_col_int TYPE bigint widens the column in the schema and the already-written values read back + * unchanged. + */ + private def alterColumnTypeWidenCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.alterColumn.typeWiden") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.int0.columnName} TYPE bigint") + + val liveColumns = table.spark.table(table.name).schema.fields.toSeq + .map(field => field.name -> field.dataType.simpleString) + .toMap + val values = table.spark + .sql( + s"SELECT ${Core.int0.columnName} FROM ${table.name} ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) + + assert( + liveColumns.get(Core.int0.columnName).contains("bigint"), + s"int0 not widened: ${liveColumns.get(Core.int0.columnName)}") + assert(values == Seq(1L, 2L, 3L), s"values not preserved after widening: $values") + } + + /** + * RENAME COLUMN renames the column in the schema: the new name is present, the old name is gone, and the row count + * is unchanged. + */ + private def renameColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation + .test("schema.renameColumn") { table => + table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN to_rename int") + table.spark.sql(s"ALTER TABLE ${table.name} RENAME COLUMN to_rename TO renamed_col") + + val columnNames = table.spark.table(table.name).schema.fields.toSeq.map(_.name) + + assert( + columnNames.contains("renamed_col") && !columnNames.contains("to_rename"), + s"RENAME COLUMN silently no-oped: $columnNames") + assert(table.rows.size == table.preparedRows.size, "RENAME COLUMN changed the row count") + } + .copy(knownBugReason = Some( + "RENAME COLUMN is a silent no-op because server-side schema casing normalization " + + "restores the old name.")) + + /** ALTER TABLE DROP COLUMN is rejected with a BadRequestException naming the column that would be dropped. */ + private def dropColumnRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.dropColumn.rejected") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} DROP COLUMN ${Core.int0.columnName}")) + + assert( + exception.getMessage.contains("not found in newSchema"), + s"unexpected message: ${exception.getMessage.take(160)}") + assert( + exception.getMessage.contains(Core.int0.columnName), + s"message should name the dropped column: ${exception.getMessage.take(160)}") + } + + /** + * DROP COLUMN on a column that holds data is rejected, the column's data remains readable, and the table remains + * writable. + */ + private def dropColumnWithDataRejectedCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.dropColumn.withData.rejected") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} DROP COLUMN extra_col")) + + assert( + exception.getMessage.contains("not found in newSchema"), + s"drop rejection message changed: ${exception.getMessage.take(200)}") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} WHERE extra_col = 42") == "1", + "rejected drop should leave the column data readable") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert10") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "5", + "rejected drop should leave the table writable") + } + + /** + * ALTER TABLE ALTER COLUMN to a narrower type (bigint to int) is rejected with an AnalysisException about the + * unsupported column change. + */ + private def alterColumnNarrowTypeRejectedCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.alterColumn.narrowType.rejected") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.long0.columnName} TYPE int")) + + assert( + exception.getMessage.contains("NOT_SUPPORTED_CHANGE_COLUMN"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + + /** + * ALTER TABLE ALTER COLUMN SET NOT NULL on a nullable column is rejected with an AnalysisException about the + * nullable-to-non-nullable change. + */ + private def alterColumnSetNotNullRejectedCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.alterColumn.setNotNull.rejected") { table => + val exception = Check.intercept[AnalysisException]( + table.spark.sql( + s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.string0.columnName} SET NOT NULL")) + + assert( + exception.getMessage.contains("Cannot change nullable column to non-nullable"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + + /** On a side table, dropping NOT NULL from a column allows a subsequent insert of a null value for that column. */ + private def alterColumnDropNotNullCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.alterColumn.dropNotNull") { table => + val sideTable = s"${table.name}_nn" + withOwnedTable(table.spark.sql(_), sideTable)( + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, req INT NOT NULL) USING $dataSource")) { + table.spark.sql( + s"ALTER TABLE $sideTable ALTER COLUMN req DROP NOT NULL") + table.spark.sql( + s"INSERT INTO $sideTable VALUES (CAST(1 AS BIGINT), NULL)") + assert( + countOf(table.spark, s"SELECT count(*) FROM $sideTable WHERE req IS NULL") == "1", + "relaxing NOT NULL should allow a null write") + } + } + + /** + * On a side table, widening a decimal column's precision preserves the original row and accepts a new row whose + * value only fits the wider precision. + */ + private def alterColumnDecimalWidenCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.alterColumn.decimalWiden") { table => + val sideTable = s"${table.name}_dec" + withOwnedTable(table.spark.sql(_), sideTable)( + table.spark.sql( + s"CREATE TABLE $sideTable " + + s"(id BIGINT, dec DECIMAL(10,2)) USING $dataSource")) { + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(1 AS BIGINT), CAST(12345678.99 AS DECIMAL(10,2)))") + table.spark.sql( + s"ALTER TABLE $sideTable ALTER COLUMN dec TYPE DECIMAL(12,2)") + table.spark.sql( + s"INSERT INTO $sideTable VALUES " + + "(CAST(2 AS BIGINT), CAST(1234567890.99 AS DECIMAL(12,2)))") + assert( + countOf(table.spark, s"SELECT count(*) FROM $sideTable") == "2", + "decimal widening should preserve old and new values") + } + } + + /** ALTER TABLE ALTER COLUMN ... FIRST moves that column to the front of the schema while preserving all 3 rows. */ + private def alterColumnReorderFirstCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("schema.alterColumn.reorderFirst") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} " + + s"ALTER COLUMN ${Core.string0.columnName} FIRST") + val columns = table.spark + .sql(s"SELECT * FROM ${table.name} LIMIT 1") + .columns + .toSeq + + assert( + columns.head == Core.string0.columnName, + s"FIRST should move the column to the front: $columns") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", + "column reorder should preserve the rows") + } + + /** The created-schema case on every unseeded core layout. */ + private val createdSchemaCases: List[Plan.Case] = + preparedEmptyCoreTables.map(createdSchemaCase) + + /** The accepted schema changes on every seeded core layout. */ + private val schemaChangeCases: List[Plan.Case] = + preparedCoreTables.flatMap { preparation => + List( + addColumnSingleCase(preparation), + addColumnMultipleCase(preparation), + addColumnCommentCase(preparation), + addColumnPositionCase(preparation), + alterColumnTypeWidenCase(preparation), + renameColumnCase(preparation)) + } + + /** The rejected schema changes and the side-table schema changes, in each of the two columnar formats. */ + private val schemaBoundaryCases: List[Plan.Case] = + preparedCoreFormats.flatMap { preparation => + List( + dropColumnRejectedCase(preparation), + dropColumnWithDataRejectedCase(preparation), + alterColumnNarrowTypeRejectedCase(preparation), + alterColumnSetNotNullRejectedCase(preparation), + alterColumnDropNotNullCase(preparation), + alterColumnDecimalWidenCase(preparation), + alterColumnReorderFirstCase(preparation)) + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SnapshotRestoreScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SnapshotRestoreScenarios.scala new file mode 100644 index 000000000..1b6c01550 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SnapshotRestoreScenarios.scala @@ -0,0 +1,89 @@ +package harness + +/** + * Snapshot restore: returning a table to an earlier snapshot, and what the restored table keeps. + * + * Operations: rollback_to_snapshot and set_current_snapshot back to the seed snapshot, and rollback_to_snapshot back + * to a pre-evolution snapshot after ADD COLUMN and an insert into the new column. + * + * Preparation axes: in each of the two columnar formats, the two-snapshot core table for the two restore procedures, + * and the standard seeded core table for the schema-evolution family. + * + * Case families: three families contributing 6 cases. + */ +trait SnapshotRestoreScenarios extends ScenarioKit { + + /** Every snapshot-restore case, one file format at a time. */ + lazy val snapshotRestoreCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + rollbackToSnapshotCase(preparedTwoSnapshotTable(format)), + setCurrentSnapshotCase(preparedTwoSnapshotTable(format)), + afterAddColumnCase(preparedStandardTable(format))) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** rollback_to_snapshot to the first snapshot restores the 3 rows the seed commit wrote. */ + private def rollbackToSnapshotCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("restore.rollbackToSnapshot") { table => + val firstSnapshotId = snapshotIds(table.spark, table.name).head + + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $firstSnapshotId)") + + assert(table.rows.size == 3) + } + + /** set_current_snapshot to the first snapshot restores the 3 rows the seed commit wrote. */ + private def setCurrentSnapshotCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("restore.setCurrentSnapshot") { table => + val firstSnapshotId = snapshotIds(table.spark, table.name).head + + table.spark.sql( + "CALL openhouse.system.set_current_snapshot(" + + s"'${catalogRelative(table.name)}', $firstSnapshotId)") + + assert(table.rows.size == 3) + } + + /** + * Rolling back to the pre-evolution snapshot after ADD COLUMN and an insert keeps the evolved schema, restores 3 + * rows that read null for the new column, and leaves the table accepting writes into that column. + */ + private def afterAddColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("restore.afterAddColumn") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $seedSnapshotId)") + val currentColumns = table.spark + .sql(s"SELECT * FROM ${table.name} LIMIT 1") + .columns + .toSeq + + assert( + currentColumns.contains("extra_col"), + s"rollback should retain the evolved schema: $currentColumns") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", + "rollback should restore 3 rows") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} WHERE extra_col IS NOT NULL") == "0", + "rolled-back rows should read the evolved column as null") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert10") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", + "the rolled-back table should accept evolved-schema writes") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SortOrderScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SortOrderScenarios.scala new file mode 100644 index 000000000..88a25e05c --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SortOrderScenarios.scala @@ -0,0 +1,61 @@ +package harness + +/** + * Sort order: ALTER TABLE WRITE ORDERED BY records a write sort order on the table, which the catalog pairs with range + * distribution, and the table keeps accepting writes under it. + * + * Operations: WRITE ORDERED BY a single column, and WRITE ORDERED BY two columns with an explicit direction and null + * ordering followed by an insert. + * + * Preparation axes: the standard seeded core table in each of the two columnar formats. + * + * Case families: two families contributing 4 cases. + */ +trait SortOrderScenarios extends ScenarioKit { + + /** Every sort-order case, one file format at a time. */ + lazy val sortOrderCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + orderedByCase(preparedStandardTable(format)), + orderedByMultipleColumnsCase(preparedStandardTable(format))) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** ALTER TABLE WRITE ORDERED BY a single column sets write.distribution-mode to range. */ + private def orderedByCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("sortOrder.orderedBy") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} WRITE ORDERED BY ${Core.long0.columnName}") + + val distributionMode = + tableProps(table.spark, table.name).get("write.distribution-mode") + + assert( + distributionMode.contains("range"), + s"a write sort order should set range distribution, got $distributionMode") + } + + /** + * ALTER TABLE WRITE ORDERED BY multiple columns sets range distribution and the table remains writable, growing from + * 3 to 5 rows after a follow-up insert. + */ + private def orderedByMultipleColumnsCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("sortOrder.orderedByMultipleColumns") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} WRITE ORDERED BY " + + s"${Core.string0.columnName} DESC NULLS FIRST, ${Core.long0.columnName}") + + assert( + tableProps(table.spark, table.name).get("write.distribution-mode").contains("range"), + "a multi-column write sort order should set range distribution") + + table.spark.sql( + s"INSERT INTO ${table.name} ${RowGenerator.valuesClause(Core, 2)}") + + assert(table.rows.size == 5, "the multi-column ordered write path should accept two rows") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/StreamingScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/StreamingScenarios.scala new file mode 100644 index 000000000..e6b3d55bf --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/StreamingScenarios.scala @@ -0,0 +1,213 @@ +package harness + +import java.nio.file.Files +import org.apache.spark.sql.SQLContext +import org.apache.spark.sql.execution.streaming.MemoryStream +import org.apache.spark.sql.streaming.Trigger + +/** + * Structured streaming: reading a table as a stream, writing a stream into a table, resuming a stream across a + * restart, and the snapshot histories a resumed stream rejects. + * + * Operations: a streaming read into a memory sink; a streaming append of two rows through the iceberg write-stream + * format; a streaming read into a destination table, restarted after an append; the same restart after a DELETE + * snapshot; and the same restart after the checkpoint's offset snapshot has been expired. + * + * Preparation axes: the standard seeded core table in each of the two columnar formats. The three restart families + * create and drop their own destination table in the same format. + * + * Case families: five families contributing 10 cases. + */ +trait StreamingScenarios extends ScenarioKit { + + /** Every streaming case, one file format at a time. */ + lazy val streamingCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + readCase(preparedStandardTable(format)), + writeCase(preparedStandardTable(format)), + readAcrossRestartCase(preparedStandardTable(format), format), + deleteSnapshotRejectedCase(preparedStandardTable(format), format), + expiredCheckpointCase(preparedStandardTable(format), format)) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + // Runs one AvailableNow batch of a streaming read of `source` into `destination`, resuming from `checkpoint`. Each + // call returns after the batch has been committed, so the caller can assert on the destination and then run again. + private def streamOneBatch( + table: PreparedTable[CoreTable.type], + destination: String, + checkpoint: String): Unit = { + val query = table.spark.readStream + .table(table.name) + .writeStream + .format("iceberg") + .outputMode("append") + .trigger(Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .toTable(destination) + assert(query.awaitTermination(120000), "stream did not finish") + query.stop() + } + + /** + * A Spark structured streaming read of the table, run in AvailableNow batch mode, delivers all 3 seed rows to a + * memory sink within 120 seconds. + */ + private def readCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("streaming.read") { table => + val checkpoint = Files.createTempDirectory("ck-read").toString + val sink = s"memsink_${System.nanoTime}" + val query = table.spark.readStream + .table(table.name) + .writeStream + .format("memory") + .queryName(sink) + .trigger(Trigger.AvailableNow()) + .option("checkpointLocation", checkpoint) + .start() + + assert( + query.awaitTermination(120000), + "streaming read did not finish in 120 seconds") + assert( + countOf(table.spark, s"SELECT count(*) FROM $sink") == "3", + "streaming read should deliver the three seed rows") + } + + /** + * A Spark structured streaming append of two rows through the iceberg write-stream format lands both rows, growing + * the table from 3 to 5 rows. + */ + private def writeCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("streaming.write") { table => + import table.spark.implicits._ + implicit val sqlContext: SQLContext = table.spark.sqlContext + val memoryStream = MemoryStream[Long] + memoryStream.addData(100L, 101L) + val rows = memoryStream.toDF().selectExpr( + s"value AS ${Core.long0.columnName}", + s"CAST(value AS INT) AS ${Core.int0.columnName}", + s"concat('row-', value) AS ${Core.string0.columnName}", + s"CAST(value AS DOUBLE) AS ${Core.double0.columnName}", + s"true AS ${Core.boolean0.columnName}", + s"'2024-01-01-00' AS ${Core.date0.columnName}") + val checkpoint = Files.createTempDirectory("ck-write").toString + val query = rows.writeStream + .format("iceberg") + .outputMode("append") + .option("checkpointLocation", checkpoint) + .toTable(table.name) + + query.processAllAvailable() + query.stop() + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "5", + "streaming write should append two rows") + } + + /** + * A streaming read of the table delivers the seed rows on first run and the newly inserted row after restart, into a + * destination table. + */ + private def readAcrossRestartCase( + preparation: TablePreparation[CoreTable.type], + format: String): Plan.Case = + preparation.test("streaming.readAcrossRestart") { table => + val destination = s"${table.name}_s" + val checkpoint = Files.createTempDirectory("ck-restart").toString + + withOwnedTable(table.spark.sql(_), destination)( + table.spark.sql(coreCreate(destination, format))) { + streamOneBatch(table, destination, checkpoint) + assert( + countOf(table.spark, s"SELECT count(*) FROM $destination") == "3", + "initial stream did not deliver the seed") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + streamOneBatch(table, destination, checkpoint) + assert( + countOf(table.spark, s"SELECT count(*) FROM $destination") == "4", + "stream restart did not deliver the appended row") + } + } + + /** + * An append-only stream restarted after a DELETE snapshot was written fails, with an error mentioning delete or + * overwrite. + */ + private def deleteSnapshotRejectedCase( + preparation: TablePreparation[CoreTable.type], + format: String): Plan.Case = + preparation.test("streaming.deleteSnapshot.rejected") { table => + val destination = s"${table.name}_sd" + val checkpoint = Files.createTempDirectory("ck-delete").toString + + withOwnedTable(table.spark.sql(_), destination)( + table.spark.sql(coreCreate(destination, format))) { + streamOneBatch(table, destination, checkpoint) + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val exception = + Check.intercept[Exception](streamOneBatch(table, destination, checkpoint)) + + assert( + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage).exists(message => + message.toLowerCase.contains("delete") || + message.toLowerCase.contains("overwrite"))), + "an append-only stream rejects a delete snapshot: " + + s"${exception.getClass.getSimpleName} ${Option(exception.getMessage).getOrElse("").take(140)}") + } + } + + /** + * A streaming read that resumes after its earliest offset snapshot has been expired fails, with an error naming the + * expired or missing snapshot. + */ + private def expiredCheckpointCase( + preparation: TablePreparation[CoreTable.type], + format: String): Plan.Case = + preparation.test("streaming.expiredCheckpoint") { table => + val destination = s"${table.name}_sink" + val checkpoint = Files.createTempDirectory("ck-expired").toString + + withOwnedTable(table.spark.sql(_), destination)( + table.spark.sql(coreCreate(destination, format))) { + streamOneBatch(table, destination, checkpoint) + assert( + countOf(table.spark, s"SELECT count(*) FROM $destination") == "3", + "initial stream should deliver the seed") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") + streamOneBatch(table, destination, checkpoint) + assert( + countOf(table.spark, s"SELECT count(*) FROM $destination") == "4", + "control restart should deliver one incremental row") + + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + val exception = + Check.intercept[Exception](streamOneBatch(table, destination, checkpoint)) + + assert( + Exceptions.causeChain(exception).exists(error => + Option(error.getMessage).exists(message => + message.contains("expired or removed") || + message.contains("Cannot load current offset") || + message.contains("Cannot find snapshot"))), + "stream restart should report the expired checkpoint offset") + } + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala deleted file mode 100644 index 5be1e36fb..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SurfaceScenarios.scala +++ /dev/null @@ -1,767 +0,0 @@ -package harness - -import org.apache.spark.sql.{AnalysisException, Row, SparkSession} -import org.apache.iceberg.exceptions.BadRequestException -import org.apache.iceberg.exceptions.ValidationException -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException -import java.time.LocalDateTime -import java.time.format.DateTimeFormatter -import scala.annotation.tailrec -import scala.reflect.{ClassTag, classTag} -import scala.util.control.NonFatal - -// The standard surface families. A surface case pins one edge of what the catalog exposes on a plain copy-on-write -// table: a reader, a procedure, a metadata table, a concurrency outcome, a schema change, or a write property. Each -// family builds the starting states it needs, so a family reads on its own. The concurrency helpers below are feature -// neutral, so a feature layer reuses them through a self-type on this trait. The cases run on parquet and orc. -trait SurfaceScenarios extends ScenarioKit { - import Rows._ - - protected def runConcurrently(functions: Seq[() => Unit]): Seq[Throwable] = { - val errors = new java.util.concurrent.ConcurrentLinkedQueue[Throwable]() - val start = new java.util.concurrent.CountDownLatch(1) - val threads = functions.zipWithIndex.map { case (function, index) => - val thread = new Thread( - () => - try { - start.await() - function() - } catch { - case interrupted: InterruptedException => - Thread.currentThread().interrupt() - errors.add(interrupted) - case throwable: Throwable => - errors.add(throwable) - }, - s"delta-harness-concurrent-$index") - thread.setDaemon(true) - thread - } - threads.foreach(_.start()) - start.countDown() - - val deadline = - System.nanoTime() + java.util.concurrent.TimeUnit.MINUTES.toNanos(3) - threads.foreach { thread => - val remainingNanos = deadline - System.nanoTime() - if (remainingNanos > 0) { - java.util.concurrent.TimeUnit.NANOSECONDS.timedJoin(thread, remainingNanos) - } - } - - threads.filter(_.isAlive).foreach { thread => - errors.add( - new AssertionError( - s"${thread.getName} did not complete within 3 minutes")) - thread.interrupt() - } - errors.toArray(Array.empty[Throwable]).toSeq - } - - protected def isTypedCommitConflict(throwable: Throwable): Boolean = - Exceptions.causeChain(throwable).exists { cause => - val className = cause.getClass.getName - className.contains("CommitFailed") || - className.contains("CommitStateUnknown") || - className.contains("Validation") || - className.contains("BadRequest") || - className.contains("WebClientResponse") - } - - /** - * Three seed rows with keys 1, 2 and 3 in an unpartitioned table in the given file format. This is the plainest - * starting state here, so the feature layers build their cases on it too. - */ - protected def surfaceBasePreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)()) - - /** - * Five rows across two snapshots, a three-row seed then a two-row insert, in an unpartitioned table in the given file - * format. - */ - private def surfaceTwoSnapshotPreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(3)() - .sql("insertMore")(table => - s"INSERT INTO $table VALUES " + - "(CAST(4 AS BIGINT), 4, 'row-4', 4.5, true, '2024-01-04-03'), " + - "(CAST(5 AS BIGINT), 5, 'row-5', 5.5, false, '2024-01-05-04')")()) - - /** An unseeded, empty unpartitioned table in the given file format. */ - private def surfaceEmptyPreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")()) - - /** - * Three seed rows in a table in the given file format, partitioned by the date column and carrying - * write.distribution-mode=hash. - */ - private def surfaceHashPreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.date0.columnName}) " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', " + - "'write.distribution-mode'='hash')")() - .insert(3)()) - - /** - * Three seed rows in an unpartitioned table in the given file format, carrying write.target-file-size-bytes=1048576. - */ - private def surfaceTargetFileSizePreparation(format: String): TablePreparation[CoreTable.type] = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', " + - "'write.target-file-size-bytes'='1048576')")() - .insert(3)()) - - /** - * A Spark structured streaming read of the table, run in AvailableNow batch mode, delivers all 3 seed rows to a - * memory sink within 120 seconds. - */ - private def surfaceStreamReadCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.stream.read") { table => - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-read").toString - val sink = s"memsink_${System.nanoTime}" - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("memory") - .queryName(sink) - .trigger(org.apache.spark.sql.streaming.Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .start() - - assert( - query.awaitTermination(120000), - "streaming read did not finish in 120 seconds") - assert( - countOf(table.spark, s"SELECT count(*) FROM $sink") == "3", - "streaming read should deliver the three seed rows") - } - - /** - * A Spark structured streaming append of two rows through the iceberg write-stream format lands both rows, growing - * the table from 3 to 5 rows. - */ - private def surfaceStreamWriteCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.stream.write") { table => - import table.spark.implicits._ - implicit val sqlContext: org.apache.spark.sql.SQLContext = - table.spark.sqlContext - val memoryStream = - org.apache.spark.sql.execution.streaming.MemoryStream[Long] - memoryStream.addData(100L, 101L) - val rows = memoryStream.toDF().selectExpr( - s"value AS ${Core.long0.columnName}", - s"CAST(value AS INT) AS ${Core.int0.columnName}", - s"concat('row-', value) AS ${Core.string0.columnName}", - s"CAST(value AS DOUBLE) AS ${Core.double0.columnName}", - s"true AS ${Core.boolean0.columnName}", - s"'2024-01-01-00' AS ${Core.date0.columnName}") - val checkpoint = - java.nio.file.Files.createTempDirectory("ck-write").toString - val query = rows.writeStream - .format("iceberg") - .outputMode("append") - .option("checkpointLocation", checkpoint) - .toTable(table.name) - - query.processAllAvailable() - query.stop() - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "5", - "streaming write should append two rows") - } - - /** create_changelog_view over an append-only history reports 5 changes, all of change type INSERT. */ - private def surfaceCdcChangelogViewCase(format: String): Plan.Case = - surfaceTwoSnapshotPreparation(format).test("surface.cdc.changelogView") { table => - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}')") - .collect()(0) - .getString(0) - val changeCount = table.spark - .sql(s"SELECT count(*) FROM $view") - .collect()(0) - .getLong(0) - val changeTypes = table.spark - .sql(s"SELECT DISTINCT _change_type FROM $view") - .collect() - .map(_.getString(0)) - .toSet - - assert( - changeCount == 5, - s"append-only changelog should contain 5 changes, got $changeCount") - assert( - changeTypes == Set("INSERT"), - s"append-only changelog should contain only INSERT: $changeTypes") - } - - /** The structured-streaming reader and writer, and the changelog view. */ - def surfaceReaderCases(format: String): List[Plan.Case] = - List( - surfaceStreamReadCase(format), - surfaceStreamWriteCase(format), - surfaceCdcChangelogViewCase(format)) - - /** - * After 5 single-row inserts fragment the manifest list, rewrite_manifests compacts it to fewer manifests while - * preserving all 5 rows. - */ - private def surfaceProcRewriteManifestsCase(format: String): Plan.Case = - surfaceEmptyPreparation(format).test("surface.proc.rewriteManifests") { table => - (1 to 5).foreach(index => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - coreRow(index, s"r$index"))) - val manifestCountBefore = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.manifests") - .collect()(0) - .getLong(0) - table.spark.sql( - "CALL openhouse.system.rewrite_manifests(" + - s"table => '${catalogRelative(table.name)}', " + - "use_caching => false)") - val manifestCountAfter = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.manifests") - .collect()(0) - .getLong(0) - - println( - "DIAG surface.proc.rewriteManifests: " + - s"manifests before=$manifestCountBefore after=$manifestCountAfter") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "5", - "rewrite_manifests should preserve the five rows") - assert( - manifestCountBefore >= 2 && - manifestCountAfter < manifestCountBefore, - "rewrite_manifests should compact the manifest set") - } - - /** - * The rewrite procedure that compacts the manifest set. The case starts from an unseeded table in the given file - * format and fragments the manifest list itself. - */ - def surfaceRewriteProcedureCases(format: String): List[Plan.Case] = - List( - surfaceProcRewriteManifestsCase(format)) - - /** ancestors_of lists both snapshots of the table's two-snapshot history. */ - private def surfaceProcAncestorsOfCase(format: String): Plan.Case = - surfaceTwoSnapshotPreparation(format).test("surface.proc.ancestorsOf") { table => - val ancestorCount = table.spark - .sql( - "CALL openhouse.system.ancestors_of(" + - s"table => '${catalogRelative(table.name)}')") - .collect() - .length - - assert( - ancestorCount == 2, - s"ancestors_of should list two snapshots, got $ancestorCount") - } - - /** - * remove_orphan_files deletes a planted, backdated stray file next to a real data file while the table's 3 live rows - * remain intact. - */ - private def surfaceProcRemoveOrphanRealCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.proc.removeOrphanReal") { table => - val dataFile = table.spark - .sql(s"SELECT file_path FROM ${table.name}.files LIMIT 1") - .collect()(0) - .getString(0) - .stripPrefix("file:") - val orphanFile = java.nio.file.Paths - .get(dataFile) - .getParent - .resolve("zz_orphan_plant.parquet") - java.nio.file.Files.write( - orphanFile, - "not-a-real-parquet".getBytes) - java.nio.file.Files.setLastModifiedTime( - orphanFile, - java.nio.file.attribute.FileTime.fromMillis(1546300800000L)) - - table.spark.sql( - "CALL openhouse.system.remove_orphan_files(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2020-01-01 00:00:00')") - assert( - java.nio.file.Files.notExists(orphanFile), - "remove_orphan_files should delete the planted orphan") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "remove_orphan_files should preserve live data") - } - - /** - * The procedures that read snapshot ancestry and remove orphan files. Ancestry runs on a two-snapshot table and - * orphan removal on a seeded table, each in the given file format. - */ - def surfaceSnapshotProcedureCases(format: String): List[Plan.Case] = - List( - surfaceProcAncestorsOfCase(format), - surfaceProcRemoveOrphanRealCase(format)) - - /** - * Selecting the hidden metadata columns _file, _pos, _spec_id and _partition returns one row per seed row, each with - * a populated file path and a non-negative position. - */ - private def surfaceMetaHiddenColumnsCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.meta.hiddenColumns") { - table => - val rows = table.spark - .sql( - s"SELECT _file, _pos, _spec_id, _partition FROM ${table.name}") - .collect() - .toSeq - - assert( - rows.size == 3, - s"hidden metadata columns should return 3 rows, got ${rows.size}") - assert( - rows.forall(row => - Option(row.getString(0)).exists(_.nonEmpty)), - "_file should be populated for every row") - assert( - rows.forall(_.getLong(1) >= 0), - "_pos should be non-negative for every row") - } - - /** - * Every Iceberg metadata table (entries, files, manifests, snapshots, history, refs, partitions, and their all_* - * variants) is queryable without error, and the snapshots metadata table reports the table's 2 snapshots. - */ - private def surfaceMetaTableSweepCase(format: String): Plan.Case = - surfaceTwoSnapshotPreparation(format).test("surface.meta.tableSweep") { table => - val metadataTables = Seq( - "entries", - "files", - "manifests", - "snapshots", - "history", - "refs", - "partitions", - "metadata_log_entries", - "data_files", - "all_data_files", - "all_manifests", - "all_entries", - "all_files") - metadataTables.foreach { metadataTable => - table.spark - .sql( - s"SELECT count(*) FROM ${table.name}.`$metadataTable`") - .collect() - } - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}.snapshots") == "2", - "snapshot metadata should contain two snapshots") - } - - /** The hidden metadata columns and the Iceberg metadata tables. */ - def surfaceMetadataCases(format: String): List[Plan.Case] = - List( - surfaceMetaHiddenColumnsCase(format), - surfaceMetaTableSweepCase(format)) - - /** - * Two threads concurrently insert 3 rows each; every insert either commits or fails with a typed commit-conflict - * exception, and the final row count matches 3 plus the number of inserts that actually committed. - */ - private def surfaceConcAppendAppendCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.conc.appendAppend") { table => - val failureCount = - new java.util.concurrent.atomic.AtomicInteger(0) - def writer(base: Int): () => Unit = () => - (0 until 3).foreach { offset => - val value = base + offset - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - s"(CAST($value AS BIGINT), $value, 'row-c', 1.5, " + - "true, '2024-01-09-01')") - } catch { - case exception: Throwable => - assert( - isTypedCommitConflict(exception), - "concurrent append failed with an untyped error: " + - s"${exception.getClass.getName}") - failureCount.incrementAndGet() - } - } - val threadErrors = - runConcurrently(Seq(writer(100), writer(200))) - val expectedRowCount = 3 + 6 - failureCount.get - val actualRowCount = countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") - - assert( - threadErrors.isEmpty, - s"writer thread failed outside the insert loop: $threadErrors") - assert( - actualRowCount == expectedRowCount.toString, - s"expected $expectedRowCount rows, got $actualRowCount") - println( - s"DIAG conc.appendAppend: ${failureCount.get}/6 inserts " + - "hit a typed commit conflict") - } - - /** - * Two threads concurrently UPDATE the same row to different values; the row count stays at 3, and the final value is - * one of the two competing updates or the original seed value, with any failure being a typed commit conflict. - */ - private def surfaceConcUpdateUpdateCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.conc.updateUpdate") { table => - val column = Core.string0.columnName - def updater(value: String): () => Unit = () => - try { - table.spark.sql( - s"UPDATE ${table.name} SET $column = '$value' " + - s"WHERE ${Core.long0.columnName} = 2") - } catch { - case exception: Throwable => - assert( - isTypedCommitConflict(exception), - "concurrent update failed with an untyped error: " + - s"${exception.getClass.getName}") - } - val threadErrors = - runConcurrently(Seq(updater("AAA"), updater("BBB"))) - val finalValue = table.spark - .sql( - s"SELECT $column FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 2") - .collect()(0) - .getString(0) - - assert( - threadErrors.isEmpty, - s"updater thread failed with a non-conflict error: $threadErrors") - assert( - finalValue == "AAA" || - finalValue == "BBB" || - finalValue == "row-2", - s"concurrent updates produced a torn value: $finalValue") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "concurrent updates should not change row count") - } - - /** Two writers racing on one table. Every outcome is either a commit or a typed commit conflict. */ - def surfaceConcurrencyCases(format: String): List[Plan.Case] = - List( - surfaceConcAppendAppendCase(format), - surfaceConcUpdateUpdateCase(format)) - - /** On a side table, dropping NOT NULL from a column allows a subsequent insert of a null value for that column. */ - private def surfaceSchemaRelaxNotNullCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.schema.relaxNotNull") { table => - val sideTable = s"${table.name}_nn" - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - try { - table.spark.sql( - s"CREATE TABLE $sideTable " + - s"(id BIGINT, req INT NOT NULL) USING $dataSource") - table.spark.sql( - s"ALTER TABLE $sideTable ALTER COLUMN req DROP NOT NULL") - table.spark.sql( - s"INSERT INTO $sideTable VALUES (CAST(1 AS BIGINT), NULL)") - assert( - table.spark - .sql(s"SELECT count(*) FROM $sideTable WHERE req IS NULL") - .collect()(0) - .getLong(0) == 1, - "relaxing NOT NULL should allow a null write") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - } - } - - /** - * On a side table, widening a decimal column's precision preserves the original row and accepts a new row whose value - * only fits the wider precision. - */ - private def surfaceSchemaDecimalWidenCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.schema.decimalWiden") { table => - val sideTable = s"${table.name}_dec" - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - try { - table.spark.sql( - s"CREATE TABLE $sideTable " + - s"(id BIGINT, dec DECIMAL(10,2)) USING $dataSource") - table.spark.sql( - s"INSERT INTO $sideTable VALUES " + - "(CAST(1 AS BIGINT), CAST(12345678.99 AS DECIMAL(10,2)))") - table.spark.sql( - s"ALTER TABLE $sideTable ALTER COLUMN dec TYPE DECIMAL(12,2)") - table.spark.sql( - s"INSERT INTO $sideTable VALUES " + - "(CAST(2 AS BIGINT), CAST(1234567890.99 AS DECIMAL(12,2)))") - assert( - table.spark - .sql(s"SELECT count(*) FROM $sideTable") - .collect()(0) - .getLong(0) == 2, - "decimal widening should preserve old and new values") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - } - } - - /** - * On a side table, ADD COLUMN of a new nested struct field null-fills it for the existing row and accepts a new row - * that sets the field. - */ - private def surfaceSchemaNestedAddFieldCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.schema.nestedAddField") { table => - val sideTable = s"${table.name}_nst" - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - try { - table.spark.sql( - s"CREATE TABLE $sideTable " + - s"(id BIGINT, s STRUCT) USING $dataSource") - table.spark.sql( - s"INSERT INTO $sideTable VALUES " + - "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") - table.spark.sql( - s"ALTER TABLE $sideTable ADD COLUMN s.w INT") - assert( - table.spark - .sql(s"SELECT count(*) FROM $sideTable WHERE s.w IS NULL") - .collect()(0) - .getLong(0) == 1, - "new nested field should null-fill the existing row") - - table.spark.sql( - s"INSERT INTO $sideTable VALUES " + - "(CAST(2 AS BIGINT), " + - "named_struct('x', 2, 'y', 'b', 'w', 9))") - assert( - table.spark - .sql(s"SELECT count(*) FROM $sideTable WHERE s.w = 9") - .collect()(0) - .getLong(0) == 1, - "new nested field should be writable") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - } - } - - /** - * On a side table, ALTER TABLE DROP COLUMN of a nested struct field is rejected with an exception, and the field - * remains readable afterward. - */ - private def surfaceSchemaNestedDropFieldCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.schema.nestedDropField") { table => - val sideTable = s"${table.name}_nsd" - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - try { - table.spark.sql( - s"CREATE TABLE $sideTable " + - s"(id BIGINT, s STRUCT) USING $dataSource") - table.spark.sql( - s"INSERT INTO $sideTable VALUES " + - "(CAST(1 AS BIGINT), named_struct('x', 1, 'y', 'a'))") - Check.intercept[Exception]( - table.spark.sql( - s"ALTER TABLE $sideTable DROP COLUMN s.x")) - - assert( - table.spark - .sql(s"SELECT s.x FROM $sideTable") - .collect()(0) - .getInt(0) == 1, - "rejected nested drop should leave the field readable") - } finally { - table.spark.sql(s"DROP TABLE IF EXISTS $sideTable") - } - } - - /** ALTER TABLE ALTER COLUMN ... FIRST moves that column to the front of the schema while preserving all 3 rows. */ - private def surfaceSchemaReorderExistingCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.schema.reorderExisting") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} " + - s"ALTER COLUMN ${Core.string0.columnName} FIRST") - val columns = table.spark - .sql(s"SELECT * FROM ${table.name} LIMIT 1") - .columns - .toSeq - - assert( - columns.head == Core.string0.columnName, - s"FIRST should move the column to the front: $columns") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "column reorder should preserve the rows") - } - - /** The schema changes Iceberg allows and the ones the catalog rejects. */ - def surfaceSchemaCases(format: String): List[Plan.Case] = - List( - surfaceSchemaRelaxNotNullCase(format), - surfaceSchemaDecimalWidenCase(format), - surfaceSchemaNestedAddFieldCase(format), - surfaceSchemaNestedDropFieldCase(format), - surfaceSchemaReorderExistingCase(format)) - - /** - * The write.distribution-mode=hash property requested at creation is retained and the table holds its 3 seed rows. - */ - private def surfaceWriteDistributionHashCase(format: String): Plan.Case = - surfaceHashPreparation(format).test("surface.write.distributionHash") { table => - val properties = tableProps(table.spark, table.name) - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - properties.get("write.distribution-mode").contains("hash"), - "hash distribution mode should be retained") - assert( - rowCount == 3, - s"hash-distributed seed should contain 3 rows, got $rowCount") - } - - /** - * The write.target-file-size-bytes=1048576 property requested at creation is retained and the table holds its 3 seed - * rows. - */ - private def surfaceWriteTargetFileSizeCase(format: String): Plan.Case = - surfaceTargetFileSizePreparation(format).test("surface.write.targetFileSize") { table => - val properties = tableProps(table.spark, table.name) - val rowCount = table.spark - .sql(s"SELECT count(*) FROM ${table.name}") - .collect()(0) - .getLong(0) - - assert( - properties - .get("write.target-file-size-bytes") - .contains("1048576"), - "target file size should be retained") - assert( - rowCount == 3, - s"custom target-size seed should contain 3 rows, got $rowCount") - } - - /** The write-planning properties: distribution mode and target file size. */ - def surfaceWriteCases(format: String): List[Plan.Case] = - List( - surfaceWriteDistributionHashCase(format), - surfaceWriteTargetFileSizeCase(format)) - - /** - * register_table onto a new name makes the source table's snapshot readable there (3 rows) and leaves the source - * unchanged, and dropping the registered table leaves the source unchanged. The system.snapshot and system.add_files - * procedures each reject their unsupported inputs with an exception. - */ - private def surfacePinImportProcsCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.pin.importProcs") { table => - val registeredTable = s"${table.name}_registered" - val metadataFile = table.spark - .sql( - s"SELECT file FROM ${table.name}.metadata_log_entries " + - "ORDER BY timestamp DESC LIMIT 1") - .collect()(0) - .getString(0) - - try { - table.spark.sql( - "CALL openhouse.system.register_table(" + - s"table => '${catalogRelative(registeredTable)}', " + - s"metadata_file => '$metadataFile')") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM $registeredTable") == "3", - "register_table should make all source rows readable") - } finally { - try { - table.spark.sql( - s"DROP TABLE IF EXISTS $registeredTable") - } catch { - case NonFatal(_) => () - } - } - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name}") == "3", - "dropping the registered table should not remove source rows") - - Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.snapshot(" + - s"source_table => '${catalogRelative(table.name)}', " + - "table => 'dbMatrix.zz_snap')")) - - Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.add_files(" + - s"table => '${catalogRelative(table.name)}', " + - "source_table => '`parquet`.`/tmp/zz_nope_dir`')")) - } - - /** CREATE VIEW and ANALYZE TABLE COMPUTE STATISTICS are each rejected with an exception. */ - private def surfacePinViewsAnalyzeCase(format: String): Plan.Case = - surfaceBasePreparation(format).test("surface.pin.viewsAnalyze") { table => - Check.intercept[Exception]( - table.spark.sql( - "CREATE VIEW openhouse.dbMatrix.zz_v1 AS SELECT 1 AS one")) - - Check.intercept[Exception]( - table.spark.sql( - s"ANALYZE TABLE ${table.name} COMPUTE STATISTICS")) - } - - /** Pins on the surfaces the catalog rejects: the import procedures, views and ANALYZE TABLE. */ - def surfacePinCases(format: String): List[Plan.Case] = - List( - surfacePinImportProcsCase(format), - surfacePinViewsAnalyzeCase(format)) -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TableEvolutionCompatibilityScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TableEvolutionCompatibilityScenarios.scala new file mode 100644 index 000000000..d4252b1be --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TableEvolutionCompatibilityScenarios.scala @@ -0,0 +1,170 @@ +package harness + +/** + * One alteration a table can carry into the follow-up operations: the case-ID prefix its preparations contribute, the + * preparation step that applies it, and the ALTER TABLE statement that step runs. + */ +private[harness] final case class TableAlteration( + casePrefix: String, + stepLabel: String, + statement: String => String +) + +/** + * Table evolution compatibility: after a table has been altered, the reads, writes, snapshot operations and + * maintenance procedures that worked before the alteration still work. + * + * Operations: INSERT INTO after the alteration, row-level DELETE after the alteration, a VERSION AS OF read of the + * pre-alteration snapshot, rollback_to_snapshot back to that snapshot, expire_snapshots down to the newest snapshot, + * and rewrite_data_files over the files written across the alteration. + * + * Preparation axes: the four Parquet and ORC core layouts (each format crossed with unpartitioned and + * date-partitioned), each seeded with the standard rows and then altered in one of four ways: ADD COLUMN cc int, + * widening foo_col_int from int to bigint, WRITE ORDERED BY foo_col_long, or setting write.distribution-mode to + * range. That is 16 preparations. + * + * Case families: six families over 16 preparations, contributing 96 cases. + */ +trait TableEvolutionCompatibilityScenarios extends ScenarioKit { + + /** Every follow-up operation on every altered preparation, one preparation at a time. */ + lazy val tableEvolutionCompatibilityCases: List[Plan.Case] = + alteredTablePreparations.flatMap(preparation => + List( + insertCase(preparation), + deleteCase(preparation), + timeTravelCase(preparation), + rollbackCase(preparation), + expireSnapshotsCase(preparation), + rewriteDataFilesCase(preparation))) + + /** + * One preparation per Parquet and ORC layout and per alteration: the table is created, seeded with the standard + * rows, then altered. Each alteration carries its own step label and case-ID prefix, so a case ID names the + * alteration it ran after. Plan walks this list so every family lands on one preparation before the next + * preparation starts. + */ + lazy val alteredTablePreparations: List[TablePreparation[CoreTable.type]] = + parquetAndOrcLayouts.flatMap { layout => + alterations.map { alteration => + TablePreparation( + layout.label, + create(layout) + .insert(standardSeedRowCount)() + .sql(alteration.stepLabel)(alteration.statement)(), + alteration.casePrefix) + } + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** The four alterations the follow-up operations run after, in the order Plan walks them. */ + private val alterations: List[TableAlteration] = + List( + TableAlteration( + "afterAddColumn:", + "addColumn", + table => s"ALTER TABLE $table ADD COLUMN cc int"), + TableAlteration( + "afterTypeWiden:", + "widenIntColumnToBigint", + table => s"ALTER TABLE $table ALTER COLUMN ${Core.int0.columnName} TYPE bigint"), + TableAlteration( + "afterWriteOrder:", + "writeOrderedByLongKey", + table => s"ALTER TABLE $table WRITE ORDERED BY ${Core.long0.columnName}"), + TableAlteration( + "afterDistributionMode:", + "setRangeDistributionMode", + table => + s"ALTER TABLE $table SET TBLPROPERTIES ('write.distribution-mode'='range')")) + + /** A plain INSERT still lands on the table after the alteration, taking it to four rows. */ + private def insertCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("insert") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", + "table is not writable after the alteration") + } + + /** A row-level DELETE still lands on the table after the alteration, taking it to two rows. */ + private def deleteCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("delete") { table => + table.spark.sql( + s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "2", + "mutation failed after the alteration") + } + + /** The seed snapshot from before the alteration is still readable through VERSION AS OF and returns its 3 rows. */ + private def timeTravelCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("timeTravel") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF $seedSnapshotId") == "3", + "seed snapshot is not readable after the alteration") + } + + /** + * rollback_to_snapshot back to the seed snapshot undoes an INSERT made after the alteration and returns the table to + * its three seed rows. + */ + private def rollbackCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("rollback") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $seedSnapshotId)") + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", + "rollback across the alteration failed") + } + + /** expire_snapshots retaining only the newest snapshot leaves the table readable with its four current rows. */ + private def expireSnapshotsCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("expireSnapshots") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + "older_than => TIMESTAMP '2999-01-01 00:00:00', " + + "retain_last => 1)") + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", + "table is unreadable after snapshot expiration") + } + + /** rewrite_data_files compacts the files written across the alteration and preserves the four current rows. */ + private def rewriteDataFilesCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("rewriteDataFiles") { table => + table.spark.sql( + s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 1") + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('min-input-files', '2'))") + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", + "compaction changed rows after the alteration") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TablePropertyScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TablePropertyScenarios.scala new file mode 100644 index 000000000..4e024d7a4 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TablePropertyScenarios.scala @@ -0,0 +1,146 @@ +package harness + +import org.apache.iceberg.exceptions.BadRequestException + +/** + * Table properties: which properties a table keeps as written, which ones the catalog owns and overrides, and which + * ones it refuses to change. + * + * Operations: SET and UNSET TBLPROPERTIES for a user property; SET TBLPROPERTIES on the reserved openhouse.tableUUID + * property; reading format-version back from a table created with format-version=1; reading + * write.metadata.previous-versions-max back from a table that requested 7; reading write.target-file-size-bytes back + * from a table that requested 1048576; and SET TBLPROPERTIES on openhouse.tableType. + * + * Preparation axes: in each of the two columnar formats, the standard seeded core table for the two families that + * change properties after creation, plus one purpose-built table per family that asserts a property requested at + * creation. + * + * Case families: six families contributing 12 cases. + */ +trait TablePropertyScenarios extends ScenarioKit { + + /** Every table-property case, one file format at a time. */ + lazy val tablePropertyCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + userRoundTripCase(preparedStandardTable(format)), + reservedPropertyRejectedCase(preparedStandardTable(format)), + tableTypeImmutableCase(preparedStandardTable(format)), + formatVersionForcedCase(format), + previousVersionsHonoredCase(format), + targetFileSizeCase(format)) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** SET TBLPROPERTIES adds a user property that reads back, and UNSET TBLPROPERTIES removes it. */ + private def userRoundTripCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("tableProperty.userRoundTrip") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('my_key'='my_val')") + assert( + tableProps(table.spark, table.name).get("my_key").contains("my_val"), + "user property was not set") + + table.spark.sql(s"ALTER TABLE ${table.name} UNSET TBLPROPERTIES ('my_key')") + assert( + !tableProps(table.spark, table.name).contains("my_key"), + "user property was not removed") + } + + /** + * SET TBLPROPERTIES on the reserved openhouse.tableUUID property is rejected with a BadRequestException about the + * restriction. + */ + private def reservedPropertyRejectedCase( + preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("tableProperty.reservedOpenhouse.rejected") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + + "'openhouse.tableUUID'='deadbeef')")) + + assert( + exception.getMessage.toLowerCase.contains("restriction"), + s"unexpected message: ${exception.getMessage.take(200)}") + } + + /** + * ALTER TABLE SET TBLPROPERTIES ('openhouse.tableType'='REPLICA_TABLE') is rejected with a BadRequestException, + * since the table type is fixed at creation. + */ + private def tableTypeImmutableCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("tableProperty.tableTypeImmutable") { table => + val exception = Check.intercept[BadRequestException]( + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES (" + + "'openhouse.tableType'='REPLICA_TABLE')")) + + assert( + exception.getMessage.contains("restriction"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + + /** + * Even though format-version=1 was requested at creation, the catalog stores the table at format-version=2 and the + * table remains writable there. + */ + private def formatVersionForcedCase(format: String): Plan.Case = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'format-version'='1')")() + .insert(standardSeedRowCount)()) + .test("tableProperty.formatVersionForced") { table => + val formatVersion = tableProps(table.spark, table.name).get("format-version") + + assert( + formatVersion.contains("2"), + s"expected the catalog to store format-version=2, got $formatVersion") + assert( + table.rows.size == standardSeedRowCount, + "table not writable at the stored format-version") + } + + /** The write.metadata.previous-versions-max property requested at creation is honored and reads back as 7. */ + private def previousVersionsHonoredCase(format: String): Plan.Case = + TablePreparation( + format, + TableTest(Core).sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'write.metadata.previous-versions-max'='7')")()) + .test("tableProperty.previousVersionsHonored") { table => + val previousVersions = + tableProps(table.spark, table.name).get("write.metadata.previous-versions-max") + + assert( + previousVersions.contains("7"), + s"expected previous-versions-max=7, got $previousVersions") + } + + /** + * The write.target-file-size-bytes=1048576 property requested at creation is retained and the table holds its 3 seed + * rows. + */ + private def targetFileSizeCase(format: String): Plan.Case = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'write.target-file-size-bytes'='1048576')")() + .insert(standardSeedRowCount)()) + .test("tableProperty.targetFileSize") { table => + assert( + tableProps(table.spark, table.name) + .get("write.target-file-size-bytes") + .contains("1048576"), + "target file size should be retained") + assert( + table.rows.size == standardSeedRowCount, + "the custom target-size table should hold its seed rows") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TimeTravelScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TimeTravelScenarios.scala new file mode 100644 index 000000000..0dbdd40ed --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TimeTravelScenarios.scala @@ -0,0 +1,98 @@ +package harness + +/** + * Time travel: reading a table as it stood at an earlier snapshot, by snapshot ID, by commit timestamp, and after the + * schema has moved on. + * + * Operations: VERSION AS OF each snapshot ID, TIMESTAMP AS OF the first commit's timestamp, and a VERSION AS OF read + * of the pre-evolution snapshot after ADD COLUMN and an insert into the new column. + * + * Preparation axes: in each of the two columnar formats, the two-snapshot core table for the snapshot and timestamp + * families, and the standard seeded core table for the schema-evolution family. + * + * Case families: three families contributing 6 cases. + */ +trait TimeTravelScenarios extends ScenarioKit { + + /** Every time-travel case, one file format at a time. */ + lazy val timeTravelCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + versionAsOfCase(preparedTwoSnapshotTable(format)), + timestampAsOfCase(preparedTwoSnapshotTable(format)), + afterAddColumnCase(preparedStandardTable(format))) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * VERSION AS OF the first snapshot ID reads the 3 rows the seed commit wrote, and VERSION AS OF the second reads all + * 5 rows. + */ + private def versionAsOfCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("timeTravel.versionAsOf") { table => + val snapshots = snapshotIds(table.spark, table.name) + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF ${snapshots(0)}") == "3") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF ${snapshots(1)}") == "5") + } + + /** TIMESTAMP AS OF the first commit's time reads the 3 rows that commit wrote. */ + private def timestampAsOfCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("timeTravel.timestampAsOf") { table => + val firstCommitTimestamp = table.spark + .sql( + s"SELECT CAST(committed_at AS STRING) FROM ${table.name}.snapshots " + + "ORDER BY committed_at LIMIT 1") + .collect()(0) + .getString(0) + + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} TIMESTAMP AS OF '$firstCommitTimestamp'") == "3") + } + + /** + * After ADD COLUMN and an insert into the new column, time travel to the pre-evolution snapshot reads the old schema + * with 3 rows, while a current read sees the new column. + */ + private def afterAddColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("timeTravel.afterAddColumn") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).last + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + table.spark.sql( + s"INSERT INTO ${table.name} VALUES $extraColInsert9") + val currentColumns = table.spark + .sql(s"SELECT * FROM ${table.name} LIMIT 1") + .columns + .toSeq + val historicalColumns = table.spark + .sql( + s"SELECT * FROM ${table.name} " + + s"VERSION AS OF $seedSnapshotId LIMIT 1") + .columns + .toSeq + + assert( + currentColumns.contains("extra_col"), + s"current read is missing the evolved column: $currentColumns") + assert( + !historicalColumns.contains("extra_col") && + historicalColumns.size == Core.tableColumns.size, + s"time travel should use the snapshot schema: $historicalColumns") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF $seedSnapshotId") == "3", + "pre-evolution snapshot should contain 3 rows") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriteDistributionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriteDistributionScenarios.scala new file mode 100644 index 000000000..06c88dc44 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriteDistributionScenarios.scala @@ -0,0 +1,161 @@ +package harness + +/** + * Write distribution: the write.distribution-mode a table is configured with is retained, and it decides how a single + * append is laid out on disk without changing the rows the table holds. + * + * Operations: creating a table with an explicit write.distribution-mode of none and of hash and reading the property + * back; appending one multi-task DataFrame into a four-partition table under each mode and comparing the rows and the + * data-file counts the two modes produce. + * + * Preparation axes: for the two retained-property families, the standard three-row seed in each of the two columnar + * formats, unpartitioned for none and date-partitioned for hash. The layout family builds its own four-partition + * tables in each format, because it needs one table per mode inside a single case. + * + * Case families: three families contributing 6 cases. + */ +trait WriteDistributionScenarios extends ScenarioKit { + + /** Every write-distribution case, one file format at a time. */ + lazy val writeDistributionCases: List[Plan.Case] = + standardFormats.flatMap { format => + List( + Plan.Case(s"writeDistribution.noneVersusHash @ $format", noneVersusHashCase(format)), + noneRetainedCase(format), + hashRetainedCase(format)) + } + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + // The two tables the layout case compares hold 400 rows spread over 4 table partitions, written from 8 input tasks + // that each hold rows for every partition. + private val distributionPartitionCount = 4 + private val distributionInputTaskCount = 8 + private val distributionRowCount = 400 + + /** + * The same multi-task append under an explicit write.distribution-mode of none and of hash keeps the mode each table + * was configured with and lands the same logical rows in both, while producing the physical layout each mode + * defines. Under none every input task writes every partition it holds, so one append produces up to (input tasks + * times partitions) data files. Under hash the writer shuffles rows so one task owns each partition, clustering the + * append to about one file per partition. + * + * The comparison needs both tables live at once, so the case nests one owned-table lifecycle inside the other. Each + * table carries a generated UUID and counter name, each lifecycle takes ownership the moment its CREATE returns, and + * each drops the one table it owns. A failure while building or appending to the hash table therefore still drops + * the none table, and the failure the case reports stays the primary one with any cleanup failure suppressed + * behind it. + */ + private def noneVersusHashCase(format: String)(ctx: Ctx): Unit = { + val spark = ctx.spark + + def createUnder(mode: String, table: String): Unit = + spark.sql( + s"CREATE TABLE $table (id bigint, p int) USING $dataSource PARTITIONED BY (p) " + + "TBLPROPERTIES ('format-version'='2', " + + s"'write.format.default'='$format', 'write.distribution-mode'='$mode')") + + def appendInputRows(table: String): Unit = + spark + .range(0, distributionRowCount.toLong) + .selectExpr("id", s"cast(id % $distributionPartitionCount as int) as p") + .repartition(distributionInputTaskCount) + .writeTo(table) + .append() + + def rowsOf(table: String): Seq[(Long, Int)] = + spark + .sql(s"SELECT id, p FROM $table ORDER BY id") + .collect() + .toSeq + .map(row => (row.getLong(0), row.getInt(1))) + + def dataFileCountOf(table: String): Long = + spark.sql(s"SELECT count(*) FROM $table.data_files").collect()(0).getLong(0) + + val noneTable = TableTest.nextQualifiedTableName(ctx.namespace) + val hashTable = TableTest.nextQualifiedTableName(ctx.namespace) + + withOwnedTable(spark.sql(_), noneTable)(createUnder("none", noneTable)) { + appendInputRows(noneTable) + + withOwnedTable(spark.sql(_), hashTable)(createUnder("hash", hashTable)) { + appendInputRows(hashTable) + + assert( + tableProps(spark, noneTable).get("write.distribution-mode").contains("none"), + s"[$format] the none table should retain write.distribution-mode=none") + assert( + tableProps(spark, hashTable).get("write.distribution-mode").contains("hash"), + s"[$format] the hash table should retain write.distribution-mode=hash") + + val noneRows = rowsOf(noneTable) + assert( + noneRows.size == distributionRowCount, + s"[$format] the none table should hold $distributionRowCount rows, got ${noneRows.size}") + assert( + noneRows == rowsOf(hashTable), + s"[$format] the two distribution modes should land the same logical rows") + + val noneFileCount = dataFileCountOf(noneTable) + val hashFileCount = dataFileCountOf(hashTable) + println( + s"DIAG writeDistribution.noneVersusHash[$format]: noneFiles=$noneFileCount " + + s"hashFiles=$hashFileCount partitions=$distributionPartitionCount " + + s"inputTasks=$distributionInputTaskCount") + assert( + hashFileCount <= distributionPartitionCount * 2, + s"[$format] hash should cluster to about $distributionPartitionCount files, " + + s"got $hashFileCount") + assert( + noneFileCount > hashFileCount && + noneFileCount <= distributionPartitionCount.toLong * distributionInputTaskCount, + s"[$format] none should spread the append across more files than hash and at most " + + s"${distributionPartitionCount * distributionInputTaskCount} " + + s"(none=$noneFileCount hash=$hashFileCount)") + } + } + } + + /** The write.distribution-mode=none requested at creation is retained and the table holds its 3 seed rows. */ + private def noneRetainedCase(format: String): Plan.Case = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'write.distribution-mode'='none')")() + .insert(standardSeedRowCount)()) + .test("writeDistribution.noneRetained") { table => + assert( + tableProps(table.spark, table.name).get("write.distribution-mode").contains("none"), + "distribution-mode none should be retained") + assert( + table.rows.size == standardSeedRowCount, + "the table should hold its seed rows under distribution-mode none") + } + + /** + * The write.distribution-mode=hash requested at creation on a date-partitioned table is retained and the table holds + * its 3 seed rows. + */ + private def hashRetainedCase(format: String): Plan.Case = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"PARTITIONED BY (${Core.date0.columnName}) " + + "TBLPROPERTIES (" + + s"'write.format.default'='$format', 'write.distribution-mode'='hash')")() + .insert(standardSeedRowCount)()) + .test("writeDistribution.hashRetained") { table => + assert( + tableProps(table.spark, table.name).get("write.distribution-mode").contains("hash"), + "distribution-mode hash should be retained") + assert( + table.rows.size == standardSeedRowCount, + "the table should hold its seed rows under distribution-mode hash") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriterCompatibilityScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriterCompatibilityScenarios.scala new file mode 100644 index 000000000..a895210b8 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriterCompatibilityScenarios.scala @@ -0,0 +1,50 @@ +package harness + +import org.apache.spark.sql.AnalysisException + +/** + * Writer compatibility: how a writer that names every column explicitly behaves after the table's column list has + * grown. + * + * Operations: an explicit-column INSERT that lists the six core columns, run once before ADD COLUMN and once after. + * The catalog accepts it before and rejects it after, naming the column the statement omits. + * + * Preparation axes: the standard seeded core table in each of the two columnar formats. + * + * Case families: one family contributing 2 cases. + */ +trait WriterCompatibilityScenarios extends ScenarioKit { + + /** The explicit-column writer case, one file format at a time. */ + lazy val writerCompatibilityCases: List[Plan.Case] = + standardFormats.map(format => afterAddColumnCase(preparedStandardTable(format))) + + // --- the preparations, shared helpers and case bodies the surface above composes --- + + /** + * An explicit-column INSERT that worked before ADD COLUMN is rejected afterward, with an error naming the new + * column. + */ + private def afterAddColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation.test("writerCompatibility.afterAddColumn") { table => + val writerStatement = + s"INSERT INTO ${table.name} ($columnNameList) VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')" + table.spark.sql(writerStatement) + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", + "explicit-column writer should work before schema evolution") + + table.spark.sql( + s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") + val exception = Check.intercept[AnalysisException]( + table.spark.sql(writerStatement)) + assert( + exception.getMessage.contains("extra_col") && + (exception.getMessage.contains("CANNOT_FIND_DATA") || + exception.getMessage.toLowerCase.contains("cannot find data")), + "a pre-evolution explicit-column writer is rejected after ADD COLUMN: " + + exception.getMessage.take(160)) + } + +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala index 9870804d5..ec42160b1 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala @@ -6,15 +6,63 @@ import java.security.MessageDigest import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} import org.junit.jupiter.api.Test +/** + * Pins the ordered catalog: its size, its fingerprint, the uniqueness of its IDs, the capability naming rule, and the + * rule that every capability contributes exactly once. Reading the catalog does not execute a case or start Spark. + */ final class CaseCatalogTest { - private val expectedCaseCount = 1181 + private val expectedCaseCount = 1177 private val expectedCatalogSha256 = - "377f65959e3034c51e078fea72491444b06a6055f37c051184bdc379234b3d57" + "a10676c9fe0af5169459a0c9ad74eb2d005b7c7c63e2b7c7d4364bb7d8cc5bb9" + + /** + * Every capability the standard catalog is built from, in the order Plan integrates them. This list is written out + * here rather than derived from `Plan.contributions`, so adding, dropping, renaming or reordering a capability fails + * this test until the intended catalog shape is restated. + */ + private val expectedContributionNames = List( + "accessControlCases", + "changelogCases", + "columnTagCases", + "compactionPlanningCases", + "concurrencyCases", + "dataTypeCases", + "dmlCases", + "dmlValidationCases", + "encryptionCases", + "fileFormatCases", + "fileReplicationCases", + "incrementalReadCases", + "lockingCases", + "maintenanceCases", + "metadataTableCases", + "namespaceCases", + "nestedTypeCases", + "partitionEvolutionCases", + "partitionTransformCases", + "procedureCases", + "renameCases", + "scanPlanningCases", + "schemaEvolutionCases", + "snapshotRestoreCases", + "sortOrderCases", + "streamingCases", + "tableEvolutionCompatibilityCases", + "tablePropertyCases", + "timeTravelCases", + "writeDistributionCases", + "writerCompatibilityCases") + + /** + * Case-ID prefixes that name where a case came from rather than the capability it covers. Every case ID is owned by + * the capability trait that defines it, so none of these appears in the catalog. + */ + private val provenanceCaseIdPrefixes = + List("fork.", "hazard.", "readerWriter.", "surface.", "interact.") @Test def orderedCaseCatalogMatchesBaseline(): Unit = { - val cases = Plan.cases - val caseIds = cases.map(_.id) + val caseIds = Plan.caseIds val actualCatalogSha256 = sha256(caseIds.mkString("\n")) val duplicateCaseIds = caseIds.groupBy(identity).collect { case (caseId, occurrences) if occurrences.size > 1 => caseId @@ -33,6 +81,52 @@ final class CaseCatalogTest { s"ordered case catalog changed; count=${caseIds.size}, sha256=$actualCatalogSha256") } + @Test + def everyCaseIdNamesTheCapabilityItCovers(): Unit = { + val provenanceNamedCaseIds = + Plan.caseIds.filter(caseId => provenanceCaseIdPrefixes.exists(caseId.startsWith)) + + assertTrue( + provenanceNamedCaseIds.isEmpty, + "a case ID names a capability, not the bucket it came from; " + + s"offenders=${provenanceNamedCaseIds.mkString(", ")}") + } + + @Test + def theCatalogIntegratesExactlyTheIntendedCapabilities(): Unit = { + val contributionNames = Plan.contributions.map { case (name, _) => name } + + assertEquals( + expectedContributionNames, + contributionNames, + "Plan integrates a different set or order of capabilities than the catalog declares") + assertEquals( + expectedContributionNames.distinct.size, + expectedContributionNames.size, + "the declared capability list names a capability more than once") + } + + @Test + def eachCapabilityContributesExactlyOnceInOrder(): Unit = { + val contributionNames = Plan.contributions.map { case (name, _) => name } + + assertEquals( + contributionNames.distinct, + contributionNames, + s"a capability contribution is integrated more than once: $contributionNames") + assertEquals( + contributionNames.sorted, + contributionNames, + s"capability contributions are integrated in alphabetical order: $contributionNames") + assertEquals( + Plan.contributions.flatMap { case (_, contribution) => contribution.map(_.id) }, + Plan.caseIds, + "the catalog is exactly its named contributions, concatenated in order") + assertTrue( + Plan.contributions.forall { case (_, contribution) => contribution.nonEmpty }, + "every named contribution supplies at least one case") + } + private def sha256(value: String): String = MessageDigest .getInstance("SHA-256") diff --git a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala index 2e8761f90..4c299bfa6 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala @@ -163,7 +163,7 @@ final class DmlCaseCatalogTest { Scenarios.orderedDmlCases, Scenarios.evolvedDmlCases, Scenarios.partitionedDmlCases, - Scenarios.layoutFormatCases).flatten + Scenarios.fileFormatCases).flatten val caseIds = describedBuckets.map(_.id) caseIds.foreach { caseId => @@ -177,6 +177,11 @@ final class DmlCaseCatalogTest { @Test def eachLayoutListCrossesItsFormatsWithItsPartitionings(): Unit = { + assertEquals(List("parquet", "orc", "avro"), Scenarios.fileFormats) + assertEquals(List("parquet", "orc"), Scenarios.standardFormats) + assertTrue( + Scenarios.standardFormats.forall(Scenarios.fileFormats.contains), + "the standard formats are drawn from the full file-format list") assertEquals( List( "unpartitioned/parquet", @@ -196,6 +201,39 @@ final class DmlCaseCatalogTest { "unpartitioned/orc", "partitioned/orc"), Scenarios.parquetAndOrcLayouts.map(_.label)) + assertEquals( + Scenarios.fileFormats.map(format => s"nested-unpartitioned/$format"), + Scenarios.nestedLayouts.map(_.label)) + assertEquals( + Scenarios.fileFormats.map(format => s"types-unpartitioned/$format"), + Scenarios.typesLayouts.map(_.label)) + } + + @Test + def everyPreparationLabelDrawsItsFormatFromTheStandardLists(): Unit = { + // A preparation label is either a layout path whose last segment is a file format, or one of the two labels for a + // case that owns no core table: `core` for an API-level case and `embedded` for a control-plane case. + val allowedLabelSuffixes = Scenarios.fileFormats ++ List("core", "embedded") + val unknownLabels = Plan.caseIds + .map(caseId => caseId.split(" @ ").last) + .map(label => label.split("/").last) + .distinct + .filterNot(allowedLabelSuffixes.contains) + + assertTrue( + unknownLabels.isEmpty, + "every preparation label names a format from ScenarioKit.fileFormats; " + + s"offenders=${unknownLabels.mkString(", ")}") + } + + @Test + def dmlCasesIsTheFourDmlBucketsInPreparationOrder(): Unit = { + assertEquals( + (Scenarios.coreDmlCases ++ + Scenarios.partitionedDmlCases ++ + Scenarios.orderedDmlCases ++ + Scenarios.evolvedDmlCases).map(_.id), + Scenarios.dmlCases.map(_.id)) } @Test @@ -205,17 +243,37 @@ final class DmlCaseCatalogTest { "format.materialization describes the preparation, not an operation run against it") assertEquals( caseIds(Scenarios.layoutFormatPreparations, "format.materialization"), - Scenarios.layoutFormatCases.map(_.id)) + Scenarios.fileFormatCases.map(_.id)) } @Test def eachBucketIsThePreparationListCrossedWithItsTestCaseList(): Unit = { val noNullStringPreparations = List.empty[TablePreparation[CoreTable.type]] val buckets = List( - ("coreDmlCases", Scenarios.coreDmlCases, Scenarios.preparedCoreTables, Scenarios.allDmlTestCases, Scenarios.preparedNullStringCoreTables), - ("orderedDmlCases", Scenarios.orderedDmlCases, Scenarios.preparedOrderedCoreTables, Scenarios.allDmlTestCases, Scenarios.preparedNullStringOrderedCoreTables), - ("evolvedDmlCases", Scenarios.evolvedDmlCases, Scenarios.preparedEvolvedCoreTables, Scenarios.testCasesCompatibleWithAnAddedColumn, noNullStringPreparations), - ("partitionedDmlCases", Scenarios.partitionedDmlCases, Scenarios.preparedPartitionedCoreTables, Scenarios.partitionedTableTestCases, noNullStringPreparations)) + ( + "coreDmlCases", + Scenarios.coreDmlCases, + Scenarios.preparedCoreTables, + Scenarios.allDmlTestCases, + Scenarios.preparedNullStringCoreTables), + ( + "orderedDmlCases", + Scenarios.orderedDmlCases, + Scenarios.preparedOrderedCoreTables, + Scenarios.allDmlTestCases, + Scenarios.preparedNullStringOrderedCoreTables), + ( + "evolvedDmlCases", + Scenarios.evolvedDmlCases, + Scenarios.preparedEvolvedCoreTables, + Scenarios.testCasesCompatibleWithAnAddedColumn, + noNullStringPreparations), + ( + "partitionedDmlCases", + Scenarios.partitionedDmlCases, + Scenarios.preparedPartitionedCoreTables, + Scenarios.partitionedTableTestCases, + noNullStringPreparations)) buckets.foreach { case (bucketName, bucket, preparations, testCases, nullStringPreparations) => val expectedIds = diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TableLifecycleTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TableLifecycleTest.scala new file mode 100644 index 000000000..7ccd5181d --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/TableLifecycleTest.scala @@ -0,0 +1,272 @@ +package harness + +import org.junit.jupiter.api.Assertions.{assertEquals, assertSame, assertThrows, assertTrue} +import org.junit.jupiter.api.Test +import scala.collection.mutable.ListBuffer + +/** + * A `ScenarioKit` whose catalog statements are recorded instead of executed, so a test drives the real lifecycle + * boundaries without a Spark session. `failingStatements` names the substrings whose statements throw, which is how a + * test injects a create, rename or cleanup failure. + */ +private final class RecordingScenarioKit extends ScenarioKit { + val statements = ListBuffer.empty[String] + var failingStatements: List[String] = Nil + + val runStatement: String => Unit = statement => { + statements += statement + failingStatements + .find(statement.contains) + .foreach(failing => throw new IllegalStateException(s"statement rejected: $failing")) + } + + def ownedTable(table: String)(create: => Unit)(use: => Unit): Unit = + withOwnedTable(runStatement, table)(create)(use) + + def cleanupStatement(statement: String)(use: => Unit): Unit = + withCleanupStatement(runStatement, statement)(use) + + def trackedRename(originalTable: String)(use: (String => Unit) => Unit): Unit = + withTrackedRename(runStatement, originalTable)(use) + + def heldLock(lock: () => (Int, String), unlock: () => (Int, String))( + use: (() => Unit) => Unit): Unit = + withTableLock(lock, unlock)(use) +} + +/** + * Pins the lifecycle boundaries a case uses for an artifact it builds for itself: the owned table, the unconditional + * cleanup around a rejected create, the rename tracker, and the lock. Every test drives the boundary in `ScenarioKit` + * itself and injects the failure it is about, so a boundary that stopped cleaning up, cleaned up the wrong artifact, + * or swallowed a failure is caught here. + */ +final class TableLifecycleTest { + private val ok: () => (Int, String) = () => (200, "") + + @Test + def anOwnedTableIsDroppedWhenItsCreateSucceeds(): Unit = { + val kit = new RecordingScenarioKit + + kit.ownedTable("db.t_owned")(kit.runStatement("CREATE TABLE db.t_owned"))( + kit.runStatement("SELECT 1")) + + assertEquals( + List("CREATE TABLE db.t_owned", "SELECT 1", "DROP TABLE IF EXISTS db.t_owned"), + kit.statements.toList) + } + + @Test + def anOwnedTableIsNotDroppedWhenItsCreateFails(): Unit = { + val kit = new RecordingScenarioKit + kit.failingStatements = List("CREATE TABLE db.t_conflict") + + val thrown = assertThrows( + classOf[IllegalStateException], + () => + kit.ownedTable("db.t_conflict")(kit.runStatement("CREATE TABLE db.t_conflict"))( + kit.runStatement("SELECT 1"))) + + assertTrue(thrown.getMessage.contains("CREATE TABLE db.t_conflict")) + assertEquals(List("CREATE TABLE db.t_conflict"), kit.statements.toList) + } + + @Test + def anOwnedTableBodyFailureStaysPrimaryWhenItsDropAlsoFails(): Unit = { + val kit = new RecordingScenarioKit + kit.failingStatements = List("DROP TABLE IF EXISTS db.t_owned") + val bodyFailure = new Exception("body failed") + + val thrown = assertThrows( + classOf[Exception], + () => + kit.ownedTable("db.t_owned")(kit.runStatement("CREATE TABLE db.t_owned"))(throw bodyFailure)) + + assertSame(bodyFailure, thrown) + assertEquals(1, thrown.getSuppressed.length) + assertTrue(thrown.getSuppressed.head.getMessage.contains("DROP TABLE IF EXISTS db.t_owned")) + assertEquals( + List("CREATE TABLE db.t_owned", "DROP TABLE IF EXISTS db.t_owned"), + kit.statements.toList) + } + + @Test + def anOwnedTableDropFailureSurfacesWhenTheBodySucceeds(): Unit = { + val kit = new RecordingScenarioKit + kit.failingStatements = List("DROP TABLE IF EXISTS db.t_owned") + + val thrown = assertThrows( + classOf[IllegalStateException], + () => kit.ownedTable("db.t_owned")(kit.runStatement("CREATE TABLE db.t_owned"))(())) + + assertTrue(thrown.getMessage.contains("DROP TABLE IF EXISTS db.t_owned")) + } + + @Test + def nestedOwnedTablesEachDropOnlyTheTableTheyCreated(): Unit = { + val kit = new RecordingScenarioKit + kit.failingStatements = List("CREATE TABLE db.t_inner") + val outerCreate = "CREATE TABLE db.t_outer" + + val thrown = assertThrows( + classOf[IllegalStateException], + () => + kit.ownedTable("db.t_outer")(kit.runStatement(outerCreate)) { + kit.ownedTable("db.t_inner")(kit.runStatement("CREATE TABLE db.t_inner"))(()) + }) + + assertTrue(thrown.getMessage.contains("CREATE TABLE db.t_inner")) + assertEquals( + List(outerCreate, "CREATE TABLE db.t_inner", "DROP TABLE IF EXISTS db.t_outer"), + kit.statements.toList) + } + + @Test + def aRejectedCreateIsCleanedUpWhateverTheRejectionDid(): Unit = { + val scratchDrop = "DROP TABLE IF EXISTS db.t_scratch" + + // The rejection arrives as expected. + val expectedKit = new RecordingScenarioKit + expectedKit.failingStatements = List("CREATE TABLE db.t_scratch") + expectedKit.cleanupStatement(scratchDrop) { + Check.intercept[IllegalStateException]( + expectedKit.runStatement("CREATE TABLE db.t_scratch")) + } + assertEquals(List("CREATE TABLE db.t_scratch", scratchDrop), expectedKit.statements.toList) + + // The create unexpectedly succeeds, so the interception fails and the scratch table still goes. + val unexpectedSuccessKit = new RecordingScenarioKit + val successThrown = assertThrows( + classOf[AssertionError], + () => + unexpectedSuccessKit.cleanupStatement(scratchDrop) { + Check.intercept[IllegalStateException]( + unexpectedSuccessKit.runStatement("CREATE TABLE db.t_scratch")) + }) + assertTrue(successThrown.getMessage.contains("to be thrown")) + assertEquals( + List("CREATE TABLE db.t_scratch", scratchDrop), + unexpectedSuccessKit.statements.toList) + + // The create throws a different type, so the interception fails and the scratch table still goes. + val wrongTypeKit = new RecordingScenarioKit + val wrongTypeThrown = assertThrows( + classOf[AssertionError], + () => + wrongTypeKit.cleanupStatement(scratchDrop) { + Check.intercept[IllegalArgumentException](throw new IllegalStateException("other")) + }) + assertTrue(wrongTypeThrown.getMessage.contains("but got")) + assertEquals(List(scratchDrop), wrongTypeKit.statements.toList) + + // The assertion after the interception fails, and its failure stays primary over the cleanup failure. + val assertionKit = new RecordingScenarioKit + assertionKit.failingStatements = List(scratchDrop) + val assertionFailure = new Exception("message assertion failed") + val assertionThrown = assertThrows( + classOf[Exception], + () => assertionKit.cleanupStatement(scratchDrop)(throw assertionFailure)) + assertSame(assertionFailure, assertionThrown) + assertEquals(1, assertionThrown.getSuppressed.length) + assertEquals(List(scratchDrop), assertionKit.statements.toList) + } + + @Test + def aTrackedRenameLeavesNothingBehindUnderTheNameItLastAccepted(): Unit = { + // The two names share no prefix, so an injected failure names exactly one of the two renames. + val originalTable = "db.t_alpha" + val renamedTable = "db.t_beta" + val renameAway = s"ALTER TABLE $originalTable RENAME TO $renamedTable" + val renameBack = s"ALTER TABLE $renamedTable RENAME TO $originalTable" + + // The case renames away and back, so the table ends under its original name and nothing is dropped. + val kit = new RecordingScenarioKit + kit.trackedRename(originalTable) { renameTo => + renameTo(renamedTable) + renameTo(originalTable) + } + assertEquals(List(renameAway, renameBack), kit.statements.toList) + + // An assertion between the two renames fails, so the live name is the renamed one and that is what goes. + val assertionKit = new RecordingScenarioKit + val assertionFailure = new Exception("row count assertion failed") + val assertionThrown = assertThrows( + classOf[Exception], + () => + assertionKit.trackedRename(originalTable) { renameTo => + renameTo(renamedTable) + throw assertionFailure + }) + assertSame(assertionFailure, assertionThrown) + assertEquals( + List(renameAway, s"DROP TABLE IF EXISTS $renamedTable"), + assertionKit.statements.toList) + + // The rename back fails, so the table is still live under the renamed name and that is what goes. + val renameBackKit = new RecordingScenarioKit + renameBackKit.failingStatements = List(renameBack) + val renameBackThrown = assertThrows( + classOf[IllegalStateException], + () => + renameBackKit.trackedRename(originalTable) { renameTo => + renameTo(renamedTable) + renameTo(originalTable) + }) + assertTrue(renameBackThrown.getMessage.contains(renameBack)) + assertEquals( + List(renameAway, renameBack, s"DROP TABLE IF EXISTS $renamedTable"), + renameBackKit.statements.toList) + + // The first rename fails, so the table never left its original name and the boundary drops nothing. + val renameAwayKit = new RecordingScenarioKit + renameAwayKit.failingStatements = List(renameAway) + val renameAwayThrown = assertThrows( + classOf[IllegalStateException], + () => renameAwayKit.trackedRename(originalTable)(renameTo => renameTo(renamedTable))) + assertTrue(renameAwayThrown.getMessage.contains(renameAway)) + assertEquals(List(renameAway), renameAwayKit.statements.toList) + } + + @Test + def aHeldLockIsReleasedOnceAndItsResponsesAreChecked(): Unit = { + // The case releases the lock itself, so the boundary does not release it again. + val releases = ListBuffer.empty[String] + new RecordingScenarioKit().heldLock(ok, () => { releases += "release"; (200, "") }) { release => + release() + } + assertEquals(List("release"), releases.toList) + + // The case leaves the lock held, so the boundary releases it. + releases.clear() + new RecordingScenarioKit().heldLock(ok, () => { releases += "release"; (200, "") })(_ => ()) + assertEquals(List("release"), releases.toList) + + // A rejected lock request fails the case before the body runs. + var bodyRan = false + val lockThrown = assertThrows( + classOf[AssertionError], + () => + new RecordingScenarioKit() + .heldLock(() => (503, "unavailable"), ok)(_ => bodyRan = true)) + assertTrue(lockThrown.getMessage.contains("lock request failed: 503")) + assertTrue(!bodyRan, "the body should not run when the lock was refused") + + // A rejected release fails the case. + val releaseThrown = assertThrows( + classOf[AssertionError], + () => new RecordingScenarioKit().heldLock(ok, () => (500, "boom"))(_ => ())) + assertTrue(releaseThrown.getMessage.contains("unlock request failed: 500")) + + // A release failure rides along behind a body failure, and the boundary tries the release exactly once. + releases.clear() + val bodyFailure = new Exception("locked-write assertion failed") + val bodyThrown = assertThrows( + classOf[Exception], + () => + new RecordingScenarioKit() + .heldLock(ok, () => { releases += "release"; (500, "boom") })(_ => throw bodyFailure)) + assertSame(bodyFailure, bodyThrown) + assertEquals(1, bodyThrown.getSuppressed.length) + assertTrue(bodyThrown.getSuppressed.head.getMessage.contains("unlock request failed: 500")) + assertEquals(List("release"), releases.toList) + } +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala index 46b09b539..f347eec3b 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala @@ -2,6 +2,7 @@ package harness import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} import org.junit.jupiter.api.Test +import scala.collection.mutable.ListBuffer /** * Pins how a preparation turns a test body into a catalog case: the ID it builds, the post-test hook every case from @@ -31,7 +32,7 @@ final class TablePreparationTest { @Test def buildsACaseWithoutRunningItsBodyOrItsPostTestHook(): Unit = { - val calls = scala.collection.mutable.ListBuffer.empty[String] + val calls = ListBuffer.empty[String] val preparation = TablePreparation[CoreTable.type]( "unpartitioned/parquet", emptyPreparation, @@ -44,7 +45,7 @@ final class TablePreparationTest { @Test def runsADmlTestCaseUnderTheIdOfThePreparationItIsGiven(): Unit = { - val calls = scala.collection.mutable.ListBuffer.empty[String] + val calls = ListBuffer.empty[String] val preparation = TablePreparation("unpartitioned/parquet", emptyPreparation) val dmlTestCase = DmlTestCase( "insert.into", diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala index e77244ebe..177085b82 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala @@ -9,6 +9,7 @@ import org.junit.jupiter.api.Assertions.{ assertTrue } import org.junit.jupiter.api.Test +import scala.collection.mutable.ListBuffer /** * Pins fresh table identity and ownership cleanup: generated names stay namespace-scoped and unique across counter @@ -81,4 +82,5 @@ final class TableTestTest { assertSame(testFailure, thrown) assertEquals(List(cleanupFailure), thrown.getSuppressed.toList) } + } From d8860404b7807633903d4d7ca37d7c7f4556f9c4 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Tue, 1 Sep 2026 18:23:44 -0700 Subject: [PATCH 13/24] refactor(delta-harness): prefix scenarios Name every scenario source and trait ScenarioFoo so scenario files group together and the framework files remain visually distinct. Preserve the catalog contributions, IDs, ordering, count, and fingerprint unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/openhouse/OpenHouseMatrix.scala | 62 +++++++++---------- ...rios.scala => ScenarioAccessControl.scala} | 2 +- ...cenarios.scala => ScenarioChangelog.scala} | 2 +- ...cenarios.scala => ScenarioColumnTag.scala} | 2 +- ...scala => ScenarioCompactionPlanning.scala} | 2 +- ...narios.scala => ScenarioConcurrency.scala} | 2 +- ...Scenarios.scala => ScenarioDataType.scala} | 2 +- .../{DmlScenarios.scala => ScenarioDml.scala} | 2 +- ...rios.scala => ScenarioDmlValidation.scala} | 2 +- ...enarios.scala => ScenarioEncryption.scala} | 2 +- ...enarios.scala => ScenarioFileFormat.scala} | 2 +- ...os.scala => ScenarioFileReplication.scala} | 2 +- ...os.scala => ScenarioIncrementalRead.scala} | 2 +- ...gScenarios.scala => ScenarioLocking.scala} | 2 +- ...narios.scala => ScenarioMaintenance.scala} | 2 +- ...rios.scala => ScenarioMetadataTable.scala} | 2 +- ...cenarios.scala => ScenarioNamespace.scala} | 2 +- ...enarios.scala => ScenarioNestedType.scala} | 2 +- ...scala => ScenarioPartitionEvolution.scala} | 2 +- ...scala => ScenarioPartitionTransform.scala} | 2 +- ...cenarios.scala => ScenarioProcedure.scala} | 2 +- ...meScenarios.scala => ScenarioRename.scala} | 2 +- ...arios.scala => ScenarioScanPlanning.scala} | 2 +- ...os.scala => ScenarioSchemaEvolution.scala} | 2 +- ...os.scala => ScenarioSnapshotRestore.scala} | 2 +- ...cenarios.scala => ScenarioSortOrder.scala} | 2 +- ...cenarios.scala => ScenarioStreaming.scala} | 2 +- ...ScenarioTableEvolutionCompatibility.scala} | 2 +- ...rios.scala => ScenarioTableProperty.scala} | 2 +- ...enarios.scala => ScenarioTimeTravel.scala} | 2 +- ....scala => ScenarioWriteDistribution.scala} | 2 +- ...cala => ScenarioWriterCompatibility.scala} | 2 +- 32 files changed, 62 insertions(+), 62 deletions(-) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{AccessControlScenarios.scala => ScenarioAccessControl.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ChangelogScenarios.scala => ScenarioChangelog.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ColumnTagScenarios.scala => ScenarioColumnTag.scala} (96%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{CompactionPlanningScenarios.scala => ScenarioCompactionPlanning.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ConcurrencyScenarios.scala => ScenarioConcurrency.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{DataTypeScenarios.scala => ScenarioDataType.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{DmlScenarios.scala => ScenarioDml.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{DmlValidationScenarios.scala => ScenarioDmlValidation.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{EncryptionScenarios.scala => ScenarioEncryption.scala} (97%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{FileFormatScenarios.scala => ScenarioFileFormat.scala} (98%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{FileReplicationScenarios.scala => ScenarioFileReplication.scala} (98%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{IncrementalReadScenarios.scala => ScenarioIncrementalRead.scala} (98%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{LockingScenarios.scala => ScenarioLocking.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{MaintenanceScenarios.scala => ScenarioMaintenance.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{MetadataTableScenarios.scala => ScenarioMetadataTable.scala} (98%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{NamespaceScenarios.scala => ScenarioNamespace.scala} (97%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{NestedTypeScenarios.scala => ScenarioNestedType.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{PartitionEvolutionScenarios.scala => ScenarioPartitionEvolution.scala} (97%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{PartitionTransformScenarios.scala => ScenarioPartitionTransform.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ProcedureScenarios.scala => ScenarioProcedure.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{RenameScenarios.scala => ScenarioRename.scala} (98%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ScanPlanningScenarios.scala => ScenarioScanPlanning.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{SchemaEvolutionScenarios.scala => ScenarioSchemaEvolution.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{SnapshotRestoreScenarios.scala => ScenarioSnapshotRestore.scala} (98%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{SortOrderScenarios.scala => ScenarioSortOrder.scala} (98%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{StreamingScenarios.scala => ScenarioStreaming.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{TableEvolutionCompatibilityScenarios.scala => ScenarioTableEvolutionCompatibility.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{TablePropertyScenarios.scala => ScenarioTableProperty.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{TimeTravelScenarios.scala => ScenarioTimeTravel.scala} (98%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{WriteDistributionScenarios.scala => ScenarioWriteDistribution.scala} (99%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{WriterCompatibilityScenarios.scala => ScenarioWriterCompatibility.scala} (97%) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala index 5fdd0c0b3..68e26be37 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala @@ -2,34 +2,34 @@ package harness /** Mixes every standard capability trait into one catalog source. */ object Scenarios - extends AccessControlScenarios - with ChangelogScenarios - with ColumnTagScenarios - with CompactionPlanningScenarios - with ConcurrencyScenarios - with DataTypeScenarios - with DmlScenarios - with DmlValidationScenarios - with EncryptionScenarios - with FileFormatScenarios - with FileReplicationScenarios - with IncrementalReadScenarios - with LockingScenarios - with MaintenanceScenarios - with MetadataTableScenarios - with NamespaceScenarios - with NestedTypeScenarios - with PartitionEvolutionScenarios - with PartitionTransformScenarios - with ProcedureScenarios - with RenameScenarios - with ScanPlanningScenarios - with SchemaEvolutionScenarios - with SnapshotRestoreScenarios - with SortOrderScenarios - with StreamingScenarios - with TableEvolutionCompatibilityScenarios - with TablePropertyScenarios - with TimeTravelScenarios - with WriteDistributionScenarios - with WriterCompatibilityScenarios + extends ScenarioAccessControl + with ScenarioChangelog + with ScenarioColumnTag + with ScenarioCompactionPlanning + with ScenarioConcurrency + with ScenarioDataType + with ScenarioDml + with ScenarioDmlValidation + with ScenarioEncryption + with ScenarioFileFormat + with ScenarioFileReplication + with ScenarioIncrementalRead + with ScenarioLocking + with ScenarioMaintenance + with ScenarioMetadataTable + with ScenarioNamespace + with ScenarioNestedType + with ScenarioPartitionEvolution + with ScenarioPartitionTransform + with ScenarioProcedure + with ScenarioRename + with ScenarioScanPlanning + with ScenarioSchemaEvolution + with ScenarioSnapshotRestore + with ScenarioSortOrder + with ScenarioStreaming + with ScenarioTableEvolutionCompatibility + with ScenarioTableProperty + with ScenarioTimeTravel + with ScenarioWriteDistribution + with ScenarioWriterCompatibility diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/AccessControlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioAccessControl.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/AccessControlScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioAccessControl.scala index 12ab6965d..c4e0ff084 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/AccessControlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioAccessControl.scala @@ -17,7 +17,7 @@ import org.apache.iceberg.exceptions.BadRequestException * * Case families: eight families contributing 16 cases. */ -trait AccessControlScenarios extends ScenarioKit { +trait ScenarioAccessControl extends ScenarioKit { /** Every access-control case, one file format at a time. */ lazy val accessControlCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioChangelog.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioChangelog.scala index 4a445ed38..33038ce84 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioChangelog.scala @@ -26,7 +26,7 @@ final case class ChangelogOperation( * Case families: three families contributing 14 cases, 10 operation cases, 2 append-only history cases and 2 * expired-range cases. */ -trait ChangelogScenarios extends ScenarioKit { +trait ScenarioChangelog extends ScenarioKit { /** Every changelog case, one file format at a time. */ lazy val changelogCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ColumnTagScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioColumnTag.scala similarity index 96% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ColumnTagScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioColumnTag.scala index ed72d1288..f0f278152 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ColumnTagScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioColumnTag.scala @@ -10,7 +10,7 @@ package harness * * Case families: one family contributing 2 cases. */ -trait ColumnTagScenarios extends ScenarioKit { +trait ScenarioColumnTag extends ScenarioKit { /** The column-tag case, one file format at a time. */ lazy val columnTagCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/CompactionPlanningScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCompactionPlanning.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/CompactionPlanningScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCompactionPlanning.scala index bf4001709..032cc6a5d 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/CompactionPlanningScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCompactionPlanning.scala @@ -16,7 +16,7 @@ import org.apache.spark.sql.SparkSession * * Case families: two families contributing 3 cases. */ -trait CompactionPlanningScenarios extends ScenarioKit { +trait ScenarioCompactionPlanning extends ScenarioKit { /** The bin-packing case in each columnar format, then the file-sequence ordering case on Parquet. */ lazy val compactionPlanningCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencyScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioConcurrency.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencyScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioConcurrency.scala index 71fad133a..bb7a33322 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencyScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioConcurrency.scala @@ -15,7 +15,7 @@ import java.util.concurrent.atomic.AtomicInteger * * Case families: two families contributing 4 cases. */ -trait ConcurrencyScenarios extends ScenarioKit { +trait ScenarioConcurrency extends ScenarioKit { /** Every concurrency case, one file format at a time. */ lazy val concurrencyCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DataTypeScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDataType.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/DataTypeScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDataType.scala index 15d281710..fa432d018 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DataTypeScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDataType.scala @@ -15,7 +15,7 @@ import java.math.BigDecimal * * Case families: five families over three layouts, contributing 15 cases. */ -trait DataTypeScenarios extends ScenarioKit { +trait ScenarioDataType extends ScenarioKit { /** Every scalar-type case, one layout at a time. */ lazy val dataTypeCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDml.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDml.scala index 08ad5884a..695b7f3ba 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDml.scala @@ -21,7 +21,7 @@ import org.apache.spark.sql.functions.lit * Case families: 804 cases in four families, `coreDmlCases` (312), `partitionedDmlCases` (6), `orderedDmlCases` (312) * and `evolvedDmlCases` (174). */ -trait DmlScenarios extends ScenarioKit { +trait ScenarioDml extends ScenarioKit { import Rows._ /** Every DML case, in preparation order: core, partition-scoped, write-ordered, then evolved. */ diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlValidationScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDmlValidation.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlValidationScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDmlValidation.scala index fd9b3001b..56b94ba7d 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlValidationScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDmlValidation.scala @@ -14,7 +14,7 @@ import org.apache.spark.sql.AnalysisException * * Case families: six families contributing 12 cases. */ -trait DmlValidationScenarios extends ScenarioKit { +trait ScenarioDmlValidation extends ScenarioKit { /** Every DML-validation case, one file format at a time. */ lazy val dmlValidationCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/EncryptionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioEncryption.scala similarity index 97% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/EncryptionScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioEncryption.scala index 6bdb56d8b..c8da2af6c 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/EncryptionScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioEncryption.scala @@ -14,7 +14,7 @@ import java.nio.file.{Files, Paths} * * Case families: one family contributing 1 case. */ -trait EncryptionScenarios extends ScenarioKit { +trait ScenarioEncryption extends ScenarioKit { /** The plaintext data-file case, on the standard seeded Parquet table. */ lazy val encryptionCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileFormatScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileFormat.scala similarity index 98% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileFormatScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileFormat.scala index 20ff523e7..7d0e7f499 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileFormatScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileFormat.scala @@ -13,7 +13,7 @@ package harness * * Case families: one family, `format.materialization`, contributing 12 cases. */ -trait FileFormatScenarios extends ScenarioKit { +trait ScenarioFileFormat extends ScenarioKit { /** The format-materialization case on every standard preparation that writes data files. */ lazy val fileFormatCases: List[Plan.Case] = layoutFormatCasesFor(layoutFormatPreparations) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileReplicationScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileReplication.scala similarity index 98% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileReplicationScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileReplication.scala index cc6e2a711..7005e4ffe 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/FileReplicationScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileReplication.scala @@ -17,7 +17,7 @@ import scala.util.Try * * Case families: one family contributing 1 case. */ -trait FileReplicationScenarios extends ScenarioKit { +trait ScenarioFileReplication extends ScenarioKit { /** The output-file replication property case. */ lazy val fileReplicationCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/IncrementalReadScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioIncrementalRead.scala similarity index 98% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/IncrementalReadScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioIncrementalRead.scala index baa8bcc4c..9af630008 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/IncrementalReadScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioIncrementalRead.scala @@ -12,7 +12,7 @@ package harness * * Case families: five families contributing 10 cases. */ -trait IncrementalReadScenarios extends ScenarioKit { +trait ScenarioIncrementalRead extends ScenarioKit { /** Every incremental-read case, one file format at a time. */ lazy val incrementalReadCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LockingScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioLocking.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/LockingScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioLocking.scala index d8f1ec046..36c92f264 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LockingScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioLocking.scala @@ -15,7 +15,7 @@ package harness * * Case families: two families contributing 2 cases. */ -trait LockingScenarios extends ScenarioKit { +trait ScenarioLocking extends ScenarioKit { /** The lock cases, each driven over HTTP against the embedded server. */ lazy val lockingCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintenanceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMaintenance.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintenanceScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMaintenance.scala index 5ce5ae921..e024268a4 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MaintenanceScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMaintenance.scala @@ -16,7 +16,7 @@ import java.nio.file.attribute.FileTime * * Case families: six families contributing 12 cases. */ -trait MaintenanceScenarios extends ScenarioKit { +trait ScenarioMaintenance extends ScenarioKit { /** Every maintenance case, one file format at a time. */ lazy val maintenanceCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MetadataTableScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMetadataTable.scala similarity index 98% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/MetadataTableScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMetadataTable.scala index 4684b9943..b091881d5 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/MetadataTableScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMetadataTable.scala @@ -13,7 +13,7 @@ package harness * * Case families: three families contributing 6 cases. */ -trait MetadataTableScenarios extends ScenarioKit { +trait ScenarioMetadataTable extends ScenarioKit { /** Every metadata-table case, one file format at a time. */ lazy val metadataTableCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NamespaceScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNamespace.scala similarity index 97% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/NamespaceScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNamespace.scala index 11dd401b6..044488275 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NamespaceScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNamespace.scala @@ -11,7 +11,7 @@ package harness * * Case families: two families contributing 4 cases. */ -trait NamespaceScenarios extends ScenarioKit { +trait ScenarioNamespace extends ScenarioKit { /** Every namespace case, one file format at a time. */ lazy val namespaceCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypeScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNestedType.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypeScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNestedType.scala index 9d8a46711..2c525444b 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/NestedTypeScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNestedType.scala @@ -15,7 +15,7 @@ package harness * * Case families: nine families contributing 25 cases, 21 on the nested layouts and 4 on the standard formats. */ -trait NestedTypeScenarios extends ScenarioKit { +trait ScenarioNestedType extends ScenarioKit { /** Every nested-type case: the reads and writes on the nested layouts, then the struct-evolution cases. */ lazy val nestedTypeCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionEvolutionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionEvolution.scala similarity index 97% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionEvolutionScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionEvolution.scala index 5bd6b09a6..826de7d3f 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionEvolutionScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionEvolution.scala @@ -11,7 +11,7 @@ package harness * * Case families: two families contributing 4 cases. */ -trait PartitionEvolutionScenarios extends ScenarioKit { +trait ScenarioPartitionEvolution extends ScenarioKit { /** The rejected partition-evolution statements, one file format at a time. */ lazy val partitionEvolutionCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionTransformScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionTransform.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionTransformScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionTransform.scala index 88ef88e86..3c925c944 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/PartitionTransformScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionTransform.scala @@ -17,7 +17,7 @@ import org.apache.spark.sql.types.StructType * * Case families: ten families contributing 20 cases, 14 accepted transforms and 6 rejections. */ -trait PartitionTransformScenarios extends ScenarioKit { +trait ScenarioPartitionTransform extends ScenarioKit { /** Every partition-transform case, one file format at a time. */ lazy val partitionTransformCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ProcedureScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioProcedure.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ProcedureScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioProcedure.scala index 0ccff3c48..e81d52eac 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ProcedureScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioProcedure.scala @@ -12,7 +12,7 @@ package harness * * Case families: three families contributing 6 cases. */ -trait ProcedureScenarios extends ScenarioKit { +trait ScenarioProcedure extends ScenarioKit { /** Every catalog-procedure case, one file format at a time. */ lazy val procedureCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RenameScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRename.scala similarity index 98% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/RenameScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRename.scala index d840c60e3..87158fa86 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/RenameScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRename.scala @@ -14,7 +14,7 @@ import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageE * * Case families: two families contributing 4 cases. */ -trait RenameScenarios extends ScenarioKit { +trait ScenarioRename extends ScenarioKit { /** Every rename case, one file format at a time. */ lazy val renameCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScanPlanningScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioScanPlanning.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScanPlanningScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioScanPlanning.scala index e1e97fb5d..582bd8659 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScanPlanningScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioScanPlanning.scala @@ -17,7 +17,7 @@ import scala.collection.JavaConverters._ * * Case families: one family contributing 2 cases. */ -trait ScanPlanningScenarios extends ScenarioKit { +trait ScenarioScanPlanning extends ScenarioKit { /** The split-size case, one file format at a time. */ lazy val scanPlanningCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SchemaEvolutionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSchemaEvolution.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/SchemaEvolutionScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSchemaEvolution.scala index 4cc596273..8165eb997 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SchemaEvolutionScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSchemaEvolution.scala @@ -19,7 +19,7 @@ import org.apache.iceberg.exceptions.BadRequestException * Case families: 14 families contributing 56 cases, 6 created-schema, 36 evolution, and 14 rejection or side-table * cases. */ -trait SchemaEvolutionScenarios extends ScenarioKit { +trait ScenarioSchemaEvolution extends ScenarioKit { /** Every schema-evolution case: the created schema, then the accepted changes, then the boundaries. */ lazy val schemaEvolutionCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SnapshotRestoreScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSnapshotRestore.scala similarity index 98% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/SnapshotRestoreScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSnapshotRestore.scala index 1b6c01550..0cf04f6e0 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SnapshotRestoreScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSnapshotRestore.scala @@ -11,7 +11,7 @@ package harness * * Case families: three families contributing 6 cases. */ -trait SnapshotRestoreScenarios extends ScenarioKit { +trait ScenarioSnapshotRestore extends ScenarioKit { /** Every snapshot-restore case, one file format at a time. */ lazy val snapshotRestoreCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SortOrderScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSortOrder.scala similarity index 98% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/SortOrderScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSortOrder.scala index 88a25e05c..a9ddc5c7e 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/SortOrderScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSortOrder.scala @@ -11,7 +11,7 @@ package harness * * Case families: two families contributing 4 cases. */ -trait SortOrderScenarios extends ScenarioKit { +trait ScenarioSortOrder extends ScenarioKit { /** Every sort-order case, one file format at a time. */ lazy val sortOrderCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/StreamingScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioStreaming.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/StreamingScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioStreaming.scala index e6b3d55bf..6c89f3f0a 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/StreamingScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioStreaming.scala @@ -18,7 +18,7 @@ import org.apache.spark.sql.streaming.Trigger * * Case families: five families contributing 10 cases. */ -trait StreamingScenarios extends ScenarioKit { +trait ScenarioStreaming extends ScenarioKit { /** Every streaming case, one file format at a time. */ lazy val streamingCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TableEvolutionCompatibilityScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableEvolutionCompatibility.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/TableEvolutionCompatibilityScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableEvolutionCompatibility.scala index d4252b1be..fa750ad55 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TableEvolutionCompatibilityScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableEvolutionCompatibility.scala @@ -25,7 +25,7 @@ private[harness] final case class TableAlteration( * * Case families: six families over 16 preparations, contributing 96 cases. */ -trait TableEvolutionCompatibilityScenarios extends ScenarioKit { +trait ScenarioTableEvolutionCompatibility extends ScenarioKit { /** Every follow-up operation on every altered preparation, one preparation at a time. */ lazy val tableEvolutionCompatibilityCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TablePropertyScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableProperty.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/TablePropertyScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableProperty.scala index 4e024d7a4..8943e6965 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TablePropertyScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableProperty.scala @@ -17,7 +17,7 @@ import org.apache.iceberg.exceptions.BadRequestException * * Case families: six families contributing 12 cases. */ -trait TablePropertyScenarios extends ScenarioKit { +trait ScenarioTableProperty extends ScenarioKit { /** Every table-property case, one file format at a time. */ lazy val tablePropertyCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TimeTravelScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTimeTravel.scala similarity index 98% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/TimeTravelScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTimeTravel.scala index 0dbdd40ed..03b880d18 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/TimeTravelScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTimeTravel.scala @@ -12,7 +12,7 @@ package harness * * Case families: three families contributing 6 cases. */ -trait TimeTravelScenarios extends ScenarioKit { +trait ScenarioTimeTravel extends ScenarioKit { /** Every time-travel case, one file format at a time. */ lazy val timeTravelCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriteDistributionScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriteDistribution.scala similarity index 99% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriteDistributionScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriteDistribution.scala index 06c88dc44..ed3a9a85c 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriteDistributionScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriteDistribution.scala @@ -14,7 +14,7 @@ package harness * * Case families: three families contributing 6 cases. */ -trait WriteDistributionScenarios extends ScenarioKit { +trait ScenarioWriteDistribution extends ScenarioKit { /** Every write-distribution case, one file format at a time. */ lazy val writeDistributionCases: List[Plan.Case] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriterCompatibilityScenarios.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriterCompatibility.scala similarity index 97% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriterCompatibilityScenarios.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriterCompatibility.scala index a895210b8..a2d13ade0 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/WriterCompatibilityScenarios.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriterCompatibility.scala @@ -13,7 +13,7 @@ import org.apache.spark.sql.AnalysisException * * Case families: one family contributing 2 cases. */ -trait WriterCompatibilityScenarios extends ScenarioKit { +trait ScenarioWriterCompatibility extends ScenarioKit { /** The explicit-column writer case, one file format at a time. */ lazy val writerCompatibilityCases: List[Plan.Case] = From 3353c58633e446a0aadfc2ffbef63473995d1dde Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Tue, 1 Sep 2026 19:40:33 -0700 Subject: [PATCH 14/24] refactor(delta-harness): narrow foundation Keep PR 682 focused on reusable DDL and DML coverage while moving orthogonal capabilities to extension branches. - retain 642 Parquet and ORC foundation cases - extract reusable changelog and concurrency support - preserve Plan and Scenarios consumer compatibility - add extension-stable catalog and support contract tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/openhouse/ChangelogSupport.scala | 114 +++++++++ .../openhouse/ConcurrencySupport.scala | 75 ++++++ .../scala/harness/openhouse/Framework.scala | 23 +- .../scala/harness/openhouse/LocalRunner.scala | 12 +- .../harness/openhouse/OpenHouseMatrix.scala | 35 --- .../main/scala/harness/openhouse/Plan.scala | 60 ----- .../openhouse/ScenarioAccessControl.scala | 196 ---------------- .../harness/openhouse/ScenarioCatalog.scala | 106 +++++++++ .../harness/openhouse/ScenarioChangelog.scala | 219 ------------------ .../harness/openhouse/ScenarioColumnTag.scala | 43 ---- .../ScenarioCompactionPlanning.scala | 137 ----------- .../openhouse/ScenarioConcurrency.scala | 155 ------------- .../harness/openhouse/ScenarioDataType.scala | 14 +- .../scala/harness/openhouse/ScenarioDml.scala | 20 +- .../openhouse/ScenarioDmlValidation.scala | 16 +- .../openhouse/ScenarioEncryption.scala | 44 ---- .../openhouse/ScenarioFileFormat.scala | 12 +- .../openhouse/ScenarioFileReplication.scala | 82 ------- .../openhouse/ScenarioIncrementalRead.scala | 93 -------- .../scala/harness/openhouse/ScenarioKit.scala | 33 ++- .../harness/openhouse/ScenarioLocking.scala | 114 --------- .../openhouse/ScenarioMaintenance.scala | 169 -------------- .../openhouse/ScenarioMetadataTable.scala | 96 -------- .../harness/openhouse/ScenarioNamespace.scala | 48 ---- .../openhouse/ScenarioNestedType.scala | 22 +- .../ScenarioPartitionEvolution.scala | 10 +- .../ScenarioPartitionTransform.scala | 155 ------------- .../harness/openhouse/ScenarioProcedure.scala | 113 --------- .../harness/openhouse/ScenarioRename.scala | 68 ------ .../openhouse/ScenarioScanPlanning.scala | 115 --------- .../openhouse/ScenarioSchemaEvolution.scala | 46 ++-- .../openhouse/ScenarioSnapshotRestore.scala | 89 ------- .../harness/openhouse/ScenarioSortOrder.scala | 61 ----- .../harness/openhouse/ScenarioStreaming.scala | 213 ----------------- .../ScenarioTableEvolutionCompatibility.scala | 170 -------------- .../openhouse/ScenarioTableProperty.scala | 18 +- .../openhouse/ScenarioTimeTravel.scala | 98 -------- .../openhouse/ScenarioWriteDistribution.scala | 161 ------------- .../ScenarioWriterCompatibility.scala | 50 ---- .../test/scala/harness/CaseCatalogTest.scala | 143 +++++------- .../scala/harness/DmlCaseCatalogTest.scala | 29 +-- .../scala/harness/FoundationCatalogTest.scala | 122 ++++++++++ .../PublicSurfaceCompatibilityTest.scala | 103 ++++++++ .../scala/harness/SupportContractTest.scala | 130 +++++++++++ .../scala/harness/TablePreparationTest.scala | 2 +- 45 files changed, 833 insertions(+), 3001 deletions(-) create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogSupport.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencySupport.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioAccessControl.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioChangelog.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioColumnTag.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCompactionPlanning.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioConcurrency.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioEncryption.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileReplication.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioIncrementalRead.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioLocking.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMaintenance.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMetadataTable.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNamespace.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionTransform.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioProcedure.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRename.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioScanPlanning.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSnapshotRestore.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSortOrder.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioStreaming.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableEvolutionCompatibility.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTimeTravel.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriteDistribution.scala delete mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriterCompatibility.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/FoundationCatalogTest.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/PublicSurfaceCompatibilityTest.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/SupportContractTest.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogSupport.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogSupport.scala new file mode 100644 index 000000000..eb26ec953 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogSupport.scala @@ -0,0 +1,114 @@ +package harness + +/** + * One changelog operation: the name its case carries, the statement it runs against the prepared table, and the + * change-type histogram the changelog view reports for the snapshot range that statement opened. + */ +final case class ChangelogOperation( + name: String, + statement: String => String, + expectedChangeCounts: Map[String, Long] +) + +/** + * Reusable changelog support for capability layers. It contributes zero catalog cases while holding the row-level + * operations whose change feed `create_changelog_view` reports and the factory that turns those operations into cases + * on preparations a caller supplies. + * + * A feature layer that needs changelog signal mixes this trait in and crosses `changelogOperations` with its own + * preparations. The replace-table layer uses it to require rejection when a changelog range crosses a table + * replacement. The follow-up standard changelog scenario builds on the same operation definitions. + */ +trait ChangelogSupport extends ScenarioKit { + + /** + * The five row-level operations whose change feed the catalog reports: an append, an INSERT OVERWRITE that drops one + * row, a row-level DELETE, an UPDATE, and a MERGE that updates one row and inserts another. Every one starts from + * the standard three-row seed, so its expected histogram holds on any preparation that seeds those rows. + */ + lazy val changelogOperations: List[ChangelogOperation] = + List( + ChangelogOperation( + "changelog.append", + table => + s"INSERT INTO $table VALUES " + + "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')", + Map("INSERT" -> 1L)), + ChangelogOperation( + "changelog.overwrite", + table => + s"INSERT OVERWRITE $table SELECT * FROM $table " + + s"WHERE ${Core.long0.columnName} <= 2", + Map("DELETE" -> 1L)), + ChangelogOperation( + "changelog.delete", + table => s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1", + Map("DELETE" -> 1L)), + ChangelogOperation( + "changelog.update", + table => + s"UPDATE $table SET ${Core.string0.columnName} = 'upd' " + + s"WHERE ${Core.long0.columnName} = 2", + Map("DELETE" -> 1L, "INSERT" -> 1L)), + ChangelogOperation( + "changelog.merge", + table => + s"MERGE INTO $table target " + + "USING (SELECT CAST(2 AS BIGINT) key " + + "UNION ALL SELECT CAST(9 AS BIGINT)) source " + + s"ON target.${Core.long0.columnName} = source.key " + + s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + + "WHEN NOT MATCHED THEN INSERT " + + s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + + s"${Core.string0.columnName}, ${Core.double0.columnName}, " + + s"${Core.boolean0.columnName}, ${Core.date0.columnName}) " + + "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')", + Map("DELETE" -> 1L, "INSERT" -> 2L))) + + /** The changelog cases for every operation on every preparation given, one preparation at a time. */ + def changelogOperationCasesFor( + preparations: List[TablePreparation[CoreTable.type]] + ): List[TestCase] = + preparations.flatMap(preparation => + changelogOperations.map(operation => changelogOperationCase(preparation, operation))) + + /** The change-type histogram the named changelog view reports. */ + def changeCounts(table: PreparedTable[CoreTable.type], view: String): Map[String, Long] = + table.spark + .sql(s"SELECT _change_type, count(*) FROM $view GROUP BY _change_type") + .collect() + .map(row => row.getString(0) -> row.getLong(1)) + .toMap + + /** The name of a changelog view over `table`, opened at `startSnapshotId`. */ + def changelogViewFrom( + table: PreparedTable[CoreTable.type], + startSnapshotId: Long): String = + table.spark + .sql( + "CALL openhouse.system.create_changelog_view(" + + s"table => '${catalogRelative(table.name)}', " + + s"options => map('start-snapshot-id', '$startSnapshotId'))") + .collect()(0) + .getString(0) + + // --- the case body the surface above composes --- + + /** + * Running the operation against a seeded table and opening a changelog view at the seed snapshot reports exactly the + * change types and counts that operation is defined to produce. + */ + private def changelogOperationCase( + preparation: TablePreparation[CoreTable.type], + operation: ChangelogOperation): TestCase = + preparation.test(operation.name) { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql(operation.statement(table.name)) + val actualChangeCounts = changeCounts(table, changelogViewFrom(table, seedSnapshotId)) + + assert( + actualChangeCounts == operation.expectedChangeCounts, + s"${operation.name} reported $actualChangeCounts, expected ${operation.expectedChangeCounts}") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencySupport.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencySupport.scala new file mode 100644 index 000000000..4c975e36f --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencySupport.scala @@ -0,0 +1,75 @@ +package harness + +import java.util.concurrent.{ConcurrentLinkedQueue, CountDownLatch, TimeUnit} + +/** + * Reusable concurrency support for racing-writer cases. It contributes zero catalog cases while exposing two + * primitives: concurrent function execution and explicit classification of typed commit conflicts. + * + * Both primitives are feature neutral and free of table state, so a feature layer reuses them for its own table mode. + * The replace-table layer uses them to prove that a replacement racing an append either commits or fails with a typed + * commit conflict. The general standard concurrency cases live in a follow-up scenario. + */ +object ConcurrencySupport { + + /** How long `runConcurrently` waits for every thread before it reports the stragglers as failures. */ + val completionTimeoutMinutes: Long = 3 + + /** + * Runs every function on its own daemon thread, releases them together, and waits up to + * `completionTimeoutMinutes` for all of them. Returns the throwables the threads raised, plus one for each thread + * still running at the deadline. A caller that expects conflicts catches them inside its own function, so a + * non-empty result always means a thread failed outside the operation under test. + */ + def runConcurrently(functions: Seq[() => Unit]): Seq[Throwable] = { + val errors = new ConcurrentLinkedQueue[Throwable]() + val start = new CountDownLatch(1) + val threads = functions.zipWithIndex.map { case (function, index) => + val thread = new Thread( + () => + try { + start.await() + function() + } catch { + case interrupted: InterruptedException => + Thread.currentThread().interrupt() + errors.add(interrupted) + case throwable: Throwable => + errors.add(throwable) + }, + s"delta-harness-concurrent-$index") + thread.setDaemon(true) + thread + } + threads.foreach(_.start()) + start.countDown() + + val deadline = System.nanoTime() + TimeUnit.MINUTES.toNanos(completionTimeoutMinutes) + threads.foreach { thread => + val remainingNanos = deadline - System.nanoTime() + if (remainingNanos > 0) { + TimeUnit.NANOSECONDS.timedJoin(thread, remainingNanos) + } + } + + threads.filter(_.isAlive).foreach { thread => + errors.add( + new AssertionError( + s"${thread.getName} did not complete within $completionTimeoutMinutes minutes")) + thread.interrupt() + } + errors.toArray(Array.empty[Throwable]).toSeq + } + + /** A commit conflict the catalog reports through one of its typed commit, validation or transport exceptions. */ + def isTypedCommitConflict(throwable: Throwable): Boolean = + Exceptions.causeChain(throwable).exists { cause => + val className = cause.getClass.getName + className.contains("CommitFailed") || + className.contains("CommitStateUnknown") || + className.contains("Validation") || + className.contains("BadRequest") || + className.contains("WebClientResponse") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala index 3b46ad169..3d4022e1a 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala @@ -13,9 +13,24 @@ import scala.annotation.tailrec import scala.reflect.{ClassTag, classTag} import scala.util.control.NonFatal -// The harness defines typed, reusable table preparations and localized Plan.Case bodies. Each case gets a fresh table, +// The harness defines typed, reusable table preparations and localized TestCase bodies. Each case gets a fresh table, // executes its preparation, runs its action and assertions, and drops the table during teardown. +/** + * One catalog case: the ID that names it, the body that runs it, and its two skip policies. + * `knownBugReason` marks a case the catalog under test is known to fail; `embeddedSkipReason` marks a case the + * embedded local catalog cannot run at all. A runner reports either policy as a skip. + */ +final case class TestCase( + id: String, + run: Ctx => Unit, + knownBugReason: Option[String] = None, + embeddedSkipReason: Option[String] = None +) { + /** The skip reason a known bug produces, phrased so a run log explains the skip. */ + def bugReason: Option[String] = knownBugReason.map(reason => s"bug: $reason") +} + final case class Ctx(spark: SparkSession, namespace: String, restUri: String = "", restToken: String = "") // Minimal REST client to the embedded OpenHouse server (control-plane ops with no SQL surface: lock/unlock). Uses JDK @@ -406,8 +421,8 @@ final case class TablePreparation[S <: Schema]( * Build the case that runs `body` against one freshly prepared table. The case ID combines the preparation's prefix * and label with `caseName`, so one test body yields a separate case on every preparation it runs on. */ - def test(caseName: String)(body: PreparedTable[S] => Unit): Plan.Case = - Plan.Case( + def test(caseName: String)(body: PreparedTable[S] => Unit): TestCase = + TestCase( s"$casePrefix$caseName @ $label", context => preparation.prepare(context) { table => var testFailure: Option[Throwable] = None @@ -435,7 +450,7 @@ final case class DmlTestCase[S <: Schema]( knownBugReason: Option[String] = None ) { /** Build the case that runs this operation against a table `preparation` produces. */ - def runOn(preparation: TablePreparation[S]): Plan.Case = + def runOn(preparation: TablePreparation[S]): TestCase = preparation .test(id)(run) .copy(knownBugReason = knownBugReason) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LocalRunner.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LocalRunner.scala index e039784ec..c5c5c7db1 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LocalRunner.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/LocalRunner.scala @@ -13,7 +13,7 @@ object Runner { val MaxAttempts = 3 /** Runs a case, retrying only a transient-infrastructure failure. */ - def execute(testCase: Plan.Case, context: Ctx): (Outcome, Int) = { + def execute(testCase: TestCase, context: Ctx): (Outcome, Int) = { @tailrec def attempt(attemptIndex: Int): (Outcome, Int) = { val outcome = try { @@ -44,7 +44,7 @@ object Main { // Each command-line argument is an include substring. A case runs when its ID contains every provided substring. // An empty argument list selects the full catalog. val filters = args.toList - val cases = Plan.cases.filter(testCase => + val cases = ScenarioCatalog.cases.filter(testCase => filters.forall(testCase.id.contains)) val header = @@ -61,10 +61,10 @@ object Main { .getOrElse(math.max(1, Runtime.getRuntime.availableProcessors())) println(s"parallelism: $parallelism worker sessions\n") - def runOne(testCase: Plan.Case): (Plan.Case, (Outcome, Int)) = + def runOne(testCase: TestCase): (TestCase, (Outcome, Int)) = testCase.embeddedSkipReason .map(reason => s"embedded limitation: $reason") - .orElse(Plan.bugReason(testCase)) match { + .orElse(testCase.bugReason) match { case Some(reason) => (testCase, (Outcome.Skipped(reason): Outcome, 0)) case None => @@ -79,8 +79,8 @@ object Main { try { val futures = cases.map(testCase => pool.submit( - new Callable[(Plan.Case, (Outcome, Int))] { - def call(): (Plan.Case, (Outcome, Int)) = runOne(testCase) + new Callable[(TestCase, (Outcome, Int))] { + def call(): (TestCase, (Outcome, Int)) = runOne(testCase) })) futures.map(_.get(60, TimeUnit.MINUTES)) } finally { diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala deleted file mode 100644 index 68e26be37..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/OpenHouseMatrix.scala +++ /dev/null @@ -1,35 +0,0 @@ -package harness - -/** Mixes every standard capability trait into one catalog source. */ -object Scenarios - extends ScenarioAccessControl - with ScenarioChangelog - with ScenarioColumnTag - with ScenarioCompactionPlanning - with ScenarioConcurrency - with ScenarioDataType - with ScenarioDml - with ScenarioDmlValidation - with ScenarioEncryption - with ScenarioFileFormat - with ScenarioFileReplication - with ScenarioIncrementalRead - with ScenarioLocking - with ScenarioMaintenance - with ScenarioMetadataTable - with ScenarioNamespace - with ScenarioNestedType - with ScenarioPartitionEvolution - with ScenarioPartitionTransform - with ScenarioProcedure - with ScenarioRename - with ScenarioScanPlanning - with ScenarioSchemaEvolution - with ScenarioSnapshotRestore - with ScenarioSortOrder - with ScenarioStreaming - with ScenarioTableEvolutionCompatibility - with ScenarioTableProperty - with ScenarioTimeTravel - with ScenarioWriteDistribution - with ScenarioWriterCompatibility diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala deleted file mode 100644 index 3d8096f9b..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/Plan.scala +++ /dev/null @@ -1,60 +0,0 @@ -package harness - -/** - * The ordered catalog of scenario-owned test cases. - * - * Every capability trait contributes exactly one case list. Plan names each contribution once, in alphabetical order - * by contribution name, and concatenates them. Composition is all Plan does: a scenario body, a preparation and a case - * ID all belong to the capability that owns them. - */ -object Plan { - final case class Case( - id: String, - run: Ctx => Unit, - knownBugReason: Option[String] = None, - embeddedSkipReason: Option[String] = None - ) - - /** The deterministic ordered case catalog. Reading it does not execute a case or start Spark. */ - def caseIds: List[String] = cases.map(_.id) - - def bugReason(testCase: Case): Option[String] = - testCase.knownBugReason.map(reason => s"bug: $reason") - - /** Every capability contribution, named once, in the order Plan integrates them. */ - def contributions: List[(String, List[Case])] = - List( - "accessControlCases" -> Scenarios.accessControlCases, - "changelogCases" -> Scenarios.changelogCases, - "columnTagCases" -> Scenarios.columnTagCases, - "compactionPlanningCases" -> Scenarios.compactionPlanningCases, - "concurrencyCases" -> Scenarios.concurrencyCases, - "dataTypeCases" -> Scenarios.dataTypeCases, - "dmlCases" -> Scenarios.dmlCases, - "dmlValidationCases" -> Scenarios.dmlValidationCases, - "encryptionCases" -> Scenarios.encryptionCases, - "fileFormatCases" -> Scenarios.fileFormatCases, - "fileReplicationCases" -> Scenarios.fileReplicationCases, - "incrementalReadCases" -> Scenarios.incrementalReadCases, - "lockingCases" -> Scenarios.lockingCases, - "maintenanceCases" -> Scenarios.maintenanceCases, - "metadataTableCases" -> Scenarios.metadataTableCases, - "namespaceCases" -> Scenarios.namespaceCases, - "nestedTypeCases" -> Scenarios.nestedTypeCases, - "partitionEvolutionCases" -> Scenarios.partitionEvolutionCases, - "partitionTransformCases" -> Scenarios.partitionTransformCases, - "procedureCases" -> Scenarios.procedureCases, - "renameCases" -> Scenarios.renameCases, - "scanPlanningCases" -> Scenarios.scanPlanningCases, - "schemaEvolutionCases" -> Scenarios.schemaEvolutionCases, - "snapshotRestoreCases" -> Scenarios.snapshotRestoreCases, - "sortOrderCases" -> Scenarios.sortOrderCases, - "streamingCases" -> Scenarios.streamingCases, - "tableEvolutionCompatibilityCases" -> Scenarios.tableEvolutionCompatibilityCases, - "tablePropertyCases" -> Scenarios.tablePropertyCases, - "timeTravelCases" -> Scenarios.timeTravelCases, - "writeDistributionCases" -> Scenarios.writeDistributionCases, - "writerCompatibilityCases" -> Scenarios.writerCompatibilityCases) - - def cases: List[Case] = contributions.flatMap { case (_, contribution) => contribution } -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioAccessControl.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioAccessControl.scala deleted file mode 100644 index c4e0ff084..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioAccessControl.scala +++ /dev/null @@ -1,196 +0,0 @@ -package harness - -import org.apache.iceberg.exceptions.BadRequestException - -/** - * Access control: the SET POLICY statements that govern how a table may be shared, retained and replicated, and the - * GRANT and REVOKE statements that decide who may read it. - * - * Operations: SET POLICY (SHARING), SET POLICY (HISTORY), SET POLICY (REPLICATION) followed by UNSET POLICY - * (REPLICATION), SET POLICY (RETENTION) on the date column, the out-of-range SET POLICY (HISTORY MAX_AGE) and SET - * POLICY (HISTORY VERSIONS) forms, GRANT SELECT on an unshared table, and GRANT then REVOKE SELECT on a shared table - * with SHOW GRANTS in between. - * - * Preparation axes: the standard seeded core table in each of the two columnar formats, except the retention family, - * which starts from a date-partitioned core table seeded with the standard rows because RETENTION names a partition - * column. - * - * Case families: eight families contributing 16 cases. - */ -trait ScenarioAccessControl extends ScenarioKit { - - /** Every access-control case, one file format at a time. */ - lazy val accessControlCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - policySharingCase(preparedStandardTable(format)), - policyHistoryCase(preparedStandardTable(format)), - policyReplicationCase(preparedStandardTable(format)), - policyRetentionCase(format), - policyHistoryMaxAgeRejectedCase(preparedStandardTable(format)), - policyHistoryVersionsRejectedCase(preparedStandardTable(format)), - grantUnsharedRejectedCase(preparedStandardTable(format)), - grantAndRevokeCase(preparedStandardTable(format))) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** SET POLICY (SHARING=TRUE) records the sharing policy and the table remains queryable. */ - private def policySharingCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("accessControl.policy.sharing") { table => - table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") - - val policies = tableProps(table.spark, table.name).getOrElse("policies", "") - - assert( - policies.toLowerCase.contains("true") || policies.toLowerCase.contains("sharing"), - s"sharing policy not stored: $policies") - assert( - table.rows.size == standardSeedRowCount, - "table not queryable after SET POLICY (SHARING)") - } - - /** SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20) records the history policy and the table remains queryable. */ - private def policyHistoryCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("accessControl.policy.history") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=2D VERSIONS=20)") - - val policies = tableProps(table.spark, table.name).getOrElse("policies", "") - - assert( - policies.contains("20") || policies.toLowerCase.contains("history"), - s"history policy not stored: $policies") - assert( - table.rows.size == standardSeedRowCount, - "table not queryable after SET POLICY (HISTORY)") - } - - /** - * SET POLICY (REPLICATION) followed by UNSET POLICY (REPLICATION) leaves the table queryable with its 3 rows intact. - */ - private def policyReplicationCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("accessControl.policy.replication") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") - table.spark.sql( - s"ALTER TABLE ${table.name} UNSET POLICY (REPLICATION)") - - assert(table.rows.size == standardSeedRowCount) - } - - /** - * SET POLICY (RETENTION = 30d ON COLUMN foo_col_date ...) records the retention policy and the table remains - * queryable. - */ - private def policyRetentionCase(format: String): Plan.Case = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.date0.columnName}) " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .insert(standardSeedRowCount)()) - .test("accessControl.policy.retention") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (" + - s"RETENTION = 30d ON COLUMN ${Core.date0.columnName} WHERE pattern = 'yyyy-MM-dd-HH')") - - val policies = tableProps(table.spark, table.name).getOrElse("policies", "") - - assert( - policies.toLowerCase.contains("retention") || policies.contains("30"), - s"retention policy not stored: $policies") - assert( - table.rows.size == standardSeedRowCount, - "table not queryable after SET POLICY (RETENTION)") - } - - /** - * SET POLICY (HISTORY MAX_AGE=5D) exceeds the allowed range and is rejected with a BadRequestException stating the - * 1-to-3-day limit. - */ - private def policyHistoryMaxAgeRejectedCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("accessControl.policy.history.maxAge.rejected") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (HISTORY MAX_AGE=5D)")) - - assert( - exception.getMessage.contains("max age must be between 1 to 3 days"), - s"unexpected message: ${exception.getMessage.take(160)}") - } - - /** - * SET POLICY (HISTORY VERSIONS=200) exceeds the allowed range and is rejected with a BadRequestException stating the - * 2-to-100-version limit. - */ - private def policyHistoryVersionsRejectedCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("accessControl.policy.history.versions.rejected") { table => - val exception = Check.intercept[BadRequestException]( - table.spark.sql( - s"ALTER TABLE ${table.name} SET POLICY (HISTORY VERSIONS=200)")) - - assert( - exception.getMessage.contains("must be between 2 to 100 versions"), - s"unexpected message: ${exception.getMessage.take(160)}") - } - - /** - * GRANT SELECT on a table that is not marked shared is rejected with an IllegalArgumentException stating the table - * is not shared. - */ - private def grantUnsharedRejectedCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("accessControl.grantUnshared.rejected") { table => - val exception = Check.intercept[IllegalArgumentException]( - table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC")) - - assert( - exception.getMessage.contains("is not a shared table"), - s"unexpected message: ${exception.getMessage.take(160)}") - } - - /** - * On a shared table, GRANT SELECT TO PUBLIC makes SHOW GRANTS list SELECT for PUBLIC and the table stays queryable; - * REVOKE SELECT then removes that grant from SHOW GRANTS. - */ - private def grantAndRevokeCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation - .test("accessControl.grantAndRevoke") { table => - table.spark.sql(s"ALTER TABLE ${table.name} SET POLICY (SHARING=TRUE)") - table.spark.sql(s"GRANT SELECT ON TABLE ${table.name} TO PUBLIC") - - val grantsAfterGrant = table.spark - .sql(s"SHOW GRANTS ON TABLE ${table.name}") - .collect() - .map(row => (row.getString(0), row.getString(1))) - .toSet - assert( - grantsAfterGrant.contains(("SELECT", "PUBLIC")), - s"SHOW GRANTS did not include SELECT for PUBLIC: $grantsAfterGrant") - assert( - table.rows.size == standardSeedRowCount, - "the shared and granted table should stay queryable") - - table.spark.sql(s"REVOKE SELECT ON TABLE ${table.name} FROM PUBLIC") - val grantsAfterRevoke = table.spark - .sql(s"SHOW GRANTS ON TABLE ${table.name}") - .collect() - .map(row => (row.getString(0), row.getString(1))) - .toSet - assert( - !grantsAfterRevoke.contains(("SELECT", "PUBLIC")), - s"SHOW GRANTS retained SELECT for PUBLIC: $grantsAfterRevoke") - } - .copy(embeddedSkipReason = Some( - "The embedded test server has no OPA endpoint configured, so grantRole and " + - "listAclPolicies are no-ops that always report an empty ACL list. GRANT and REVOKE " + - "succeed without error, while SHOW GRANTS always returns an empty ACL list. The " + - "li-openhouse acceptance environment runs the assertions against its configured " + - "authorization service.")) - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala new file mode 100644 index 000000000..9cb046b91 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala @@ -0,0 +1,106 @@ +package harness + +/** + * The single integration file: the object that mixes every scenario in, the ordered catalog built from it, and the + * source entry points earlier consumers were written against. + * + * Adding a scenario is two lines here: one mixin on `Scenarios` and one named entry in + * `ScenarioCatalog.extensionContributions`. Explicit registration makes this file the complete catalog definition. + * + * Every capability trait and every support trait, mixed into one object. + * + * Mixing them here puts ScenarioKit first in the linearization, so its vals initialize before any capability's. This + * object is what a scenario body, a preparation list and the harness configuration are read from, so it exposes the + * shared kit surface (`dataSource`, `fileFormats`, the layout and preparation lists) alongside each capability's case + * list. + * + * Support traits expose reusable operations to later feature layers while the standard matrix includes only the + * selected scenario contributions. + */ +object Scenarios + extends ScenarioDataType + with ScenarioDml + with ScenarioDmlValidation + with ScenarioFileFormat + with ScenarioNestedType + with ScenarioPartitionEvolution + with ScenarioSchemaEvolution + with ScenarioTableProperty + with ChangelogSupport + +/** + * The ordered catalog of scenario-owned test cases. + * + * The catalog is built from two explicit lists. `foundationContributions` is the reusable DDL and DML base this + * branch froze; `extensionContributions` is where a later layer names the capabilities it adds. `contributions` + * merges the two and sorts by contribution name, giving every layer a deterministic order independent of list + * placement. + * + * A layer adds a capability through two append points: one mixin on `Scenarios` and one entry in + * `extensionContributions`. It writes its own scenario source and focused pin test while the foundation tests and + * shared kit remain stable. + * + * Composition is all this object does: a scenario body, a preparation and a case ID all belong to the capability that + * owns them. + */ +object ScenarioCatalog { + + /** + * The frozen foundation: the reusable DDL and DML capabilities this branch landed, named once, in alphabetical + * order. FoundationCatalogTest pins this list, so a later layer adds to `extensionContributions` instead. + */ + def foundationContributions: List[(String, List[TestCase])] = + List( + "dataTypeCases" -> Scenarios.dataTypeCases, + "dmlCases" -> Scenarios.dmlCases, + "dmlValidationCases" -> Scenarios.dmlValidationCases, + "fileFormatCases" -> Scenarios.fileFormatCases, + "nestedTypeCases" -> Scenarios.nestedTypeCases, + "partitionEvolutionCases" -> Scenarios.partitionEvolutionCases, + "schemaEvolutionCases" -> Scenarios.schemaEvolutionCases, + "tablePropertyCases" -> Scenarios.tablePropertyCases) + + /** + * The capabilities a later layer adds on top of the foundation, named the same way. This branch adds none, so the + * list is empty here and every entry below it in the file stays untouched as layers arrive. + */ + def extensionContributions: List[(String, List[TestCase])] = + List.empty + + /** Every capability contribution, named once, in the order the catalog integrates them. */ + def contributions: List[(String, List[TestCase])] = + (foundationContributions ++ extensionContributions).sortBy { case (name, _) => name } + + /** The deterministic ordered case catalog. */ + def cases: List[TestCase] = contributions.flatMap { case (_, contribution) => contribution } + + /** The case IDs in catalog order. Reading them is a Spark-free catalog operation. */ + def caseIds: List[String] = cases.map(_.id) + +} + +/** + * The entry point earlier consumers were written against. It preserves the source contract for `Plan.Case`, + * `Plan.cases`, `Plan.caseIds` and `Plan.bugReason`. + * + * This stateless facade forwards every member to `ScenarioCatalog` or to the case itself, keeping one catalog state. + * New code inside the harness reads `ScenarioCatalog` and `TestCase` directly. + */ +object Plan { + + /** The case type, which the harness now declares as `TestCase`. */ + type Case = TestCase + + /** The case constructor and extractor, so `Plan.Case(...)` still builds and matches a case. */ + val Case: TestCase.type = TestCase + + /** The deterministic ordered case catalog. */ + def cases: List[TestCase] = ScenarioCatalog.cases + + /** The case IDs in catalog order. */ + def caseIds: List[String] = ScenarioCatalog.caseIds + + /** The skip reason a known bug produces, phrased so a run log explains the skip. */ + def bugReason(testCase: TestCase): Option[String] = testCase.bugReason + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioChangelog.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioChangelog.scala deleted file mode 100644 index 33038ce84..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioChangelog.scala +++ /dev/null @@ -1,219 +0,0 @@ -package harness - -/** - * One changelog operation: the name its case carries, the statement it runs against the prepared table, and the - * change-type histogram the changelog view reports for the snapshot range that statement opened. - */ -final case class ChangelogOperation( - name: String, - statement: String => String, - expectedChangeCounts: Map[String, Long] -) - -/** - * Changelog: the row-level change feed `create_changelog_view` reports for a snapshot range, and what it reports once - * the start of that range has been expired. - * - * Operations: five reusable changelog operations (an append, an INSERT OVERWRITE that drops one row, a row-level - * DELETE, an UPDATE, and a MERGE that updates one row and inserts another), each followed by a changelog view opened - * at the seed snapshot; a changelog view over an append-only history with no start snapshot; and a changelog view - * opened at three start points inside an expired snapshot range. - * - * Preparation axes: in each of the two columnar formats, the standard seeded core table for the five operations and - * for the expired-range family, and the two-snapshot core table for the append-only history family. The operations are - * data, so a feature layer covers its own table mode by crossing `changelogOperations` with its own preparations. - * - * Case families: three families contributing 14 cases, 10 operation cases, 2 append-only history cases and 2 - * expired-range cases. - */ -trait ScenarioChangelog extends ScenarioKit { - - /** Every changelog case, one file format at a time. */ - lazy val changelogCases: List[Plan.Case] = - standardFormats.flatMap { format => - changelogOperationCasesFor(List(preparedStandardTable(format))) ++ - List( - appendOnlyHistoryCase(preparedTwoSnapshotTable(format)), - expiredRangeCase(preparedStandardTable(format))) - } - - /** - * The five row-level operations whose change feed the catalog reports. Every one starts from the standard three-row - * seed, so its expected histogram holds on any preparation that seeds those rows. - */ - lazy val changelogOperations: List[ChangelogOperation] = - List( - ChangelogOperation( - "changelog.append", - table => - s"INSERT INTO $table VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')", - Map("INSERT" -> 1L)), - ChangelogOperation( - "changelog.overwrite", - table => - s"INSERT OVERWRITE $table SELECT * FROM $table " + - s"WHERE ${Core.long0.columnName} <= 2", - Map("DELETE" -> 1L)), - ChangelogOperation( - "changelog.delete", - table => s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1", - Map("DELETE" -> 1L)), - ChangelogOperation( - "changelog.update", - table => - s"UPDATE $table SET ${Core.string0.columnName} = 'upd' " + - s"WHERE ${Core.long0.columnName} = 2", - Map("DELETE" -> 1L, "INSERT" -> 1L)), - ChangelogOperation( - "changelog.merge", - table => - s"MERGE INTO $table target " + - "USING (SELECT CAST(2 AS BIGINT) key " + - "UNION ALL SELECT CAST(9 AS BIGINT)) source " + - s"ON target.${Core.long0.columnName} = source.key " + - s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'm' " + - "WHEN NOT MATCHED THEN INSERT " + - s"(${Core.long0.columnName}, ${Core.int0.columnName}, " + - s"${Core.string0.columnName}, ${Core.double0.columnName}, " + - s"${Core.boolean0.columnName}, ${Core.date0.columnName}) " + - "VALUES (source.key, 9, 'row-9', 9.5, true, '2024-01-09-01')", - Map("DELETE" -> 1L, "INSERT" -> 2L))) - - /** The changelog cases for every operation on every preparation given, one preparation at a time. */ - def changelogOperationCasesFor( - preparations: List[TablePreparation[CoreTable.type]] - ): List[Plan.Case] = - preparations.flatMap(preparation => - changelogOperations.map(operation => changelogOperationCase(preparation, operation))) - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** The change-type histogram the named changelog view reports. */ - private def changeCounts(table: PreparedTable[CoreTable.type], view: String): Map[String, Long] = - table.spark - .sql(s"SELECT _change_type, count(*) FROM $view GROUP BY _change_type") - .collect() - .map(row => row.getString(0) -> row.getLong(1)) - .toMap - - /** - * Running the operation against a seeded table and opening a changelog view at the seed snapshot reports exactly the - * change types and counts that operation is defined to produce. - */ - private def changelogOperationCase( - preparation: TablePreparation[CoreTable.type], - operation: ChangelogOperation): Plan.Case = - preparation.test(operation.name) { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql(operation.statement(table.name)) - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('start-snapshot-id', '$seedSnapshotId'))") - .collect()(0) - .getString(0) - - val actualChangeCounts = changeCounts(table, view) - - assert( - actualChangeCounts == operation.expectedChangeCounts, - s"${operation.name} reported $actualChangeCounts, expected ${operation.expectedChangeCounts}") - } - - /** create_changelog_view over an append-only history reports 5 changes, all of change type INSERT. */ - private def appendOnlyHistoryCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("changelog.appendOnlyHistory") { table => - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}')") - .collect()(0) - .getString(0) - val actualChangeCounts = changeCounts(table, view) - - assert( - actualChangeCounts == Map("INSERT" -> 5L), - s"append-only changelog should report five inserts: $actualChangeCounts") - } - - /** - * After expire_snapshots removes a changelog start point, create_changelog_view over that start point either throws - * or reports fewer changes than the table's history holds, and any message it throws leaves expiration unnamed. The - * case covers three start points: an expired snapshot ID, a timestamp older than the whole history, and a timestamp - * inside the expired range. - */ - private def expiredRangeCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("changelog.expiredRange") { table => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - val snapshots = snapshotIds(table.spark, table.name) - val firstTimestamp = table.spark - .sql( - s"SELECT committed_at FROM ${table.name}.snapshots " + - "ORDER BY committed_at LIMIT 1") - .collect()(0) - .getTimestamp(0) - val middleTimestamp = table.spark - .sql( - s"SELECT committed_at FROM ${table.name}.snapshots " + - s"WHERE snapshot_id = ${snapshots(1)}") - .collect()(0) - .getTimestamp(0) - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - def changelogOutcome( - optionKey: String, - optionValue: String, - trueChangeCount: Long): String = - try { - val view = table.spark - .sql( - "CALL openhouse.system.create_changelog_view(" + - s"table => '${catalogRelative(table.name)}', " + - s"options => map('$optionKey', '$optionValue'))") - .collect()(0) - .getString(0) - val actualChangeCount = table.spark - .sql(s"SELECT count(*) FROM $view") - .collect()(0) - .getLong(0) - if (actualChangeCount < trueChangeCount) { - s"SILENT under-report: $actualChangeCount of $trueChangeCount true changes" - } else { - s"FULL: $actualChangeCount of $trueChangeCount" - } - } catch { - case exception: Throwable => - s"TYPED: ${exception.getClass.getSimpleName} :: " + - Option(exception.getMessage).getOrElse("").take(140) - } - - val outcomes = List( - "explicitExpiredId" -> changelogOutcome("start-snapshot-id", snapshots.head.toString, 5), - "timestampBeforeHistory" -> - changelogOutcome("start-timestamp", (firstTimestamp.getTime - 1000).toString, 5), - "timestampInsideExpiredRange" -> - changelogOutcome("start-timestamp", (middleTimestamp.getTime - 1).toString, 2)) - - outcomes.foreach { case (startPoint, outcome) => - println(s"DIAG changelog.expiredRange $startPoint: $outcome") - assert( - !outcome.startsWith("FULL"), - s"expired-lineage changelog returned full truth for $startPoint") - assert( - !outcome.toLowerCase.contains("expir"), - s"expired-lineage message now names expiration for $startPoint") - } - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioColumnTag.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioColumnTag.scala deleted file mode 100644 index f0f278152..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioColumnTag.scala +++ /dev/null @@ -1,43 +0,0 @@ -package harness - -/** - * Column tags: ALTER TABLE MODIFY COLUMN SET TAG records a classification on a column and leaves the values that - * column returns exactly as they were written. - * - * Operations: SET TAG = (PII) on the string column, followed by a read of that column. - * - * Preparation axes: the standard seeded core table in each of the two columnar formats. - * - * Case families: one family contributing 2 cases. - */ -trait ScenarioColumnTag extends ScenarioKit { - - /** The column-tag case, one file format at a time. */ - lazy val columnTagCases: List[Plan.Case] = - standardFormats.map(format => setTagCase(preparedStandardTable(format))) - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** - * ALTER TABLE MODIFY COLUMN SET TAG = (PII) tags a column, and queries keep returning the values the seed wrote. - */ - private def setTagCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("columnTag.setTag") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} MODIFY COLUMN " + - s"${Core.string0.columnName} SET TAG = (PII)") - - val values = table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - s"ORDER BY ${Core.long0.columnName}") - .collect() - .toSeq - .map(_.getString(0)) - - assert( - values == Seq("row-1", "row-2", "row-3"), - s"SET TAG changed the values the column returns: $values") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCompactionPlanning.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCompactionPlanning.scala deleted file mode 100644 index 032cc6a5d..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCompactionPlanning.scala +++ /dev/null @@ -1,137 +0,0 @@ -package harness - -import org.apache.spark.sql.SparkSession - -/** - * Compaction planning: rewrite_data_files packs data files into rewrite groups weighted by file length and spends a - * budget in file-sequence-number order, and the rewrite it commits preserves every row. - * - * Operations: rewrite_data_files with rewrite-all over a table whose data files are unevenly sized, and - * rewrite_data_files with rewrite-all over a table whose live data-file entries carry distinct, increasing - * file_sequence_numbers. - * - * Preparation axes: one table per family, built inside the case with write.distribution-mode=none so each insert - * commits its own data file. The bin-packing family runs in each of the two columnar formats. Sequence numbers order - * commits the same way in every file format, so the ordering family runs on Parquet alone. - * - * Case families: two families contributing 3 cases. - */ -trait ScenarioCompactionPlanning extends ScenarioKit { - - /** The bin-packing case in each columnar format, then the file-sequence ordering case on Parquet. */ - lazy val compactionPlanningCases: List[Plan.Case] = - standardFormats.map(format => - Plan.Case( - s"compactionPlanning.binPackByFileLength @ $format", - binPackByFileLengthCase(format))) ++ - List( - Plan.Case("compactionPlanning.fileSequenceOrder @ parquet", fileSequenceOrderCase)) - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** The count and the total byte size of the table's current data files. */ - private def dataFileStats(spark: SparkSession, table: String): (Long, Long) = { - val stats = spark - .sql(s"SELECT count(*), coalesce(sum(file_size_in_bytes), 0) FROM $table.data_files") - .collect()(0) - (stats.getLong(0), stats.getLong(1)) - } - - private def rewriteAll(spark: SparkSession, table: String): Unit = - spark.sql( - s"CALL openhouse.system.rewrite_data_files(table => '${catalogRelative(table)}', " + - "options => map('rewrite-all', 'true'))") - - /** - * Compacting a table whose data files are unevenly sized preserves the row count and every row's value, which is the - * observable result of packing rewrite groups by file length; the weighting itself is a planner decision that no SQL - * surface exposes. - */ - private def binPackByFileLengthCase(format: String)(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = TableTest.nextQualifiedTableName(ctx.namespace) - - withOwnedTable(spark.sql(_), table)( - spark.sql( - s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'write.distribution-mode'='none')")) { - // Unevenly sized data files: a tiny one, a small one, and a big one. - spark.sql(s"INSERT INTO $table VALUES (1,'a')") - spark.sql(s"INSERT INTO $table VALUES (2,'b'),(3,'c')") - spark.sql(s"INSERT INTO $table SELECT id, repeat('x', 200) FROM range(100, 400)") - - val (filesBefore, bytesBefore) = dataFileStats(spark, table) - assert(filesBefore >= 3, s"[$format] expected at least 3 uneven data files, got $filesBefore") - val rowsBefore = countOf(spark, s"SELECT count(*) FROM $table") - - rewriteAll(spark, table) - - val (filesAfter, bytesAfter) = dataFileStats(spark, table) - assert( - countOf(spark, s"SELECT count(*) FROM $table") == rowsBefore, - s"[$format] rewrite_data_files changed the row count from $rowsBefore") - val smallestRowValue = - spark.sql(s"SELECT s FROM $table WHERE id = 1").collect()(0).getString(0) - assert(smallestRowValue == "a", s"[$format] rewrite altered a row: id=1 s=$smallestRowValue") - - println( - s"DIAG compactionPlanning.binPackByFileLength[$format]: filesBefore=$filesBefore " + - s"bytesBefore=$bytesBefore filesAfter=$filesAfter bytesAfter=$bytesAfter rows=$rowsBefore") - } - } - - /** - * file_sequence_number is exposed on the live data-file entries of the entries metadata table and increases - * monotonically across commits, and rewrite_data_files with rewrite-all preserves the row count and the row set. A - * budgeted rewrite spends its budget in file-sequence-number order, so that column is the observable half of the - * ordering decision. - */ - private def fileSequenceOrderCase(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = TableTest.nextQualifiedTableName(ctx.namespace) - - withOwnedTable(spark.sql(_), table)( - spark.sql( - s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES (" + - "'write.format.default'='parquet', 'write.distribution-mode'='none')")) { - // Several commits produce several data files with distinct, increasing file-sequence-numbers. - val numberOfCommits = 4 - (0 until numberOfCommits).foreach { commitIndex => - spark.sql(s"INSERT INTO $table VALUES (${commitIndex}L, 'c$commitIndex')") - } - - val sequenceNumbers = spark - .sql( - s"SELECT file_sequence_number FROM $table.entries " + - "WHERE status != 2 AND data_file.content = 0 ORDER BY file_sequence_number") - .collect() - .toSeq - .map(_.getLong(0)) - assert( - sequenceNumbers.size >= numberOfCommits, - s"expected at least $numberOfCommits live data-file entries with sequence numbers, " + - s"got ${sequenceNumbers.size}: $sequenceNumbers") - assert( - sequenceNumbers == sequenceNumbers.sorted, - s"file sequence numbers not monotonic: $sequenceNumbers") - assert( - sequenceNumbers.distinct.size >= 2, - s"expected multiple distinct file sequence numbers, got ${sequenceNumbers.distinct}") - val rowsBefore = countOf(spark, s"SELECT count(*) FROM $table") - - rewriteAll(spark, table) - - assert( - countOf(spark, s"SELECT count(*) FROM $table") == rowsBefore, - s"rewrite changed the row count from $rowsBefore") - val keys = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) - assert(keys == (0 until numberOfCommits).map(_.toLong), s"rewrite altered the row set: $keys") - - println( - s"DIAG compactionPlanning.fileSequenceOrder: fileSequenceNumbers=" + - s"${sequenceNumbers.mkString(",")} filesAfter=" + - s"${dataFileStats(spark, table)._1} rows=$rowsBefore") - } - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioConcurrency.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioConcurrency.scala deleted file mode 100644 index bb7a33322..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioConcurrency.scala +++ /dev/null @@ -1,155 +0,0 @@ -package harness - -import java.util.concurrent.{ConcurrentLinkedQueue, CountDownLatch, TimeUnit} -import java.util.concurrent.atomic.AtomicInteger - -/** - * Concurrency: two writers racing on one table. Every write either commits or fails with a typed commit-conflict - * exception, and the table the race leaves behind is consistent with the writes that committed. - * - * Operations: two threads each running three single-row INSERTs against the same table, and two threads each running - * an UPDATE of the same row to a different value. - * - * Preparation axes: the standard seeded core table in each of the two columnar formats. The concurrency helpers are - * feature neutral, so a feature layer reuses them for its own table mode. - * - * Case families: two families contributing 4 cases. - */ -trait ScenarioConcurrency extends ScenarioKit { - - /** Every concurrency case, one file format at a time. */ - lazy val concurrencyCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - appendAppendCase(preparedStandardTable(format)), - updateUpdateCase(preparedStandardTable(format))) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** - * Runs every function on its own daemon thread, releases them together, and waits up to three minutes for all of - * them. Returns the throwables the threads raised, plus one for each thread still running at the deadline. - */ - protected def runConcurrently(functions: Seq[() => Unit]): Seq[Throwable] = { - val errors = new ConcurrentLinkedQueue[Throwable]() - val start = new CountDownLatch(1) - val threads = functions.zipWithIndex.map { case (function, index) => - val thread = new Thread( - () => - try { - start.await() - function() - } catch { - case interrupted: InterruptedException => - Thread.currentThread().interrupt() - errors.add(interrupted) - case throwable: Throwable => - errors.add(throwable) - }, - s"delta-harness-concurrent-$index") - thread.setDaemon(true) - thread - } - threads.foreach(_.start()) - start.countDown() - - val deadline = System.nanoTime() + TimeUnit.MINUTES.toNanos(3) - threads.foreach { thread => - val remainingNanos = deadline - System.nanoTime() - if (remainingNanos > 0) { - TimeUnit.NANOSECONDS.timedJoin(thread, remainingNanos) - } - } - - threads.filter(_.isAlive).foreach { thread => - errors.add( - new AssertionError(s"${thread.getName} did not complete within 3 minutes")) - thread.interrupt() - } - errors.toArray(Array.empty[Throwable]).toSeq - } - - /** A commit conflict the catalog reports through one of its typed commit, validation or transport exceptions. */ - protected def isTypedCommitConflict(throwable: Throwable): Boolean = - Exceptions.causeChain(throwable).exists { cause => - val className = cause.getClass.getName - className.contains("CommitFailed") || - className.contains("CommitStateUnknown") || - className.contains("Validation") || - className.contains("BadRequest") || - className.contains("WebClientResponse") - } - - /** - * Two threads concurrently insert 3 rows each; every insert either commits or fails with a typed commit-conflict - * exception, and the final row count matches 3 plus the number of inserts that actually committed. - */ - private def appendAppendCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("concurrency.appendAppend") { table => - val failureCount = new AtomicInteger(0) - def writer(base: Int): () => Unit = () => - (0 until 3).foreach { offset => - val value = base + offset - try { - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - s"(CAST($value AS BIGINT), $value, 'row-c', 1.5, true, '2024-01-09-01')") - } catch { - case exception: Throwable => - assert( - isTypedCommitConflict(exception), - "concurrent append failed with an untyped error: " + - s"${exception.getClass.getName}") - failureCount.incrementAndGet() - } - } - val threadErrors = runConcurrently(Seq(writer(100), writer(200))) - val expectedRowCount = 3 + 6 - failureCount.get - - assert( - threadErrors.isEmpty, - s"writer thread failed outside the insert loop: $threadErrors") - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == expectedRowCount.toString, - s"expected $expectedRowCount rows after ${failureCount.get} of 6 inserts hit a conflict") - } - - /** - * Two threads concurrently UPDATE the same row to different values; the row count stays at 3, and the final value is - * one of the two competing updates or the original seed value, with any failure being a typed commit conflict. - */ - private def updateUpdateCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("concurrency.updateUpdate") { table => - def updater(value: String): () => Unit = () => - try { - table.spark.sql( - s"UPDATE ${table.name} SET ${Core.string0.columnName} = '$value' " + - s"WHERE ${Core.long0.columnName} = 2") - } catch { - case exception: Throwable => - assert( - isTypedCommitConflict(exception), - "concurrent update failed with an untyped error: " + - s"${exception.getClass.getName}") - } - val threadErrors = runConcurrently(Seq(updater("AAA"), updater("BBB"))) - val finalValue = table.spark - .sql( - s"SELECT ${Core.string0.columnName} FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 2") - .collect()(0) - .getString(0) - - assert( - threadErrors.isEmpty, - s"updater thread failed with a non-conflict error: $threadErrors") - assert( - finalValue == "AAA" || finalValue == "BBB" || finalValue == "row-2", - s"concurrent updates produced a torn value: $finalValue") - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", - "concurrent updates should leave the row count at 3") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDataType.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDataType.scala index fa432d018..453d50196 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDataType.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDataType.scala @@ -13,12 +13,12 @@ import java.math.BigDecimal * Preparation axes: one unpartitioned TypesTable layout per file format, each seeded with three rows covering every * scalar column. * - * Case families: five families over three layouts, contributing 15 cases. + * Case families: five families over two layouts, contributing 10 cases. */ trait ScenarioDataType extends ScenarioKit { /** Every scalar-type case, one layout at a time. */ - lazy val dataTypeCases: List[Plan.Case] = + lazy val dataTypeCases: List[TestCase] = preparedTypesTables.flatMap(preparation => List( roundtripCase(preparation), @@ -54,7 +54,7 @@ trait ScenarioDataType extends ScenarioKit { * Selecting id, n, x, dec and str for the first seeded row reads back the exact long, int, double, decimal and * string values that were seeded. */ - private def roundtripCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + private def roundtripCase(preparation: TablePreparation[TypesTable.type]): TestCase = preparation.test("types.roundtrip") { table => val row = table.spark .sql( @@ -74,7 +74,7 @@ trait ScenarioDataType extends ScenarioKit { * Inserting a row with every non-key column NULL reads back as null for the int, double, string, timestamp and * timestamp_ntz columns. */ - private def nullsCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + private def nullsCase(preparation: TablePreparation[TypesTable.type]): TestCase = preparation.test("types.nulls") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES (" + @@ -90,7 +90,7 @@ trait ScenarioDataType extends ScenarioKit { } /** Inserting rows with double('NaN') and double('Infinity') reads back as NaN and positive infinity respectively. */ - private def specialFloatsCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + private def specialFloatsCase(preparation: TablePreparation[TypesTable.type]): TestCase = preparation.test("types.specialFloats") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES " + @@ -115,7 +115,7 @@ trait ScenarioDataType extends ScenarioKit { * Inserting a row at Long.MaxValue, Int.MaxValue and a max-precision decimal reads those boundary values back * unchanged. */ - private def boundariesCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + private def boundariesCase(preparation: TablePreparation[TypesTable.type]): TestCase = preparation.test("types.boundaries") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES " + @@ -139,7 +139,7 @@ trait ScenarioDataType extends ScenarioKit { } /** Inserting rows with a unicode string and an empty string reads each back unchanged. */ - private def unicodeAndEmptyCase(preparation: TablePreparation[TypesTable.type]): Plan.Case = + private def unicodeAndEmptyCase(preparation: TablePreparation[TypesTable.type]): TestCase = preparation.test("types.unicodeAndEmpty") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES " + diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDml.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDml.scala index 695b7f3ba..f11b574a5 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDml.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDml.scala @@ -13,19 +13,19 @@ import org.apache.spark.sql.functions.lit * observable state change within its family. Every operation is defined once here, so a feature layer covers its own * table mode by crossing these same definitions with its own preparations. * - * Preparation axes: ScenarioKit supplies the starting states. Six core layouts cross three file formats with - * partitioned and unpartitioned tables. Three date-partitioned layouts receive the partition-scoped writes. Six - * write-ordered layouts exercise the same catalog under a sort order. Six evolved layouts receive the 29 operations + * Preparation axes: ScenarioKit supplies the starting states. Four core layouts cross the two columnar formats with + * partitioned and unpartitioned tables. Two date-partitioned layouts receive the partition-scoped writes. Four + * write-ordered layouts exercise the same catalog under a sort order. Four evolved layouts receive the 29 operations * that address columns by name. Null-string variants isolate the one operation that requires a null value. * - * Case families: 804 cases in four families, `coreDmlCases` (312), `partitionedDmlCases` (6), `orderedDmlCases` (312) - * and `evolvedDmlCases` (174). + * Case families: 536 cases in four families, `coreDmlCases` (208), `partitionedDmlCases` (4), `orderedDmlCases` (208) + * and `evolvedDmlCases` (116). */ trait ScenarioDml extends ScenarioKit { import Rows._ /** Every DML case, in preparation order: core, partition-scoped, write-ordered, then evolved. */ - lazy val dmlCases: List[Plan.Case] = + lazy val dmlCases: List[TestCase] = coreDmlCases ++ partitionedDmlCases ++ orderedDmlCases ++ evolvedDmlCases /** @@ -93,24 +93,24 @@ trait ScenarioDml extends ScenarioKit { * Every DML case on the core preparations, plus the null-string DELETE on the same preparations extended with a * null-string row. */ - lazy val coreDmlCases: List[Plan.Case] = + lazy val coreDmlCases: List[TestCase] = preparedCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) ++ preparedNullStringCoreTables.flatMap(preparation => nullStringRowTestCases.map(_.runOn(preparation))) /** The partition-scoped writes on the partitioned preparations. */ - lazy val partitionedDmlCases: List[Plan.Case] = + lazy val partitionedDmlCases: List[TestCase] = preparedPartitionedCoreTables.flatMap(preparation => partitionedTableTestCases.map(_.runOn(preparation))) /** Every DML case on the write-ordered preparations, plus the null-string DELETE on their null-string form. */ - lazy val orderedDmlCases: List[Plan.Case] = + lazy val orderedDmlCases: List[TestCase] = preparedOrderedCoreTables.flatMap(preparation => orderedDmlTestCases.map(_.runOn(preparation))) ++ preparedNullStringOrderedCoreTables.flatMap(preparation => nullStringRowTestCases.map(_.runOn(preparation))) /** The cases that address columns by name, on the preparations that added a column. */ - lazy val evolvedDmlCases: List[Plan.Case] = + lazy val evolvedDmlCases: List[TestCase] = preparedEvolvedCoreTables.flatMap(preparation => testCasesCompatibleWithAnAddedColumn.map(_.runOn(preparation))) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDmlValidation.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDmlValidation.scala index 56b94ba7d..0f18ab83e 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDmlValidation.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDmlValidation.scala @@ -10,14 +10,14 @@ import org.apache.spark.sql.AnalysisException * INSERT INTO with fewer values than the table has columns, a MERGE whose UPDATE SET assigns one target column twice, * and a MERGE whose source matches one target row twice. * - * Preparation axes: the standard seeded core table in each of the two columnar formats. + * Preparation axes: the standard seeded core table in each columnar format. * * Case families: six families contributing 12 cases. */ trait ScenarioDmlValidation extends ScenarioKit { /** Every DML-validation case, one file format at a time. */ - lazy val dmlValidationCases: List[Plan.Case] = + lazy val dmlValidationCases: List[TestCase] = preparedCoreFormats.flatMap { preparation => List( nonExistentColumnCase(preparation), @@ -31,7 +31,7 @@ trait ScenarioDmlValidation extends ScenarioKit { // --- the preparations, shared helpers and case bodies the surface above composes --- /** DELETE with a WHERE clause on a nonexistent column is rejected with an AnalysisException naming that column. */ - private def nonExistentColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def nonExistentColumnCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("dmlValidation.nonExistentColumn") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( @@ -44,7 +44,7 @@ trait ScenarioDmlValidation extends ScenarioKit { * DELETE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an AnalysisException about * determinism. */ - private def nonDeterministicDeleteCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def nonDeterministicDeleteCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("dmlValidation.nonDeterministicDelete") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( @@ -57,7 +57,7 @@ trait ScenarioDmlValidation extends ScenarioKit { * UPDATE with a nondeterministic WHERE clause (rand() < 0.5) is rejected with an AnalysisException about * determinism. */ - private def nonDeterministicUpdateCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def nonDeterministicUpdateCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("dmlValidation.nonDeterministicUpdate") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( @@ -70,7 +70,7 @@ trait ScenarioDmlValidation extends ScenarioKit { * INSERT INTO with too few values for the table's columns is rejected with an AnalysisException about the missing * data columns. */ - private def insertArityCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def insertArityCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("dmlValidation.insertArity") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( @@ -84,7 +84,7 @@ trait ScenarioDmlValidation extends ScenarioKit { * assignments. */ private def mergeConflictingUpdatesCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("dmlValidation.mergeConflictingUpdates") { table => val keyColumn = Core.long0.columnName val stringColumn = Core.string0.columnName @@ -105,7 +105,7 @@ trait ScenarioDmlValidation extends ScenarioKit { * multi-row match. */ private def mergeCardinalityViolationCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("dmlValidation.mergeCardinalityViolation") { table => val keyColumn = Core.long0.columnName val stringColumn = Core.string0.columnName diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioEncryption.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioEncryption.scala deleted file mode 100644 index c8da2af6c..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioEncryption.scala +++ /dev/null @@ -1,44 +0,0 @@ -package harness - -import java.nio.file.{Files, Paths} - -/** - * Encryption: the OSS build writes table data in plaintext, because OpenHouse delegates table-data encryption to an - * external KMS plugin and the OSS build wires no KeyManagementClient into the catalog, leaving the default - * PlaintextEncryptionManager in place. - * - * Operations: read the trailing footer magic bytes of one data file. A Parquet footer reads PAR1 for plaintext and - * PARE under modular encryption regardless of compression, so that magic value settles which path wrote the file. - * - * Preparation axes: the standard seeded core table in Parquet, which is the format whose footer carries the marker. - * - * Case families: one family contributing 1 case. - */ -trait ScenarioEncryption extends ScenarioKit { - - /** The plaintext data-file case, on the standard seeded Parquet table. */ - lazy val encryptionCases: List[Plan.Case] = - List(dataFilePlaintextCase(preparedStandardTable("parquet"))) - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** A data file's Parquet footer magic bytes are the plaintext PAR1 marker. */ - private def dataFilePlaintextCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("encryption.dataFilePlaintext") { table => - val dataFilePath = table.spark - .sql(s"SELECT file_path FROM ${table.name}.data_files LIMIT 1") - .collect()(0) - .getString(0) - .stripPrefix("file:") - val bytes = Files.readAllBytes(Paths.get(dataFilePath)) - - assert( - bytes.length >= 8, - s"data file is too small to inspect: ${bytes.length} bytes") - val footerMagic = new String(bytes.takeRight(4), "US-ASCII") - assert( - footerMagic == "PAR1", - s"expected plaintext Parquet footer PAR1, got $footerMagic") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileFormat.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileFormat.scala index 7d0e7f499..e60541677 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileFormat.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileFormat.scala @@ -7,16 +7,16 @@ package harness * Operations: read the declared write.format.default from the table properties, then list the data files the * preparation wrote and compare their extensions against it. * - * Preparation axes: the twelve standard preparations that leave data files behind, which are the six core layouts - * (Parquet, ORC and Avro crossed with unpartitioned and date-partitioned) and the same six carrying a write sort - * order. A feature layer covers its own table mode by passing its own preparations to `layoutFormatCasesFor`. + * Preparation axes: the eight standard preparations that leave data files behind, which are the four core layouts + * (Parquet and ORC crossed with unpartitioned and date-partitioned) and the same four carrying a write sort order. A + * feature layer covers its own table mode by passing its own preparations to `layoutFormatCasesFor`. * - * Case families: one family, `format.materialization`, contributing 12 cases. + * Case families: one family, `format.materialization`, contributing 8 cases. */ trait ScenarioFileFormat extends ScenarioKit { /** The format-materialization case on every standard preparation that writes data files. */ - lazy val fileFormatCases: List[Plan.Case] = layoutFormatCasesFor(layoutFormatPreparations) + lazy val fileFormatCases: List[TestCase] = layoutFormatCasesFor(layoutFormatPreparations) /** * The format-materialization case for each preparation given: every data file the preparation wrote carries the @@ -26,7 +26,7 @@ trait ScenarioFileFormat extends ScenarioKit { */ def layoutFormatCasesFor( preparations: List[TablePreparation[CoreTable.type]] - ): List[Plan.Case] = + ): List[TestCase] = preparations.map { preparation => preparation.test("format.materialization") { table => val before = table.state diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileReplication.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileReplication.scala deleted file mode 100644 index 7005e4ffe..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileReplication.scala +++ /dev/null @@ -1,82 +0,0 @@ -package harness - -import org.apache.iceberg.Table -import org.apache.iceberg.spark.Spark3Util -import java.util.{Map => JavaMap} -import scala.util.Try - -/** - * File replication: the output-file property the writer stamps so the file system can set a block replication factor - * on the files a commit produces. - * - * Operations: read OutputFileFactory.FILE_REPLICATION_FACTOR, build an OutputFileFactory carrying a replication - * factor, read the property map that factory stamps onto its output files, and write to the table afterwards. - * - * Preparation axes: one format-version-2 table built inside the case, because the case needs an Iceberg Table handle - * to build a factory from. - * - * Case families: one family contributing 1 case. - */ -trait ScenarioFileReplication extends ScenarioKit { - - /** The output-file replication property case. */ - lazy val fileReplicationCases: List[Plan.Case] = - List( - Plan.Case("fileReplication.outputFileProperty @ core", outputFilePropertyCase)) - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** - * OutputFileFactory exposes FILE_REPLICATION_FACTOR as "file-replication-factor", and a factory built with a - * replication factor stamps that key into the property map of the output files it creates. Writes made through the - * table afterwards still return the correct rows. The key is the one HDFS reads to set block replication on an - * output file when a replication factor is supplied to the factory, and the delete-file write path is the one path - * that supplies one. Reflection reaches the builder and getProperties because some Iceberg artifacts leave them out - * of the public compiled API, so a direct reference would fail to compile against those artifacts. - */ - private def outputFilePropertyCase(ctx: Ctx): Unit = { - val spark = ctx.spark - val outputFileFactoryClass = Class.forName("org.apache.iceberg.io.OutputFileFactory") - - val replicationKeyField = Try(outputFileFactoryClass.getField("FILE_REPLICATION_FACTOR")) - assert(replicationKeyField.isSuccess, "OutputFileFactory.FILE_REPLICATION_FACTOR is absent") - val replicationKey = replicationKeyField.get.get(null).asInstanceOf[String] - assert( - replicationKey == "file-replication-factor", - s"""expected FILE_REPLICATION_FACTOR to equal "file-replication-factor", got "$replicationKey"""") - - val table = TableTest.nextQualifiedTableName(ctx.namespace) - withOwnedTable(spark.sql(_), table)( - spark.sql( - s"CREATE TABLE $table (id bigint, s string) USING $dataSource " + - "TBLPROPERTIES ('format-version'='2')")) { - spark.sql(s"INSERT INTO $table VALUES (1,'a'),(2,'b')") - val icebergTable = Spark3Util.loadIcebergTable(spark, table) - val builder = outputFileFactoryClass - .getMethod("builderFor", classOf[Table], classOf[Int], classOf[Long]) - .invoke(null, icebergTable, Int.box(1), Long.box(1L)) - val replicationFactorMethod = - Try(builder.getClass.getMethod("replicationFactor", classOf[Short])) - assert( - replicationFactorMethod.isSuccess, - "OutputFileFactory.Builder.replicationFactor(short) is absent") - replicationFactorMethod.get.invoke(builder, Short.box(2.toShort)) - val factory = Option(builder.getClass.getMethod("build").invoke(builder)) - .getOrElse(throw new AssertionError("OutputFileFactory build returned null")) - - val getProperties = outputFileFactoryClass.getDeclaredMethod("getProperties") - getProperties.setAccessible(true) - val outputFileProperties = - getProperties.invoke(factory).asInstanceOf[JavaMap[String, String]] - assert( - outputFileProperties.get(replicationKey) == "2", - s"expected output-file property $replicationKey=2 stamped by the factory, " + - s"got ${outputFileProperties.get(replicationKey)}") - - spark.sql(s"INSERT INTO $table VALUES (3,'c')") - val keys = spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) - assert(keys == Seq(1L, 2L, 3L), s"rows wrong after write: $keys") - } - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioIncrementalRead.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioIncrementalRead.scala deleted file mode 100644 index 9af630008..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioIncrementalRead.scala +++ /dev/null @@ -1,93 +0,0 @@ -package harness - -/** - * Incremental read: a scan bounded by a start and an end snapshot returns the rows the snapshots in that range - * appended, and nothing else. - * - * Operations: an incremental scan across an append, across a row-level DELETE, across an INSERT OVERWRITE that only - * removes rows, across an UPDATE, and across the second commit of a two-snapshot history. - * - * Preparation axes: in each of the two columnar formats, the standard seeded core table for the four - * operation-bounded scans, and the two-snapshot core table for the scan between the two seeded commits. - * - * Case families: five families contributing 10 cases. - */ -trait ScenarioIncrementalRead extends ScenarioKit { - - /** Every incremental-read case, one file format at a time. */ - lazy val incrementalReadCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - incrementalCase( - preparedStandardTable(format), - "incrementalRead.append", - table => - s"INSERT INTO $table VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')", - 1), - incrementalCase( - preparedStandardTable(format), - "incrementalRead.delete", - table => s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1", - 0), - incrementalCase( - preparedStandardTable(format), - "incrementalRead.overwrite", - table => - s"INSERT OVERWRITE $table SELECT * FROM $table " + - s"WHERE ${Core.long0.columnName} <= 2", - 0), - incrementalCase( - preparedStandardTable(format), - "incrementalRead.update", - table => - s"UPDATE $table SET ${Core.string0.columnName} = 'upd' " + - s"WHERE ${Core.long0.columnName} = 2", - 0), - betweenSnapshotsCase(preparedTwoSnapshotTable(format))) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** The number of rows an incremental scan between the two snapshot IDs returns. */ - private def incrementalRowCount( - table: PreparedTable[CoreTable.type], - startSnapshotId: Long, - endSnapshotId: Long): Long = - table.spark.read - .format("iceberg") - .option("start-snapshot-id", startSnapshotId) - .option("end-snapshot-id", endSnapshotId) - .load(table.name) - .count() - - /** - * Running the statement against a seeded table and scanning from the seed snapshot to the snapshot the statement - * committed returns exactly `expectedRowCount` rows. - */ - private def incrementalCase( - preparation: TablePreparation[CoreTable.type], - caseName: String, - statement: String => String, - expectedRowCount: Long): Plan.Case = - preparation.test(caseName) { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - table.spark.sql(statement(table.name)) - val currentSnapshotId = snapshotIds(table.spark, table.name).last - val addedRowCount = incrementalRowCount(table, seedSnapshotId, currentSnapshotId) - - assert( - addedRowCount == expectedRowCount, - s"$caseName returned $addedRowCount rows, expected $expectedRowCount") - } - - /** An incremental read spanning both snapshots of the two-snapshot table returns the 2 rows the second one added. */ - private def betweenSnapshotsCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("incrementalRead.betweenSnapshots") { table => - val snapshots = snapshotIds(table.spark, table.name) - val addedRowCount = incrementalRowCount(table, snapshots(0), snapshots(1)) - - assert(addedRowCount == 2, s"the second commit added 2 rows, scan returned $addedRowCount") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala index 10da0fad8..e4b63f86a 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala @@ -10,7 +10,7 @@ import java.util.concurrent.TimeUnit * Every capability trait extends this kit, so mixing them into `object Scenarios` puts ScenarioKit first in the * linearization and its vals initialize before any capability's. It holds copy-on-write layouts and preparations only; * each feature layer carries its own kit that extends this one. `protected` members are the shared kit; `public` ones - * are also consumed by `object Plan` and by the catalog tests. + * are also consumed by `object ScenarioCatalog`, `object Plan` and the catalog tests. */ trait ScenarioKit { @@ -53,14 +53,12 @@ trait ScenarioKit { protected val partitionings: List[Partitioning] = List(unpartitioned, partitionedByDate) - /** Every file format the catalog writes. This is the single source for a format list anywhere in the harness. */ - val fileFormats: List[String] = List("parquet", "orc", "avro") - /** - * The two columnar formats every capability family runs on when it is crossed by format. The file-format capability - * itself covers the whole of `fileFormats`; every other family covers these two. + * Every file format the standard matrix runs on. This is the single source for a format list anywhere in the + * harness, so every format-crossed family covers both columnar formats. A format beyond these two is proven by the + * file-format extension layer, which supplies its own list. */ - val standardFormats: List[String] = List("parquet", "orc") + val fileFormats: List[String] = List("parquet", "orc") /** One copy-on-write table in `format`, shaped by `partitioning`, labelled for its case IDs. */ private def coreLayout(partitioning: Partitioning, format: String): Layout = @@ -81,13 +79,6 @@ trait ScenarioKit { val partitionedLayouts: List[Layout] = fileFormats.map(format => coreLayout(partitionedByDate, format)) - /** The Parquet and ORC core layouts, each crossed with both partitionings. */ - val parquetAndOrcLayouts: List[Layout] = - for { - format <- standardFormats - partitioning <- partitionings - } yield coreLayout(partitioning, format) - /** * The standard seed writes three deterministic rows with keys 1, 2 and 3. Row `n` holds key `n` in the long column, * `n` in the int column, `row-n` in the string column, `n.5` in the double column, `n % 2 == 0` in the boolean @@ -150,7 +141,10 @@ trait ScenarioKit { val preparedEmptyCoreTables: List[TablePreparation[CoreTable.type]] = layouts.map(layout => TablePreparation(layout.label, create(layout))) - /** The CREATE statement for an unpartitioned core table in `format`. */ + /** + * The CREATE statement for an unpartitioned core table in `format`. This generic substrate contributes zero cases + * and gives later capability layers a stable shared starting point. + */ protected def coreCreate(table: String, format: String): String = coreLayout(unpartitioned, format).create(table) @@ -167,14 +161,16 @@ trait ScenarioKit { format, create(coreLayout(unpartitioned, format)).insert(standardSeedRowCount)()) - /** The standard seeded table in each of the two columnar formats. */ + /** The standard seeded table in each file format. */ val preparedCoreFormats: List[TablePreparation[CoreTable.type]] = - standardFormats.map(preparedStandardTable) + fileFormats.map(preparedStandardTable) /** * An unpartitioned core table in `format` holding five rows across two snapshots: the standard seed, then rows 4 and * 5. The step between the two commits holds until the wall clock passes the seed commit's timestamp, so the two * snapshots carry distinct commit times and a timestamp-bounded read resolves to exactly one of them. + * + * Every family that reads history needs this shape, so the shared kit owns it. */ protected def preparedTwoSnapshotTable(format: String): TablePreparation[CoreTable.type] = TablePreparation( @@ -327,11 +323,12 @@ trait ScenarioKit { protected def catalogRelative(table: String): String = table.stripPrefix("openhouse.") + /** One core row in the seed shape, keyed by `long` and tagged in the string column. */ protected def coreRow(long: Long, tag: String): String = s"(CAST($long AS BIGINT), ${long.toInt}, '$tag', ${long}.5, false, '2024-01-01-00')" // The Spark data source used by CREATE TABLE statements. The LinkedIn adapter overrides this before building - // Plan.cases. Catalog procedure calls still use the catalog name "openhouse". + // ScenarioCatalog.cases. Catalog procedure calls still use the catalog name "openhouse". var dataSource: String = "iceberg" protected def tableProps(spark: SparkSession, table: String): Map[String, String] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioLocking.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioLocking.scala deleted file mode 100644 index 36c92f264..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioLocking.scala +++ /dev/null @@ -1,114 +0,0 @@ -package harness - -/** - * Table locking: while a table carries a REST lock, the catalog rejects the commits that would change it, and - * deleting the lock lets them through again. - * - * Operations: POST the lock endpoint, then run an UPDATE and an expire_snapshots call against the locked table, then - * DELETE the lock and run each of them again. The lock endpoint has no SQL surface, so both cases drive it over HTTP - * against the embedded server, which runs the same TablesController and TablesServiceImpl as production. Both cases - * hold the lock through the shared lock boundary, which checks every lock and release response and releases the lock - * once, whichever way the case ends. - * - * Preparation axes: each case builds its own parquet core table and seeds it directly, because the REST path - * addresses the table by its database and table name. - * - * Case families: two families contributing 2 cases. - */ -trait ScenarioLocking extends ScenarioKit { - - /** The lock cases, each driven over HTTP against the embedded server. */ - lazy val lockingCases: List[Plan.Case] = - List( - Plan.Case("lock.enforcement @ embedded", lockEnforcement), - Plan.Case("lock.starvesMaintenance @ embedded", lockStarvesMaintenance)) - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** - * POSTing a table lock causes a following Spark UPDATE to be rejected server-side with LOCKED_TABLE_OPERATION, and - * DELETEing the lock lets a later UPDATE apply. - */ - private def lockEnforcement(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = TableTest.nextQualifiedTableName(ctx.namespace) - val Array(database, tableName) = table.stripPrefix("openhouse.").split("\\.", 2) - - withOwnedTable(spark.sql(_), table)(spark.sql(coreCreate(table, "parquet"))) { - spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, standardSeedRowCount)}") - withTableLock(lockRequest(ctx, database, tableName), unlockRequest(ctx, database, tableName)) { - releaseLock => - val lockedFailure = Check.intercept[Exception]( - spark.sql( - s"UPDATE $table SET ${Core.string0.columnName} = 'locked-write' " + - s"WHERE ${Core.long0.columnName} = 1")) - assert( - Exceptions.causeChain(lockedFailure).exists(cause => - Option(cause.getMessage).exists(_.toLowerCase.contains("locked"))), - s"expected a locked-table rejection, got: ${lockedFailure.getMessage.take(200)}") - - releaseLock() - spark.sql( - s"UPDATE $table SET ${Core.string0.columnName} = 'unlocked-write' " + - s"WHERE ${Core.long0.columnName} = 1") - assert( - countOf( - spark, - s"SELECT count(*) FROM $table WHERE ${Core.string0.columnName} = 'unlocked-write'") == "1", - "post-unlock update did not apply") - } - } - } - - /** - * While a table is REST-locked, an expire_snapshots call is rejected and snapshots keep accumulating. After the lock - * is deleted, expire_snapshots succeeds and the snapshot count drops, so the lock holds off every maintenance commit - * for as long as it is held. - */ - private def lockStarvesMaintenance(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = TableTest.nextQualifiedTableName(ctx.namespace) - val Array(database, tableName) = table.stripPrefix("openhouse.").split("\\.", 2) - val expireSnapshots = - s"CALL openhouse.system.expire_snapshots(table => '${catalogRelative(table)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', retain_last => 1)" - - withOwnedTable(spark.sql(_), table)(spark.sql(coreCreate(table, "parquet"))) { - spark.sql(s"INSERT INTO $table ${RowGenerator.valuesClause(Core, standardSeedRowCount)}") - spark.sql( - s"INSERT INTO $table VALUES (CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - withTableLock(lockRequest(ctx, database, tableName), unlockRequest(ctx, database, tableName)) { - releaseLock => - val snapshotsBefore = countOf(spark, s"SELECT count(*) FROM $table.snapshots") - - val lockedFailure = Check.intercept[Exception](spark.sql(expireSnapshots)) - assert( - Exceptions.causeChain(lockedFailure).exists(cause => - Option(cause.getMessage).exists(_.toLowerCase.contains("locked"))), - "expected a LOCKED rejection for the maintenance commit: " + - s"${lockedFailure.getClass.getName} " + - Option(lockedFailure.getMessage).getOrElse("").take(180)) - spark.sql(s"REFRESH TABLE $table") - assert( - countOf(spark, s"SELECT count(*) FROM $table.snapshots") == snapshotsBefore, - "a locked table keeps every snapshot it holds") - - releaseLock() - spark.sql(expireSnapshots) - spark.sql(s"REFRESH TABLE $table") - assert( - countOf(spark, s"SELECT count(*) FROM $table.snapshots").toLong < snapshotsBefore.toLong, - "maintenance must proceed after unlock") - } - } - } - - /** The POST that takes the REST lock on the named table. */ - private def lockRequest(ctx: Ctx, database: String, tableName: String): () => (Int, String) = - () => Rest.post(ctx, s"/v1/databases/$database/tables/$tableName/lock", """{"locked":true}""") - - /** The DELETE that releases the REST lock on the named table. */ - private def unlockRequest(ctx: Ctx, database: String, tableName: String): () => (Int, String) = - () => Rest.delete(ctx, s"/v1/databases/$database/tables/$tableName/lock") - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMaintenance.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMaintenance.scala deleted file mode 100644 index e024268a4..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMaintenance.scala +++ /dev/null @@ -1,169 +0,0 @@ -package harness - -import java.nio.file.{Files, Paths} -import java.nio.file.attribute.FileTime - -/** - * Maintenance: the procedures that rewrite a table's files and metadata without changing the rows a reader sees. - * - * Operations: expire_snapshots down to the newest snapshot, rewrite_data_files over the table's data files, - * remove_orphan_files over the table's directory, rewrite_manifests over a fragmented manifest list, remove_orphan_ - * files against a planted backdated stray file, and rewrite_data_files across an ADD COLUMN. - * - * Preparation axes: in each of the two columnar formats, the two-snapshot core table for the three procedures that run - * over an existing history, an unseeded core table for the manifest family, which fragments the manifest list itself, - * and the standard seeded core table for the planted-orphan and schema-evolution families. - * - * Case families: six families contributing 12 cases. - */ -trait ScenarioMaintenance extends ScenarioKit { - - /** Every maintenance case, one file format at a time. */ - lazy val maintenanceCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - expireSnapshotsCase(preparedTwoSnapshotTable(format)), - rewriteDataFilesCase(preparedTwoSnapshotTable(format)), - removeOrphanFilesCase(preparedTwoSnapshotTable(format)), - rewriteManifestsCase(preparedEmptyStandardTable(format)), - removeOrphanFilesPlantedCase(preparedStandardTable(format)), - rewriteDataFilesAfterAddColumnCase(preparedStandardTable(format))) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** expire_snapshots with retain_last=1 removes the seed snapshot and leaves all 5 current rows unchanged. */ - private def expireSnapshotsCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("maintenance.expireSnapshots") { table => - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - assert( - table.rows.size == 5, - "expire_snapshots changed the current data") - assert( - table.snapshotCount < table.preparedSnapshotCount, - "expire_snapshots did not remove a snapshot: " + - s"${table.preparedSnapshotCount} -> ${table.snapshotCount}") - } - - /** rewrite_data_files compacts the data files and leaves all 5 rows unchanged. */ - private def rewriteDataFilesCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("maintenance.rewriteDataFiles") { table => - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}')") - - assert(table.rows.size == 5, "compaction changed rows") - } - - /** remove_orphan_files over a table with no stray files leaves all 5 rows unchanged. */ - private def removeOrphanFilesCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("maintenance.removeOrphanFiles") { table => - table.spark.sql( - "CALL openhouse.system.remove_orphan_files(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2020-01-01 00:00:00')") - - assert(table.rows.size == 5, "orphan removal changed rows") - } - - /** - * After 5 single-row inserts fragment the manifest list, rewrite_manifests compacts it to fewer manifests while - * preserving all 5 rows. - */ - private def rewriteManifestsCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("maintenance.rewriteManifests") { table => - (1 to 5).foreach(index => - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - coreRow(index, s"r$index"))) - val manifestCountBefore = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.manifests") - .collect()(0) - .getLong(0) - table.spark.sql( - "CALL openhouse.system.rewrite_manifests(" + - s"table => '${catalogRelative(table.name)}', " + - "use_caching => false)") - val manifestCountAfter = table.spark - .sql(s"SELECT count(*) FROM ${table.name}.manifests") - .collect()(0) - .getLong(0) - - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "5", - "rewrite_manifests should preserve the five rows") - assert( - manifestCountBefore >= 2 && - manifestCountAfter < manifestCountBefore, - "rewrite_manifests should compact the manifest set: " + - s"before=$manifestCountBefore after=$manifestCountAfter") - } - - /** - * remove_orphan_files deletes a planted, backdated stray file next to a real data file while the table's 3 live rows - * remain intact. - */ - private def removeOrphanFilesPlantedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("maintenance.removeOrphanFiles.planted") { table => - val dataFile = table.spark - .sql(s"SELECT file_path FROM ${table.name}.files LIMIT 1") - .collect()(0) - .getString(0) - .stripPrefix("file:") - val orphanFile = Paths - .get(dataFile) - .getParent - .resolve(s"${table.name.split('.').last}_orphan.parquet") - Files.write(orphanFile, "not-a-real-parquet".getBytes) - Files.setLastModifiedTime(orphanFile, FileTime.fromMillis(1546300800000L)) - - table.spark.sql( - "CALL openhouse.system.remove_orphan_files(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2020-01-01 00:00:00')") - assert( - Files.notExists(orphanFile), - "remove_orphan_files should delete the planted orphan") - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", - "remove_orphan_files should preserve live data") - } - - /** - * Compacting a table after an ADD COLUMN and inserts into the new column preserves all rows, the new column's - * non-null values, and null for rows written before the column was added. - */ - private def rewriteDataFilesAfterAddColumnCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("maintenance.rewriteDataFiles.afterAddColumn") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert10") - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}')") - - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "5", - "compaction should preserve 5 rows") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} WHERE extra_col IN (42, 43)") == "2", - "compaction should preserve both evolved values") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} WHERE extra_col IS NULL") == "3", - "pre-evolution rows should remain null") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMetadataTable.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMetadataTable.scala deleted file mode 100644 index b091881d5..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMetadataTable.scala +++ /dev/null @@ -1,96 +0,0 @@ -package harness - -/** - * Metadata tables: the hidden metadata columns a scan exposes and the Iceberg metadata tables the catalog serves - * alongside every table. - * - * Operations: select the hidden _file, _pos, _spec_id and _partition columns; query every metadata table the catalog - * serves (entries, files, manifests, snapshots, history, refs, partitions, metadata_log_entries, data_files and the - * all_* variants); and read the snapshot, history, files and manifests counts of a two-snapshot table. - * - * Preparation axes: in each of the two columnar formats, the standard seeded core table for the hidden-column family - * and the two-snapshot core table for the two families that count metadata rows. - * - * Case families: three families contributing 6 cases. - */ -trait ScenarioMetadataTable extends ScenarioKit { - - /** Every metadata-table case, one file format at a time. */ - lazy val metadataTableCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - hiddenColumnsCase(preparedStandardTable(format)), - tableSweepCase(preparedTwoSnapshotTable(format)), - snapshotAndHistoryCase(preparedTwoSnapshotTable(format))) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** - * Selecting the hidden metadata columns _file, _pos, _spec_id and _partition returns one row per seed row, each with - * a populated file path and a non-negative position. - */ - private def hiddenColumnsCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("metadata.hiddenColumns") { table => - val rows = table.spark - .sql(s"SELECT _file, _pos, _spec_id, _partition FROM ${table.name}") - .collect() - .toSeq - - assert( - rows.size == 3, - s"hidden metadata columns should return 3 rows, got ${rows.size}") - assert( - rows.forall(row => Option(row.getString(0)).exists(_.nonEmpty)), - "_file should be populated for every row") - assert( - rows.forall(_.getLong(1) >= 0), - "_pos should be non-negative for every row") - } - - /** - * Every Iceberg metadata table is queryable without error, and the snapshots metadata table reports the table's 2 - * snapshots. - */ - private def tableSweepCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("metadata.tableSweep") { table => - val metadataTables = Seq( - "entries", - "files", - "manifests", - "snapshots", - "history", - "refs", - "partitions", - "metadata_log_entries", - "data_files", - "all_data_files", - "all_manifests", - "all_entries", - "all_files") - metadataTables.foreach { metadataTable => - table.spark.sql(s"SELECT count(*) FROM ${table.name}.`$metadataTable`").collect() - } - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}.snapshots") == "2", - "snapshot metadata should contain two snapshots") - } - - /** - * The snapshots and history metadata tables each report the table's 2 snapshots, and the files and manifests - * metadata tables each report at least 1 row. - */ - private def snapshotAndHistoryCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("metadata.snapshotAndHistory") { table => - def metadataRowCount(metadataTable: String): Long = - table.spark - .sql(s"SELECT count(*) FROM ${table.name}.$metadataTable") - .collect()(0) - .getLong(0) - - assert(metadataRowCount("snapshots") == 2) - assert(metadataRowCount("history") == 2) - assert(metadataRowCount("files") >= 1 && metadataRowCount("manifests") >= 1) - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNamespace.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNamespace.scala deleted file mode 100644 index 044488275..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNamespace.scala +++ /dev/null @@ -1,48 +0,0 @@ -package harness - -/** - * Namespaces: the catalog serves the databases it is configured with, and it rejects the statements that would create - * or drop one. - * - * Operations: CREATE NAMESPACE and DROP NAMESPACE. - * - * Preparation axes: the standard seeded core table in each of the two columnar formats, which gives each case a live - * catalog session and a table lifecycle. - * - * Case families: two families contributing 4 cases. - */ -trait ScenarioNamespace extends ScenarioKit { - - /** Every namespace case, one file format at a time. */ - lazy val namespaceCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - createRejectedCase(preparedStandardTable(format)), - dropRejectedCase(preparedStandardTable(format))) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** CREATE NAMESPACE is rejected with an UnsupportedOperationException naming the unsupported operation. */ - private def createRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("namespace.create.rejected") { table => - val exception = Check.intercept[UnsupportedOperationException]( - table.spark.sql("CREATE NAMESPACE openhouse.a_new_db")) - - assert( - exception.getMessage.contains("not supported"), - s"unexpected message: ${exception.getMessage.take(160)}") - } - - /** DROP NAMESPACE is rejected with an UnsupportedOperationException naming the unsupported operation. */ - private def dropRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("namespace.drop.rejected") { table => - val exception = Check.intercept[UnsupportedOperationException]( - table.spark.sql("DROP NAMESPACE openhouse.dbMatrix")) - - assert( - exception.getMessage.contains("not supported"), - s"unexpected message: ${exception.getMessage.take(160)}") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNestedType.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNestedType.scala index 2c525444b..f4455e71a 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNestedType.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNestedType.scala @@ -13,12 +13,12 @@ package harness * array, map and doubly-nested struct values; plus the standard seeded core table in Parquet and ORC for the two * struct-evolution families, which build and drop their own side table. * - * Case families: nine families contributing 25 cases, 21 on the nested layouts and 4 on the standard formats. + * Case families: nine families contributing 18 cases, 14 on the nested layouts and 4 on the core formats. */ trait ScenarioNestedType extends ScenarioKit { /** Every nested-type case: the reads and writes on the nested layouts, then the struct-evolution cases. */ - lazy val nestedTypeCases: List[Plan.Case] = + lazy val nestedTypeCases: List[TestCase] = preparedNestedTables.flatMap(preparation => List( roundtripCase(preparation), @@ -55,7 +55,7 @@ trait ScenarioNestedType extends ScenarioKit { * Selecting the top-level id alongside struct, array, map and nested-struct fields reads back exactly the seeded * values for all 3 rows. */ - private def roundtripCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + private def roundtripCase(preparation: TablePreparation[NestedTable.type]): TestCase = preparation.test("nested.roundtrip") { table => val actual = table.spark .sql( @@ -85,7 +85,7 @@ trait ScenarioNestedType extends ScenarioKit { } /** Selecting only a nested struct field (s.x) returns just that field's values for all 3 rows, in id order. */ - private def projectFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + private def projectFieldCase(preparation: TablePreparation[NestedTable.type]): TestCase = preparation.test("nested.projectField") { table => val actual = table.spark .sql(s"SELECT s.x FROM ${table.name} ORDER BY id") @@ -97,7 +97,7 @@ trait ScenarioNestedType extends ScenarioKit { } /** Filtering WHERE s.x = 2 on a nested struct field returns only the matching row's id. */ - private def filterNestedFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + private def filterNestedFieldCase(preparation: TablePreparation[NestedTable.type]): TestCase = preparation.test("nested.filterNestedField") { table => val actual = table.spark .sql(s"SELECT id FROM ${table.name} WHERE s.x = 2 ORDER BY id") @@ -109,7 +109,7 @@ trait ScenarioNestedType extends ScenarioKit { } /** UPDATE SET s.x = 99 WHERE id = 2 changes only that row's nested field and leaves every other row unchanged. */ - private def updateStructFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + private def updateStructFieldCase(preparation: TablePreparation[NestedTable.type]): TestCase = preparation.test("nested.updateStructField") { table => table.spark.sql( s"UPDATE ${table.name} SET s.x = 99 WHERE id = 2") @@ -130,7 +130,7 @@ trait ScenarioNestedType extends ScenarioKit { * MERGE WHEN NOT MATCHED THEN INSERT with a fully nested source row adds a 4th row whose nested struct field reads * back as inserted. */ - private def mergeInsertCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + private def mergeInsertCase(preparation: TablePreparation[NestedTable.type]): TestCase = preparation.test("nested.mergeInsert") { table => table.spark.sql( s"""MERGE INTO ${table.name} target USING ( @@ -160,7 +160,7 @@ trait ScenarioNestedType extends ScenarioKit { } /** DELETE WHERE s.x = 2 filtering on a nested struct field removes only the matching row, leaving ids 1 and 3. */ - private def deleteByNestedFieldCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + private def deleteByNestedFieldCase(preparation: TablePreparation[NestedTable.type]): TestCase = preparation .test("nested.deleteByNestedField") { table => table.spark.sql( @@ -182,7 +182,7 @@ trait ScenarioNestedType extends ScenarioKit { * Inserting a row with NULL struct, empty array and empty map reads back a null struct and an empty array for that * row. */ - private def nullValuesCase(preparation: TablePreparation[NestedTable.type]): Plan.Case = + private def nullValuesCase(preparation: TablePreparation[NestedTable.type]): TestCase = preparation.test("nested.nullValues") { table => table.spark.sql( s"INSERT INTO ${table.name} VALUES (" + @@ -204,7 +204,7 @@ trait ScenarioNestedType extends ScenarioKit { * On a side table, ADD COLUMN of a new nested struct field null-fills it for the existing row and accepts a new row * that sets the field. */ - private def addStructFieldCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def addStructFieldCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("nested.addStructField") { table => val sideTable = s"${table.name}_nst" withOwnedTable(table.spark.sql(_), sideTable)( @@ -234,7 +234,7 @@ trait ScenarioNestedType extends ScenarioKit { * On a side table, ALTER TABLE DROP COLUMN of a nested struct field is rejected with an exception, and the field * remains readable afterward. */ - private def dropStructFieldRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def dropStructFieldRejectedCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("nested.dropStructField.rejected") { table => val sideTable = s"${table.name}_nsd" withOwnedTable(table.spark.sql(_), sideTable)( diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionEvolution.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionEvolution.scala index 826de7d3f..bbb8cfa00 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionEvolution.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionEvolution.scala @@ -6,7 +6,7 @@ package harness * Operations: ALTER TABLE ADD PARTITION FIELD on an unpartitioned table and ALTER TABLE DROP PARTITION FIELD on a * date-partitioned table. The catalog rejects both, so recreating the table is the way to change its partitioning. * - * Preparation axes: in each of the two columnar formats, the standard seeded core table for the add case and a + * Preparation axes: in each columnar format, the standard seeded core table for the add case and a * date-partitioned core table seeded with the standard rows for the drop case. * * Case families: two families contributing 4 cases. @@ -14,8 +14,8 @@ package harness trait ScenarioPartitionEvolution extends ScenarioKit { /** The rejected partition-evolution statements, one file format at a time. */ - lazy val partitionEvolutionCases: List[Plan.Case] = - standardFormats.flatMap { format => + lazy val partitionEvolutionCases: List[TestCase] = + fileFormats.flatMap { format => List( addPartitionFieldRejectedCase(format), dropPartitionFieldRejectedCase(format)) @@ -27,7 +27,7 @@ trait ScenarioPartitionEvolution extends ScenarioKit { * ALTER TABLE ADD PARTITION FIELD on an unpartitioned table is rejected with an exception stating that evolution of * table partitioning is unsupported. */ - private def addPartitionFieldRejectedCase(format: String): Plan.Case = + private def addPartitionFieldRejectedCase(format: String): TestCase = preparedStandardTable(format).test("partitionEvolution.add.rejected") { table => val exception = Check.intercept[Exception]( table.spark.sql( @@ -40,7 +40,7 @@ trait ScenarioPartitionEvolution extends ScenarioKit { * ALTER TABLE DROP PARTITION FIELD on a date-partitioned table is rejected with an exception stating that evolution * of table partitioning is unsupported. */ - private def dropPartitionFieldRejectedCase(format: String): Plan.Case = + private def dropPartitionFieldRejectedCase(format: String): TestCase = TablePreparation( format, TableTest(Core) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionTransform.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionTransform.scala deleted file mode 100644 index 3c925c944..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionTransform.scala +++ /dev/null @@ -1,155 +0,0 @@ -package harness - -import org.apache.spark.sql.AnalysisException -import org.apache.spark.sql.types.StructType - -/** - * Partition transforms: which PARTITIONED BY transforms the catalog accepts at table creation, the partition field - * each accepted transform produces, and the partition specifications it rejects. - * - * Operations: CREATE TABLE PARTITIONED BY each of identity, bucket, truncate, years, months, days and hours followed - * by a read of the partitions metadata table; CREATE TABLE PARTITIONED BY the rejected void transform, the rejected - * days transform over a date column, and a column the table does not declare. - * - * Preparation axes: for the accepted and rejected transforms, a TypesTable in each of the two columnar formats seeded - * with three rows whose timestamps fall in three distinct hours, days, months and years; for the rejected partition - * column, the standard seeded core table in the same two formats. - * - * Case families: ten families contributing 20 cases, 14 accepted transforms and 6 rejections. - */ -trait ScenarioPartitionTransform extends ScenarioKit { - - /** Every partition-transform case, one file format at a time. */ - lazy val partitionTransformCases: List[Plan.Case] = - standardFormats.flatMap { format => - acceptedTransforms.map { - case (caseName, transform, partitionField, expectedPartitionCount) => - acceptedTransformCase(format, caseName, transform, partitionField, expectedPartitionCount) - } ++ - rejectedTransforms.map { - case (caseName, transform, expectedMessage) => - rejectedTransformCase(format, caseName, transform, expectedMessage) - } - } ++ preparedCoreFormats.map(partitionByNonExistentColumnCase) - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - // A fully valued TypesTable row whose date and timestamp columns both come from `timestamp`, so one row lands in one - // partition of every time-based transform. - private def partitionRow(id: Long, str: String, timestamp: String): String = - s"(CAST($id AS BIGINT), ${id.toInt}, ${id}.5, " + - s"CAST(${id}.50 AS decimal(10,2)), '$str', CAST('bin-$id' AS binary), " + - s"DATE '${timestamp.take(10)}', TIMESTAMP '$timestamp', TIMESTAMP_NTZ '$timestamp')" - - /** - * One accepted partition transform: a table PARTITIONED BY that transform reports a single partition field with the - * expected name in its partitions metadata table, and the three seeded rows land in the expected number of distinct - * partitions. The transform, its partition field name, and that partition count are the parameters. - */ - private def acceptedTransformCase( - format: String, - caseName: String, - transform: String, - partitionField: String, - expectedPartitionCount: Int): Plan.Case = - TablePreparation( - format, - TableTest(TypesTable) - .sql("create")(table => - s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + - s"USING $dataSource PARTITIONED BY ($transform) " + - s"TBLPROPERTIES ('write.format.default'='$format')")() - .sql("insertPartitionRows")(table => - s"INSERT INTO $table VALUES " + - partitionRow(1, "aa-1", "2023-12-31 23:00:00") + ", " + - partitionRow(2, "bb-2", "2024-01-01 00:00:00") + ", " + - partitionRow(3, "cc-3", "2024-02-01 01:00:00"))(view => - assert( - view.after.size == view.before.size + 3, - s"expected three partition test rows, got ${view.after.size}"))) - .test(caseName) { table => - val partitionTable = table.spark.table(s"${table.name}.partitions") - val partitionFields = partitionTable.schema("partition").dataType - .asInstanceOf[StructType] - .fieldNames - .toSeq - - assert( - partitionFields == Seq(partitionField), - s"expected partition field $partitionField, got ${partitionFields.mkString(", ")}") - assert( - partitionTable.count() == expectedPartitionCount, - s"expected $expectedPartitionCount partitions for $transform") - } - - /** - * One rejected partition transform: CREATE TABLE PARTITIONED BY that transform fails with a RuntimeException - * carrying the expected message, and the scratch table it would have created is gone. The transform and the expected - * message are the parameters. - */ - private def rejectedTransformCase( - format: String, - caseName: String, - transform: String, - expectedMessage: String): Plan.Case = - TablePreparation( - format, - TableTest(TypesTable) - .sql("create")(table => - s"CREATE TABLE $table (${TypesTable.columnDefinitions}) " + - s"USING $dataSource " + - s"TBLPROPERTIES ('write.format.default'='$format')")()) - .test(caseName) { table => - val scratchTable = table.name + "_x" - - withCleanupStatement(table.spark.sql(_), s"DROP TABLE IF EXISTS $scratchTable") { - val exception = Check.intercept[RuntimeException]( - table.spark.sql( - s"CREATE TABLE $scratchTable " + - s"(${TypesTable.columnDefinitions}) " + - s"USING $dataSource PARTITIONED BY ($transform) " + - s"TBLPROPERTIES ('write.format.default'='$format')")) - - assert(exception.getMessage.contains(expectedMessage)) - } - } - - /** - * CREATE TABLE PARTITIONED BY a column the table does not declare is rejected with an AnalysisException naming that - * column, and the scratch table it would have created is gone. - */ - private def partitionByNonExistentColumnCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("partition.byNonExistentColumn.rejected") { table => - val scratchTable = table.name + "_x" - - withCleanupStatement(table.spark.sql(_), s"DROP TABLE IF EXISTS $scratchTable") { - val exception = Check.intercept[AnalysisException]( - table.spark.sql( - s"CREATE TABLE $scratchTable ($columnDefinitions) " + - s"USING $dataSource PARTITIONED BY (no_such_column) " + - s"TBLPROPERTIES ('write.format.default'='${preparation.label}')")) - - assert(exception.getMessage.contains("no_such_column")) - } - } - - // The accepted transforms: the case name, the PARTITIONED BY clause, the partition field the catalog derives, and - // the number of distinct partitions the three seeded rows land in. - private val acceptedTransforms: List[(String, String, String, Int)] = - List( - ("partition.identity", "id", "id", 3), - ("partition.bucket", "bucket(4, id)", "id_bucket", 2), - ("partition.truncate", "truncate(2, str)", "str_trunc", 3), - ("partition.years", "years(ts)", "ts_year", 2), - ("partition.months", "months(ts)", "ts_month", 3), - ("partition.days", "days(ts)", "ts_day", 3), - ("partition.hours", "hours(ts)", "ts_hour", 3)) - - // The rejected transforms: the case name, the PARTITIONED BY clause, and the message the rejection carries. - private val rejectedTransforms: List[(String, String, String)] = - List( - ("partition.void.rejected", "void(n)", "not supported"), - ("partition.dateDay.rejected", "days(dt)", "Unsupported column")) - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioProcedure.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioProcedure.scala deleted file mode 100644 index e81d52eac..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioProcedure.scala +++ /dev/null @@ -1,113 +0,0 @@ -package harness - -/** - * Catalog procedures and catalog-level statements: which of them this catalog implements, and which it rejects. - * - * Operations: ancestors_of over a two-snapshot history; register_table onto a new name from an existing metadata file, - * followed by the rejected system.snapshot and system.add_files import procedures; and the rejected CREATE VIEW and - * ANALYZE TABLE COMPUTE STATISTICS statements. - * - * Preparation axes: in each of the two columnar formats, the two-snapshot core table for the ancestry family and the - * standard seeded core table for the import and statement families. - * - * Case families: three families contributing 6 cases. - */ -trait ScenarioProcedure extends ScenarioKit { - - /** Every catalog-procedure case, one file format at a time. */ - lazy val procedureCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - ancestorsOfCase(preparedTwoSnapshotTable(format)), - registerTableCase(preparedStandardTable(format)), - viewAndAnalyzeRejectedCase(preparedStandardTable(format))) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** ancestors_of lists both snapshots of the table's two-snapshot history. */ - private def ancestorsOfCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("procedure.ancestorsOf") { table => - val ancestorCount = table.spark - .sql( - "CALL openhouse.system.ancestors_of(" + - s"table => '${catalogRelative(table.name)}')") - .collect() - .length - - assert( - ancestorCount == 2, - s"ancestors_of should list two snapshots, got $ancestorCount") - } - - /** - * register_table onto a new name makes the source table's snapshot readable there (3 rows) and leaves the source - * unchanged, and dropping the registered table leaves the source unchanged. The system.snapshot and system.add_files - * procedures each reject their unsupported inputs with an exception. - * - * The drop of the registered table is both its cleanup and the operation the source assertion after the ownership - * boundary depends on, so it runs as that boundary's cleanup. A drop the catalog refuses fails the case, so the case - * cannot pass while leaving the registration behind. The snapshot target extends the prepared table's generated - * name, and its own boundary removes it whether the procedure was rejected as expected, threw something else, or - * unexpectedly succeeded. - */ - private def registerTableCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("procedure.registerTable") { table => - val registeredTable = s"${table.name}_registered" - val snapshotTarget = s"${table.name}_snapshotTarget" - val absentSourceDirectory = s"/tmp/${table.name.split('.').last}_absentSource" - val metadataFile = table.spark - .sql( - s"SELECT file FROM ${table.name}.metadata_log_entries " + - "ORDER BY timestamp DESC LIMIT 1") - .collect()(0) - .getString(0) - - withOwnedTable(table.spark.sql(_), registeredTable)( - table.spark.sql( - "CALL openhouse.system.register_table(" + - s"table => '${catalogRelative(registeredTable)}', " + - s"metadata_file => '$metadataFile')")) { - assert( - countOf(table.spark, s"SELECT count(*) FROM $registeredTable") == "3", - "register_table should make all source rows readable") - } - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", - "dropping the registered table should leave the source rows in place") - - withCleanupStatement(table.spark.sql(_), s"DROP TABLE IF EXISTS $snapshotTarget") { - Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.snapshot(" + - s"source_table => '${catalogRelative(table.name)}', " + - s"table => '${catalogRelative(snapshotTarget)}')")) - } - - Check.intercept[Exception]( - table.spark.sql( - "CALL openhouse.system.add_files(" + - s"table => '${catalogRelative(table.name)}', " + - s"source_table => '`parquet`.`$absentSourceDirectory`')")) - } - - /** - * CREATE VIEW and ANALYZE TABLE COMPUTE STATISTICS are each rejected with an exception. The view name extends the - * prepared table's generated name, and its boundary removes the view whether the statement was rejected as expected, - * threw something else, or unexpectedly succeeded. - */ - private def viewAndAnalyzeRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("procedure.viewAndAnalyze.rejected") { table => - val viewName = s"${table.name}_view" - - withCleanupStatement(table.spark.sql(_), s"DROP VIEW IF EXISTS $viewName") { - Check.intercept[Exception]( - table.spark.sql(s"CREATE VIEW $viewName AS SELECT 1 AS one")) - } - - Check.intercept[Exception]( - table.spark.sql( - s"ANALYZE TABLE ${table.name} COMPUTE STATISTICS")) - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRename.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRename.scala deleted file mode 100644 index 87158fa86..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRename.scala +++ /dev/null @@ -1,68 +0,0 @@ -package harness - -import com.linkedin.openhouse.javaclient.exception.WebClientResponseWithMessageException - -/** - * Table rename: ALTER TABLE RENAME TO moves a table to a new name with its rows, and the catalog refuses a rename onto - * a name that is already taken. - * - * Operations: RENAME TO a free name followed by a read of both the new and the old name, then a rename back; and - * RENAME TO the name of a table that already exists. - * - * Preparation axes: the standard seeded core table in each of the two columnar formats. The conflict family creates - * and drops the table it collides with. - * - * Case families: two families contributing 4 cases. - */ -trait ScenarioRename extends ScenarioKit { - - /** Every rename case, one file format at a time. */ - lazy val renameCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - renameTableCase(preparedStandardTable(format)), - renameTableConflictCase(preparedStandardTable(format), format)) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** - * ALTER TABLE RENAME TO moves the table to the new name with its 3 rows intact, and the old name stops resolving. A - * second rename puts the table back under its original name, which teardown drops. The rename boundary records the - * live name after each accepted rename, so a failure between the two renames drops the table under the name it - * currently answers to. - */ - private def renameTableCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("rename.table") { table => - val renamedTable = s"${table.name}_ren" - - withTrackedRename(table.spark.sql(_), table.name) { renameTo => - renameTo(renamedTable) - assert( - countOf(table.spark, s"SELECT count(*) FROM $renamedTable") == "3", - "the renamed table should keep its rows") - Check.intercept[Exception]( - table.spark.sql(s"SELECT 1 FROM ${table.name} LIMIT 1")) - renameTo(table.name) - } - } - - /** ALTER TABLE RENAME TO a name that already exists is rejected with an error naming the conflict. */ - private def renameTableConflictCase( - preparation: TablePreparation[CoreTable.type], - format: String): Plan.Case = - preparation.test("rename.table.conflict") { table => - val conflictingTable = s"${table.name}_other" - - withOwnedTable(table.spark.sql(_), conflictingTable)( - table.spark.sql(coreCreate(conflictingTable, format))) { - val exception = Check.intercept[WebClientResponseWithMessageException]( - table.spark.sql(s"ALTER TABLE ${table.name} RENAME TO $conflictingTable")) - - assert( - exception.getMessage.contains("already exists"), - s"unexpected message: ${exception.getMessage.take(160)}") - } - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioScanPlanning.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioScanPlanning.scala deleted file mode 100644 index 582bd8659..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioScanPlanning.scala +++ /dev/null @@ -1,115 +0,0 @@ -package harness - -import org.apache.iceberg.TableProperties -import org.apache.iceberg.spark.{Spark3Util, SparkSQLProperties} -import scala.collection.JavaConverters._ - -/** - * Scan planning: the split size decides how the read path combines data files into read tasks, and every split size - * returns the same rows. - * - * Operations: read a six-file table under a large and a tiny spark.sql.iceberg.split-size, comparing the row set and - * the read RDD partition count; then plan the same table directly through the Iceberg scan API under a split size - * above the whole table and one below a single file, comparing the task-group counts. - * - * Preparation axes: one table per file format, built inside the case with write.distribution-mode=none and - * read.split.open-file-cost=1 and filled by six separate inserts, so it holds six separately weighted data files. - * - * Case families: one family contributing 2 cases. - */ -trait ScenarioScanPlanning extends ScenarioKit { - - /** The split-size case, one file format at a time. */ - lazy val scanPlanningCases: List[Plan.Case] = - standardFormats.map(format => - Plan.Case(s"scanPlanning.splitSize @ $format", splitSizeCase(format))) - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** - * Over several small files, a large split size combines them into fewer read tasks and a tiny split size splits them - * into more, visible through rdd.getNumPartitions, and both reads return the same rows. The planner shows the same - * effect directly: a split size above the whole table plans one task group, and a split size below one file plans - * one group per file. - */ - private def splitSizeCase(format: String)(ctx: Ctx): Unit = { - val spark = ctx.spark - val table = TableTest.nextQualifiedTableName(ctx.namespace) - - // distribution=none plus several separate inserts produces several distinct data files. An open-file-cost of 1 - // sets each file's planning weight to its byte length, making split-size the knob that governs task-group count. - withOwnedTable(spark.sql(_), table)( - spark.sql( - s"CREATE TABLE $table (id bigint, s string) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'write.distribution-mode'='none', " + - "'read.split.open-file-cost'='1')")) { - val numberOfFiles = 6 - (0 until numberOfFiles).foreach { fileIndex => - spark.sql(s"INSERT INTO $table SELECT ${fileIndex}L, repeat('r$fileIndex', 4000)") - } - val fileCount = spark.sql(s"SELECT count(*) FROM $table.data_files").collect()(0).getLong(0) - assert( - fileCount >= 2, - s"[$format] expected multiple data files for a split test, got $fileCount") - - val splitSizeKey = SparkSQLProperties.SPLIT_SIZE // "spark.sql.iceberg.split-size" - val savedSplitSize = spark.conf.getOption(splitSizeKey) - def keys(): Seq[Long] = - spark.sql(s"SELECT id FROM $table ORDER BY id").collect().toSeq.map(_.getLong(0)) - def readPartitionCount(): Int = spark.sql(s"SELECT * FROM $table").rdd.getNumPartitions - val expectedKeys = (0 until numberOfFiles).map(_.toLong) - try { - // The row set is invariant under the split size, while the read RDD partition count follows it. - spark.conf.set(splitSizeKey, (512L * 1024 * 1024).toString) - val keysUnderLargeSplit = keys() - val partitionsUnderLargeSplit = readPartitionCount() - spark.conf.set(splitSizeKey, "1") - val keysUnderTinySplit = keys() - val partitionsUnderTinySplit = readPartitionCount() - assert( - keysUnderLargeSplit == expectedKeys && keysUnderTinySplit == expectedKeys, - s"[$format] split-size must leave the row set alone: large=$keysUnderLargeSplit " + - s"tiny=$keysUnderTinySplit expected=$expectedKeys") - assert( - partitionsUnderTinySplit >= partitionsUnderLargeSplit, - s"[$format] a smaller split-size must keep or raise the read RDD partition count: " + - s"tiny=$partitionsUnderTinySplit large=$partitionsUnderLargeSplit") - - // The same knob checked directly at the planner: with open-file-cost=1 each file's planning weight is its - // byte length, so a split-size below one file combines nothing (one task group per file) while a split-size - // above the whole table combines everything into one group. - val icebergTable = Spark3Util.loadIcebergTable(spark, table) - val targetSizeKey = TableProperties.SPLIT_SIZE // "read.split.target-size" - def plannedTaskGroups(splitBytes: Long): Int = - icebergTable - .newScan() - .option(targetSizeKey, splitBytes.toString) - .planTasks() - .asScala - .size - val groupsUnderLargeSplit = plannedTaskGroups(512L * 1024 * 1024) - val groupsUnderTinySplit = plannedTaskGroups(1L) - assert( - groupsUnderLargeSplit == 1, - s"[$format] a split-size above the whole table should plan 1 task group, " + - s"got $groupsUnderLargeSplit") - assert( - groupsUnderTinySplit == fileCount, - s"[$format] a split-size below one file should plan one task group per file ($fileCount), " + - s"got $groupsUnderTinySplit") - - println( - s"DIAG scanPlanning.splitSize[$format]: key='$splitSizeKey' files=$fileCount " + - s"readPartitions(large=$partitionsUnderLargeSplit,tiny=$partitionsUnderTinySplit) " + - s"taskGroups(large=$groupsUnderLargeSplit,tiny=$groupsUnderTinySplit)") - } finally { - // The split size is session state, not a table, so the case restores whatever the session held before it. - savedSplitSize match { - case Some(value) => spark.conf.set(splitSizeKey, value) - case None => spark.conf.unset(splitSizeKey) - } - } - } - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSchemaEvolution.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSchemaEvolution.scala index 8165eb997..d37c9f447 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSchemaEvolution.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSchemaEvolution.scala @@ -12,17 +12,17 @@ import org.apache.iceberg.exceptions.BadRequestException * RENAME COLUMN, and the rejected forms DROP COLUMN, DROP COLUMN over written data, ALTER COLUMN TYPE to a narrower * type and ALTER COLUMN SET NOT NULL. * - * Preparation axes: the six unseeded core layouts for the created-schema family; the six seeded core layouts for the - * evolution families; the standard seeded table in Parquet and ORC for the rejection families and for the families - * that build their own side table. + * Preparation axes: the four unseeded core layouts for the created-schema family; the four seeded core layouts for + * the evolution families; the standard seeded table in Parquet and ORC for the rejection families and for the + * families that build their own side table. * - * Case families: 14 families contributing 56 cases, 6 created-schema, 36 evolution, and 14 rejection or side-table + * Case families: 14 families contributing 42 cases, 4 created-schema, 24 evolution, and 14 rejection or side-table * cases. */ trait ScenarioSchemaEvolution extends ScenarioKit { /** Every schema-evolution case: the created schema, then the accepted changes, then the boundaries. */ - lazy val schemaEvolutionCases: List[Plan.Case] = + lazy val schemaEvolutionCases: List[TestCase] = createdSchemaCases ++ schemaChangeCases ++ schemaBoundaryCases // --- the preparations, shared helpers and case bodies the surface above composes --- @@ -31,7 +31,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { * The created table's schema is exactly CoreTable's columns, in declaration order and with their declared types, and * the table holds no rows. */ - private def createdSchemaCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def createdSchemaCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.create") { table => val actual = table.spark .table(table.name) @@ -46,7 +46,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { } /** ADD COLUMN adds the column to the schema, the existing rows read null for it, and the row count is unchanged. */ - private def addColumnSingleCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def addColumnSingleCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.addColumn.single") { table => table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_int int") @@ -64,7 +64,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { } /** ADD COLUMNS with two columns in one statement adds both to the schema and leaves the row count unchanged. */ - private def addColumnMultipleCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def addColumnMultipleCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.addColumn.multiple") { table => table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMNS (added_a int, added_b string)") @@ -77,7 +77,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { } /** ADD COLUMN ... COMMENT stores the comment on the added column and the reader sees it. */ - private def addColumnCommentCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def addColumnCommentCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.addColumn.comment") { table => table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN added_c int COMMENT 'a note'") @@ -94,7 +94,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { } /** ADD COLUMN ... AFTER foo_col_long places the added column directly after that column in the schema. */ - private def addColumnPositionCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def addColumnPositionCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.addColumn.position") { table => table.spark.sql( s"ALTER TABLE ${table.name} ADD COLUMN added_after int AFTER ${Core.long0.columnName}") @@ -110,7 +110,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { * ALTER COLUMN foo_col_int TYPE bigint widens the column in the schema and the already-written values read back * unchanged. */ - private def alterColumnTypeWidenCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def alterColumnTypeWidenCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.alterColumn.typeWiden") { table => table.spark.sql( s"ALTER TABLE ${table.name} ALTER COLUMN ${Core.int0.columnName} TYPE bigint") @@ -135,7 +135,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { * RENAME COLUMN renames the column in the schema: the new name is present, the old name is gone, and the row count * is unchanged. */ - private def renameColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def renameColumnCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation .test("schema.renameColumn") { table => table.spark.sql(s"ALTER TABLE ${table.name} ADD COLUMN to_rename int") @@ -153,7 +153,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { "restores the old name.")) /** ALTER TABLE DROP COLUMN is rejected with a BadRequestException naming the column that would be dropped. */ - private def dropColumnRejectedCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def dropColumnRejectedCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.dropColumn.rejected") { table => val exception = Check.intercept[BadRequestException]( table.spark.sql( @@ -172,7 +172,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { * writable. */ private def dropColumnWithDataRejectedCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.dropColumn.withData.rejected") { table => table.spark.sql( s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") @@ -203,7 +203,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { * unsupported column change. */ private def alterColumnNarrowTypeRejectedCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.alterColumn.narrowType.rejected") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( @@ -219,7 +219,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { * nullable-to-non-nullable change. */ private def alterColumnSetNotNullRejectedCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.alterColumn.setNotNull.rejected") { table => val exception = Check.intercept[AnalysisException]( table.spark.sql( @@ -231,7 +231,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { } /** On a side table, dropping NOT NULL from a column allows a subsequent insert of a null value for that column. */ - private def alterColumnDropNotNullCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def alterColumnDropNotNullCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.alterColumn.dropNotNull") { table => val sideTable = s"${table.name}_nn" withOwnedTable(table.spark.sql(_), sideTable)( @@ -252,7 +252,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { * On a side table, widening a decimal column's precision preserves the original row and accepts a new row whose * value only fits the wider precision. */ - private def alterColumnDecimalWidenCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def alterColumnDecimalWidenCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.alterColumn.decimalWiden") { table => val sideTable = s"${table.name}_dec" withOwnedTable(table.spark.sql(_), sideTable)( @@ -274,7 +274,7 @@ trait ScenarioSchemaEvolution extends ScenarioKit { } /** ALTER TABLE ALTER COLUMN ... FIRST moves that column to the front of the schema while preserving all 3 rows. */ - private def alterColumnReorderFirstCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def alterColumnReorderFirstCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("schema.alterColumn.reorderFirst") { table => table.spark.sql( s"ALTER TABLE ${table.name} " + @@ -293,11 +293,11 @@ trait ScenarioSchemaEvolution extends ScenarioKit { } /** The created-schema case on every unseeded core layout. */ - private val createdSchemaCases: List[Plan.Case] = + private val createdSchemaCases: List[TestCase] = preparedEmptyCoreTables.map(createdSchemaCase) /** The accepted schema changes on every seeded core layout. */ - private val schemaChangeCases: List[Plan.Case] = + private val schemaChangeCases: List[TestCase] = preparedCoreTables.flatMap { preparation => List( addColumnSingleCase(preparation), @@ -308,8 +308,8 @@ trait ScenarioSchemaEvolution extends ScenarioKit { renameColumnCase(preparation)) } - /** The rejected schema changes and the side-table schema changes, in each of the two columnar formats. */ - private val schemaBoundaryCases: List[Plan.Case] = + /** The rejected schema changes and the side-table schema changes, in each columnar format. */ + private val schemaBoundaryCases: List[TestCase] = preparedCoreFormats.flatMap { preparation => List( dropColumnRejectedCase(preparation), diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSnapshotRestore.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSnapshotRestore.scala deleted file mode 100644 index 0cf04f6e0..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSnapshotRestore.scala +++ /dev/null @@ -1,89 +0,0 @@ -package harness - -/** - * Snapshot restore: returning a table to an earlier snapshot, and what the restored table keeps. - * - * Operations: rollback_to_snapshot and set_current_snapshot back to the seed snapshot, and rollback_to_snapshot back - * to a pre-evolution snapshot after ADD COLUMN and an insert into the new column. - * - * Preparation axes: in each of the two columnar formats, the two-snapshot core table for the two restore procedures, - * and the standard seeded core table for the schema-evolution family. - * - * Case families: three families contributing 6 cases. - */ -trait ScenarioSnapshotRestore extends ScenarioKit { - - /** Every snapshot-restore case, one file format at a time. */ - lazy val snapshotRestoreCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - rollbackToSnapshotCase(preparedTwoSnapshotTable(format)), - setCurrentSnapshotCase(preparedTwoSnapshotTable(format)), - afterAddColumnCase(preparedStandardTable(format))) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** rollback_to_snapshot to the first snapshot restores the 3 rows the seed commit wrote. */ - private def rollbackToSnapshotCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("restore.rollbackToSnapshot") { table => - val firstSnapshotId = snapshotIds(table.spark, table.name).head - - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $firstSnapshotId)") - - assert(table.rows.size == 3) - } - - /** set_current_snapshot to the first snapshot restores the 3 rows the seed commit wrote. */ - private def setCurrentSnapshotCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("restore.setCurrentSnapshot") { table => - val firstSnapshotId = snapshotIds(table.spark, table.name).head - - table.spark.sql( - "CALL openhouse.system.set_current_snapshot(" + - s"'${catalogRelative(table.name)}', $firstSnapshotId)") - - assert(table.rows.size == 3) - } - - /** - * Rolling back to the pre-evolution snapshot after ADD COLUMN and an insert keeps the evolved schema, restores 3 - * rows that read null for the new column, and leaves the table accepting writes into that column. - */ - private def afterAddColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("restore.afterAddColumn") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $seedSnapshotId)") - val currentColumns = table.spark - .sql(s"SELECT * FROM ${table.name} LIMIT 1") - .columns - .toSeq - - assert( - currentColumns.contains("extra_col"), - s"rollback should retain the evolved schema: $currentColumns") - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", - "rollback should restore 3 rows") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} WHERE extra_col IS NOT NULL") == "0", - "rolled-back rows should read the evolved column as null") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert10") - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", - "the rolled-back table should accept evolved-schema writes") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSortOrder.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSortOrder.scala deleted file mode 100644 index a9ddc5c7e..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSortOrder.scala +++ /dev/null @@ -1,61 +0,0 @@ -package harness - -/** - * Sort order: ALTER TABLE WRITE ORDERED BY records a write sort order on the table, which the catalog pairs with range - * distribution, and the table keeps accepting writes under it. - * - * Operations: WRITE ORDERED BY a single column, and WRITE ORDERED BY two columns with an explicit direction and null - * ordering followed by an insert. - * - * Preparation axes: the standard seeded core table in each of the two columnar formats. - * - * Case families: two families contributing 4 cases. - */ -trait ScenarioSortOrder extends ScenarioKit { - - /** Every sort-order case, one file format at a time. */ - lazy val sortOrderCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - orderedByCase(preparedStandardTable(format)), - orderedByMultipleColumnsCase(preparedStandardTable(format))) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** ALTER TABLE WRITE ORDERED BY a single column sets write.distribution-mode to range. */ - private def orderedByCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("sortOrder.orderedBy") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} WRITE ORDERED BY ${Core.long0.columnName}") - - val distributionMode = - tableProps(table.spark, table.name).get("write.distribution-mode") - - assert( - distributionMode.contains("range"), - s"a write sort order should set range distribution, got $distributionMode") - } - - /** - * ALTER TABLE WRITE ORDERED BY multiple columns sets range distribution and the table remains writable, growing from - * 3 to 5 rows after a follow-up insert. - */ - private def orderedByMultipleColumnsCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("sortOrder.orderedByMultipleColumns") { table => - table.spark.sql( - s"ALTER TABLE ${table.name} WRITE ORDERED BY " + - s"${Core.string0.columnName} DESC NULLS FIRST, ${Core.long0.columnName}") - - assert( - tableProps(table.spark, table.name).get("write.distribution-mode").contains("range"), - "a multi-column write sort order should set range distribution") - - table.spark.sql( - s"INSERT INTO ${table.name} ${RowGenerator.valuesClause(Core, 2)}") - - assert(table.rows.size == 5, "the multi-column ordered write path should accept two rows") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioStreaming.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioStreaming.scala deleted file mode 100644 index 6c89f3f0a..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioStreaming.scala +++ /dev/null @@ -1,213 +0,0 @@ -package harness - -import java.nio.file.Files -import org.apache.spark.sql.SQLContext -import org.apache.spark.sql.execution.streaming.MemoryStream -import org.apache.spark.sql.streaming.Trigger - -/** - * Structured streaming: reading a table as a stream, writing a stream into a table, resuming a stream across a - * restart, and the snapshot histories a resumed stream rejects. - * - * Operations: a streaming read into a memory sink; a streaming append of two rows through the iceberg write-stream - * format; a streaming read into a destination table, restarted after an append; the same restart after a DELETE - * snapshot; and the same restart after the checkpoint's offset snapshot has been expired. - * - * Preparation axes: the standard seeded core table in each of the two columnar formats. The three restart families - * create and drop their own destination table in the same format. - * - * Case families: five families contributing 10 cases. - */ -trait ScenarioStreaming extends ScenarioKit { - - /** Every streaming case, one file format at a time. */ - lazy val streamingCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - readCase(preparedStandardTable(format)), - writeCase(preparedStandardTable(format)), - readAcrossRestartCase(preparedStandardTable(format), format), - deleteSnapshotRejectedCase(preparedStandardTable(format), format), - expiredCheckpointCase(preparedStandardTable(format), format)) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - // Runs one AvailableNow batch of a streaming read of `source` into `destination`, resuming from `checkpoint`. Each - // call returns after the batch has been committed, so the caller can assert on the destination and then run again. - private def streamOneBatch( - table: PreparedTable[CoreTable.type], - destination: String, - checkpoint: String): Unit = { - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("iceberg") - .outputMode("append") - .trigger(Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .toTable(destination) - assert(query.awaitTermination(120000), "stream did not finish") - query.stop() - } - - /** - * A Spark structured streaming read of the table, run in AvailableNow batch mode, delivers all 3 seed rows to a - * memory sink within 120 seconds. - */ - private def readCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("streaming.read") { table => - val checkpoint = Files.createTempDirectory("ck-read").toString - val sink = s"memsink_${System.nanoTime}" - val query = table.spark.readStream - .table(table.name) - .writeStream - .format("memory") - .queryName(sink) - .trigger(Trigger.AvailableNow()) - .option("checkpointLocation", checkpoint) - .start() - - assert( - query.awaitTermination(120000), - "streaming read did not finish in 120 seconds") - assert( - countOf(table.spark, s"SELECT count(*) FROM $sink") == "3", - "streaming read should deliver the three seed rows") - } - - /** - * A Spark structured streaming append of two rows through the iceberg write-stream format lands both rows, growing - * the table from 3 to 5 rows. - */ - private def writeCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("streaming.write") { table => - import table.spark.implicits._ - implicit val sqlContext: SQLContext = table.spark.sqlContext - val memoryStream = MemoryStream[Long] - memoryStream.addData(100L, 101L) - val rows = memoryStream.toDF().selectExpr( - s"value AS ${Core.long0.columnName}", - s"CAST(value AS INT) AS ${Core.int0.columnName}", - s"concat('row-', value) AS ${Core.string0.columnName}", - s"CAST(value AS DOUBLE) AS ${Core.double0.columnName}", - s"true AS ${Core.boolean0.columnName}", - s"'2024-01-01-00' AS ${Core.date0.columnName}") - val checkpoint = Files.createTempDirectory("ck-write").toString - val query = rows.writeStream - .format("iceberg") - .outputMode("append") - .option("checkpointLocation", checkpoint) - .toTable(table.name) - - query.processAllAvailable() - query.stop() - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "5", - "streaming write should append two rows") - } - - /** - * A streaming read of the table delivers the seed rows on first run and the newly inserted row after restart, into a - * destination table. - */ - private def readAcrossRestartCase( - preparation: TablePreparation[CoreTable.type], - format: String): Plan.Case = - preparation.test("streaming.readAcrossRestart") { table => - val destination = s"${table.name}_s" - val checkpoint = Files.createTempDirectory("ck-restart").toString - - withOwnedTable(table.spark.sql(_), destination)( - table.spark.sql(coreCreate(destination, format))) { - streamOneBatch(table, destination, checkpoint) - assert( - countOf(table.spark, s"SELECT count(*) FROM $destination") == "3", - "initial stream did not deliver the seed") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - streamOneBatch(table, destination, checkpoint) - assert( - countOf(table.spark, s"SELECT count(*) FROM $destination") == "4", - "stream restart did not deliver the appended row") - } - } - - /** - * An append-only stream restarted after a DELETE snapshot was written fails, with an error mentioning delete or - * overwrite. - */ - private def deleteSnapshotRejectedCase( - preparation: TablePreparation[CoreTable.type], - format: String): Plan.Case = - preparation.test("streaming.deleteSnapshot.rejected") { table => - val destination = s"${table.name}_sd" - val checkpoint = Files.createTempDirectory("ck-delete").toString - - withOwnedTable(table.spark.sql(_), destination)( - table.spark.sql(coreCreate(destination, format))) { - streamOneBatch(table, destination, checkpoint) - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") - val exception = - Check.intercept[Exception](streamOneBatch(table, destination, checkpoint)) - - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage).exists(message => - message.toLowerCase.contains("delete") || - message.toLowerCase.contains("overwrite"))), - "an append-only stream rejects a delete snapshot: " + - s"${exception.getClass.getSimpleName} ${Option(exception.getMessage).getOrElse("").take(140)}") - } - } - - /** - * A streaming read that resumes after its earliest offset snapshot has been expired fails, with an error naming the - * expired or missing snapshot. - */ - private def expiredCheckpointCase( - preparation: TablePreparation[CoreTable.type], - format: String): Plan.Case = - preparation.test("streaming.expiredCheckpoint") { table => - val destination = s"${table.name}_sink" - val checkpoint = Files.createTempDirectory("ck-expired").toString - - withOwnedTable(table.spark.sql(_), destination)( - table.spark.sql(coreCreate(destination, format))) { - streamOneBatch(table, destination, checkpoint) - assert( - countOf(table.spark, s"SELECT count(*) FROM $destination") == "3", - "initial stream should deliver the seed") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')") - streamOneBatch(table, destination, checkpoint) - assert( - countOf(table.spark, s"SELECT count(*) FROM $destination") == "4", - "control restart should deliver one incremental row") - - table.spark.sql( - s"INSERT INTO ${table.name} VALUES " + - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, true, '2024-01-07-06')") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - val exception = - Check.intercept[Exception](streamOneBatch(table, destination, checkpoint)) - - assert( - Exceptions.causeChain(exception).exists(error => - Option(error.getMessage).exists(message => - message.contains("expired or removed") || - message.contains("Cannot load current offset") || - message.contains("Cannot find snapshot"))), - "stream restart should report the expired checkpoint offset") - } - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableEvolutionCompatibility.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableEvolutionCompatibility.scala deleted file mode 100644 index fa750ad55..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableEvolutionCompatibility.scala +++ /dev/null @@ -1,170 +0,0 @@ -package harness - -/** - * One alteration a table can carry into the follow-up operations: the case-ID prefix its preparations contribute, the - * preparation step that applies it, and the ALTER TABLE statement that step runs. - */ -private[harness] final case class TableAlteration( - casePrefix: String, - stepLabel: String, - statement: String => String -) - -/** - * Table evolution compatibility: after a table has been altered, the reads, writes, snapshot operations and - * maintenance procedures that worked before the alteration still work. - * - * Operations: INSERT INTO after the alteration, row-level DELETE after the alteration, a VERSION AS OF read of the - * pre-alteration snapshot, rollback_to_snapshot back to that snapshot, expire_snapshots down to the newest snapshot, - * and rewrite_data_files over the files written across the alteration. - * - * Preparation axes: the four Parquet and ORC core layouts (each format crossed with unpartitioned and - * date-partitioned), each seeded with the standard rows and then altered in one of four ways: ADD COLUMN cc int, - * widening foo_col_int from int to bigint, WRITE ORDERED BY foo_col_long, or setting write.distribution-mode to - * range. That is 16 preparations. - * - * Case families: six families over 16 preparations, contributing 96 cases. - */ -trait ScenarioTableEvolutionCompatibility extends ScenarioKit { - - /** Every follow-up operation on every altered preparation, one preparation at a time. */ - lazy val tableEvolutionCompatibilityCases: List[Plan.Case] = - alteredTablePreparations.flatMap(preparation => - List( - insertCase(preparation), - deleteCase(preparation), - timeTravelCase(preparation), - rollbackCase(preparation), - expireSnapshotsCase(preparation), - rewriteDataFilesCase(preparation))) - - /** - * One preparation per Parquet and ORC layout and per alteration: the table is created, seeded with the standard - * rows, then altered. Each alteration carries its own step label and case-ID prefix, so a case ID names the - * alteration it ran after. Plan walks this list so every family lands on one preparation before the next - * preparation starts. - */ - lazy val alteredTablePreparations: List[TablePreparation[CoreTable.type]] = - parquetAndOrcLayouts.flatMap { layout => - alterations.map { alteration => - TablePreparation( - layout.label, - create(layout) - .insert(standardSeedRowCount)() - .sql(alteration.stepLabel)(alteration.statement)(), - alteration.casePrefix) - } - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** The four alterations the follow-up operations run after, in the order Plan walks them. */ - private val alterations: List[TableAlteration] = - List( - TableAlteration( - "afterAddColumn:", - "addColumn", - table => s"ALTER TABLE $table ADD COLUMN cc int"), - TableAlteration( - "afterTypeWiden:", - "widenIntColumnToBigint", - table => s"ALTER TABLE $table ALTER COLUMN ${Core.int0.columnName} TYPE bigint"), - TableAlteration( - "afterWriteOrder:", - "writeOrderedByLongKey", - table => s"ALTER TABLE $table WRITE ORDERED BY ${Core.long0.columnName}"), - TableAlteration( - "afterDistributionMode:", - "setRangeDistributionMode", - table => - s"ALTER TABLE $table SET TBLPROPERTIES ('write.distribution-mode'='range')")) - - /** A plain INSERT still lands on the table after the alteration, taking it to four rows. */ - private def insertCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("insert") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", - "table is not writable after the alteration") - } - - /** A row-level DELETE still lands on the table after the alteration, taking it to two rows. */ - private def deleteCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("delete") { table => - table.spark.sql( - s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") - - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "2", - "mutation failed after the alteration") - } - - /** The seed snapshot from before the alteration is still readable through VERSION AS OF and returns its 3 rows. */ - private def timeTravelCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("timeTravel") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF $seedSnapshotId") == "3", - "seed snapshot is not readable after the alteration") - } - - /** - * rollback_to_snapshot back to the seed snapshot undoes an INSERT made after the alteration and returns the table to - * its three seed rows. - */ - private def rollbackCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("rollback") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).head - - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.rollback_to_snapshot(" + - s"'${catalogRelative(table.name)}', $seedSnapshotId)") - - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", - "rollback across the alteration failed") - } - - /** expire_snapshots retaining only the newest snapshot leaves the table readable with its four current rows. */ - private def expireSnapshotsCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("expireSnapshots") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.expire_snapshots(" + - s"table => '${catalogRelative(table.name)}', " + - "older_than => TIMESTAMP '2999-01-01 00:00:00', " + - "retain_last => 1)") - - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", - "table is unreadable after snapshot expiration") - } - - /** rewrite_data_files compacts the files written across the alteration and preserves the four current rows. */ - private def rewriteDataFilesCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("rewriteDataFiles") { table => - table.spark.sql( - s"INSERT INTO ${table.name} SELECT * FROM ${table.name} " + - s"WHERE ${Core.long0.columnName} = 1") - table.spark.sql( - "CALL openhouse.system.rewrite_data_files(" + - s"table => '${catalogRelative(table.name)}', " + - "options => map('min-input-files', '2'))") - - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", - "compaction changed rows after the alteration") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableProperty.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableProperty.scala index 8943e6965..48af60b85 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableProperty.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableProperty.scala @@ -11,7 +11,7 @@ import org.apache.iceberg.exceptions.BadRequestException * write.metadata.previous-versions-max back from a table that requested 7; reading write.target-file-size-bytes back * from a table that requested 1048576; and SET TBLPROPERTIES on openhouse.tableType. * - * Preparation axes: in each of the two columnar formats, the standard seeded core table for the two families that + * Preparation axes: in each columnar format, the standard seeded core table for the two families that * change properties after creation, plus one purpose-built table per family that asserts a property requested at * creation. * @@ -20,8 +20,8 @@ import org.apache.iceberg.exceptions.BadRequestException trait ScenarioTableProperty extends ScenarioKit { /** Every table-property case, one file format at a time. */ - lazy val tablePropertyCases: List[Plan.Case] = - standardFormats.flatMap { format => + lazy val tablePropertyCases: List[TestCase] = + fileFormats.flatMap { format => List( userRoundTripCase(preparedStandardTable(format)), reservedPropertyRejectedCase(preparedStandardTable(format)), @@ -34,7 +34,7 @@ trait ScenarioTableProperty extends ScenarioKit { // --- the preparations, shared helpers and case bodies the surface above composes --- /** SET TBLPROPERTIES adds a user property that reads back, and UNSET TBLPROPERTIES removes it. */ - private def userRoundTripCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def userRoundTripCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("tableProperty.userRoundTrip") { table => table.spark.sql( s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('my_key'='my_val')") @@ -53,7 +53,7 @@ trait ScenarioTableProperty extends ScenarioKit { * restriction. */ private def reservedPropertyRejectedCase( - preparation: TablePreparation[CoreTable.type]): Plan.Case = + preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("tableProperty.reservedOpenhouse.rejected") { table => val exception = Check.intercept[BadRequestException]( table.spark.sql( @@ -69,7 +69,7 @@ trait ScenarioTableProperty extends ScenarioKit { * ALTER TABLE SET TBLPROPERTIES ('openhouse.tableType'='REPLICA_TABLE') is rejected with a BadRequestException, * since the table type is fixed at creation. */ - private def tableTypeImmutableCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = + private def tableTypeImmutableCase(preparation: TablePreparation[CoreTable.type]): TestCase = preparation.test("tableProperty.tableTypeImmutable") { table => val exception = Check.intercept[BadRequestException]( table.spark.sql( @@ -85,7 +85,7 @@ trait ScenarioTableProperty extends ScenarioKit { * Even though format-version=1 was requested at creation, the catalog stores the table at format-version=2 and the * table remains writable there. */ - private def formatVersionForcedCase(format: String): Plan.Case = + private def formatVersionForcedCase(format: String): TestCase = TablePreparation( format, TableTest(Core) @@ -105,7 +105,7 @@ trait ScenarioTableProperty extends ScenarioKit { } /** The write.metadata.previous-versions-max property requested at creation is honored and reads back as 7. */ - private def previousVersionsHonoredCase(format: String): Plan.Case = + private def previousVersionsHonoredCase(format: String): TestCase = TablePreparation( format, TableTest(Core).sql("create")(table => @@ -124,7 +124,7 @@ trait ScenarioTableProperty extends ScenarioKit { * The write.target-file-size-bytes=1048576 property requested at creation is retained and the table holds its 3 seed * rows. */ - private def targetFileSizeCase(format: String): Plan.Case = + private def targetFileSizeCase(format: String): TestCase = TablePreparation( format, TableTest(Core) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTimeTravel.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTimeTravel.scala deleted file mode 100644 index 03b880d18..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTimeTravel.scala +++ /dev/null @@ -1,98 +0,0 @@ -package harness - -/** - * Time travel: reading a table as it stood at an earlier snapshot, by snapshot ID, by commit timestamp, and after the - * schema has moved on. - * - * Operations: VERSION AS OF each snapshot ID, TIMESTAMP AS OF the first commit's timestamp, and a VERSION AS OF read - * of the pre-evolution snapshot after ADD COLUMN and an insert into the new column. - * - * Preparation axes: in each of the two columnar formats, the two-snapshot core table for the snapshot and timestamp - * families, and the standard seeded core table for the schema-evolution family. - * - * Case families: three families contributing 6 cases. - */ -trait ScenarioTimeTravel extends ScenarioKit { - - /** Every time-travel case, one file format at a time. */ - lazy val timeTravelCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - versionAsOfCase(preparedTwoSnapshotTable(format)), - timestampAsOfCase(preparedTwoSnapshotTable(format)), - afterAddColumnCase(preparedStandardTable(format))) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** - * VERSION AS OF the first snapshot ID reads the 3 rows the seed commit wrote, and VERSION AS OF the second reads all - * 5 rows. - */ - private def versionAsOfCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("timeTravel.versionAsOf") { table => - val snapshots = snapshotIds(table.spark, table.name) - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF ${snapshots(0)}") == "3") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF ${snapshots(1)}") == "5") - } - - /** TIMESTAMP AS OF the first commit's time reads the 3 rows that commit wrote. */ - private def timestampAsOfCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("timeTravel.timestampAsOf") { table => - val firstCommitTimestamp = table.spark - .sql( - s"SELECT CAST(committed_at AS STRING) FROM ${table.name}.snapshots " + - "ORDER BY committed_at LIMIT 1") - .collect()(0) - .getString(0) - - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} TIMESTAMP AS OF '$firstCommitTimestamp'") == "3") - } - - /** - * After ADD COLUMN and an insert into the new column, time travel to the pre-evolution snapshot reads the old schema - * with 3 rows, while a current read sees the new column. - */ - private def afterAddColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("timeTravel.afterAddColumn") { table => - val seedSnapshotId = snapshotIds(table.spark, table.name).last - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - table.spark.sql( - s"INSERT INTO ${table.name} VALUES $extraColInsert9") - val currentColumns = table.spark - .sql(s"SELECT * FROM ${table.name} LIMIT 1") - .columns - .toSeq - val historicalColumns = table.spark - .sql( - s"SELECT * FROM ${table.name} " + - s"VERSION AS OF $seedSnapshotId LIMIT 1") - .columns - .toSeq - - assert( - currentColumns.contains("extra_col"), - s"current read is missing the evolved column: $currentColumns") - assert( - !historicalColumns.contains("extra_col") && - historicalColumns.size == Core.tableColumns.size, - s"time travel should use the snapshot schema: $historicalColumns") - assert( - countOf( - table.spark, - s"SELECT count(*) FROM ${table.name} VERSION AS OF $seedSnapshotId") == "3", - "pre-evolution snapshot should contain 3 rows") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriteDistribution.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriteDistribution.scala deleted file mode 100644 index ed3a9a85c..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriteDistribution.scala +++ /dev/null @@ -1,161 +0,0 @@ -package harness - -/** - * Write distribution: the write.distribution-mode a table is configured with is retained, and it decides how a single - * append is laid out on disk without changing the rows the table holds. - * - * Operations: creating a table with an explicit write.distribution-mode of none and of hash and reading the property - * back; appending one multi-task DataFrame into a four-partition table under each mode and comparing the rows and the - * data-file counts the two modes produce. - * - * Preparation axes: for the two retained-property families, the standard three-row seed in each of the two columnar - * formats, unpartitioned for none and date-partitioned for hash. The layout family builds its own four-partition - * tables in each format, because it needs one table per mode inside a single case. - * - * Case families: three families contributing 6 cases. - */ -trait ScenarioWriteDistribution extends ScenarioKit { - - /** Every write-distribution case, one file format at a time. */ - lazy val writeDistributionCases: List[Plan.Case] = - standardFormats.flatMap { format => - List( - Plan.Case(s"writeDistribution.noneVersusHash @ $format", noneVersusHashCase(format)), - noneRetainedCase(format), - hashRetainedCase(format)) - } - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - // The two tables the layout case compares hold 400 rows spread over 4 table partitions, written from 8 input tasks - // that each hold rows for every partition. - private val distributionPartitionCount = 4 - private val distributionInputTaskCount = 8 - private val distributionRowCount = 400 - - /** - * The same multi-task append under an explicit write.distribution-mode of none and of hash keeps the mode each table - * was configured with and lands the same logical rows in both, while producing the physical layout each mode - * defines. Under none every input task writes every partition it holds, so one append produces up to (input tasks - * times partitions) data files. Under hash the writer shuffles rows so one task owns each partition, clustering the - * append to about one file per partition. - * - * The comparison needs both tables live at once, so the case nests one owned-table lifecycle inside the other. Each - * table carries a generated UUID and counter name, each lifecycle takes ownership the moment its CREATE returns, and - * each drops the one table it owns. A failure while building or appending to the hash table therefore still drops - * the none table, and the failure the case reports stays the primary one with any cleanup failure suppressed - * behind it. - */ - private def noneVersusHashCase(format: String)(ctx: Ctx): Unit = { - val spark = ctx.spark - - def createUnder(mode: String, table: String): Unit = - spark.sql( - s"CREATE TABLE $table (id bigint, p int) USING $dataSource PARTITIONED BY (p) " + - "TBLPROPERTIES ('format-version'='2', " + - s"'write.format.default'='$format', 'write.distribution-mode'='$mode')") - - def appendInputRows(table: String): Unit = - spark - .range(0, distributionRowCount.toLong) - .selectExpr("id", s"cast(id % $distributionPartitionCount as int) as p") - .repartition(distributionInputTaskCount) - .writeTo(table) - .append() - - def rowsOf(table: String): Seq[(Long, Int)] = - spark - .sql(s"SELECT id, p FROM $table ORDER BY id") - .collect() - .toSeq - .map(row => (row.getLong(0), row.getInt(1))) - - def dataFileCountOf(table: String): Long = - spark.sql(s"SELECT count(*) FROM $table.data_files").collect()(0).getLong(0) - - val noneTable = TableTest.nextQualifiedTableName(ctx.namespace) - val hashTable = TableTest.nextQualifiedTableName(ctx.namespace) - - withOwnedTable(spark.sql(_), noneTable)(createUnder("none", noneTable)) { - appendInputRows(noneTable) - - withOwnedTable(spark.sql(_), hashTable)(createUnder("hash", hashTable)) { - appendInputRows(hashTable) - - assert( - tableProps(spark, noneTable).get("write.distribution-mode").contains("none"), - s"[$format] the none table should retain write.distribution-mode=none") - assert( - tableProps(spark, hashTable).get("write.distribution-mode").contains("hash"), - s"[$format] the hash table should retain write.distribution-mode=hash") - - val noneRows = rowsOf(noneTable) - assert( - noneRows.size == distributionRowCount, - s"[$format] the none table should hold $distributionRowCount rows, got ${noneRows.size}") - assert( - noneRows == rowsOf(hashTable), - s"[$format] the two distribution modes should land the same logical rows") - - val noneFileCount = dataFileCountOf(noneTable) - val hashFileCount = dataFileCountOf(hashTable) - println( - s"DIAG writeDistribution.noneVersusHash[$format]: noneFiles=$noneFileCount " + - s"hashFiles=$hashFileCount partitions=$distributionPartitionCount " + - s"inputTasks=$distributionInputTaskCount") - assert( - hashFileCount <= distributionPartitionCount * 2, - s"[$format] hash should cluster to about $distributionPartitionCount files, " + - s"got $hashFileCount") - assert( - noneFileCount > hashFileCount && - noneFileCount <= distributionPartitionCount.toLong * distributionInputTaskCount, - s"[$format] none should spread the append across more files than hash and at most " + - s"${distributionPartitionCount * distributionInputTaskCount} " + - s"(none=$noneFileCount hash=$hashFileCount)") - } - } - } - - /** The write.distribution-mode=none requested at creation is retained and the table holds its 3 seed rows. */ - private def noneRetainedCase(format: String): Plan.Case = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + - s"'write.format.default'='$format', 'write.distribution-mode'='none')")() - .insert(standardSeedRowCount)()) - .test("writeDistribution.noneRetained") { table => - assert( - tableProps(table.spark, table.name).get("write.distribution-mode").contains("none"), - "distribution-mode none should be retained") - assert( - table.rows.size == standardSeedRowCount, - "the table should hold its seed rows under distribution-mode none") - } - - /** - * The write.distribution-mode=hash requested at creation on a date-partitioned table is retained and the table holds - * its 3 seed rows. - */ - private def hashRetainedCase(format: String): Plan.Case = - TablePreparation( - format, - TableTest(Core) - .sql("create")(table => - s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + - s"PARTITIONED BY (${Core.date0.columnName}) " + - "TBLPROPERTIES (" + - s"'write.format.default'='$format', 'write.distribution-mode'='hash')")() - .insert(standardSeedRowCount)()) - .test("writeDistribution.hashRetained") { table => - assert( - tableProps(table.spark, table.name).get("write.distribution-mode").contains("hash"), - "distribution-mode hash should be retained") - assert( - table.rows.size == standardSeedRowCount, - "the table should hold its seed rows under distribution-mode hash") - } - -} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriterCompatibility.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriterCompatibility.scala deleted file mode 100644 index a2d13ade0..000000000 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioWriterCompatibility.scala +++ /dev/null @@ -1,50 +0,0 @@ -package harness - -import org.apache.spark.sql.AnalysisException - -/** - * Writer compatibility: how a writer that names every column explicitly behaves after the table's column list has - * grown. - * - * Operations: an explicit-column INSERT that lists the six core columns, run once before ADD COLUMN and once after. - * The catalog accepts it before and rejects it after, naming the column the statement omits. - * - * Preparation axes: the standard seeded core table in each of the two columnar formats. - * - * Case families: one family contributing 2 cases. - */ -trait ScenarioWriterCompatibility extends ScenarioKit { - - /** The explicit-column writer case, one file format at a time. */ - lazy val writerCompatibilityCases: List[Plan.Case] = - standardFormats.map(format => afterAddColumnCase(preparedStandardTable(format))) - - // --- the preparations, shared helpers and case bodies the surface above composes --- - - /** - * An explicit-column INSERT that worked before ADD COLUMN is rejected afterward, with an error naming the new - * column. - */ - private def afterAddColumnCase(preparation: TablePreparation[CoreTable.type]): Plan.Case = - preparation.test("writerCompatibility.afterAddColumn") { table => - val writerStatement = - s"INSERT INTO ${table.name} ($columnNameList) VALUES " + - "(CAST(6 AS BIGINT), 6, 'row-6', 6.5, true, '2024-01-06-05')" - table.spark.sql(writerStatement) - assert( - countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", - "explicit-column writer should work before schema evolution") - - table.spark.sql( - s"ALTER TABLE ${table.name} ADD COLUMN extra_col INT") - val exception = Check.intercept[AnalysisException]( - table.spark.sql(writerStatement)) - assert( - exception.getMessage.contains("extra_col") && - (exception.getMessage.contains("CANNOT_FIND_DATA") || - exception.getMessage.toLowerCase.contains("cannot find data")), - "a pre-evolution explicit-column writer is rejected after ADD COLUMN: " + - exception.getMessage.take(160)) - } - -} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala index ec42160b1..9ff3ef5b1 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala @@ -1,69 +1,29 @@ package harness -import java.nio.charset.StandardCharsets -import java.security.MessageDigest - import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} import org.junit.jupiter.api.Test /** - * Pins the ordered catalog: its size, its fingerprint, the uniqueness of its IDs, the capability naming rule, and the - * rule that every capability contributes exactly once. Reading the catalog does not execute a case or start Spark. + * Pins the rules every integrated scenario set obeys: IDs are unique, the catalog is exactly + * the foundation plus the extensions it names, it is those contributions concatenated in order, contributions are + * named once and integrated alphabetically, every contribution supplies cases, and every case ID names its capability. + * + * This extension-stable test holds structural invariants while FoundationCatalogTest pins the exact set, size and + * fingerprint of this branch's frozen foundation. Each later layer pins its contributions in a focused test. Reading + * the catalog is a Spark-free operation. */ final class CaseCatalogTest { - private val expectedCaseCount = 1177 - private val expectedCatalogSha256 = - "a10676c9fe0af5169459a0c9ad74eb2d005b7c7c63e2b7c7d4364bb7d8cc5bb9" - - /** - * Every capability the standard catalog is built from, in the order Plan integrates them. This list is written out - * here rather than derived from `Plan.contributions`, so adding, dropping, renaming or reordering a capability fails - * this test until the intended catalog shape is restated. - */ - private val expectedContributionNames = List( - "accessControlCases", - "changelogCases", - "columnTagCases", - "compactionPlanningCases", - "concurrencyCases", - "dataTypeCases", - "dmlCases", - "dmlValidationCases", - "encryptionCases", - "fileFormatCases", - "fileReplicationCases", - "incrementalReadCases", - "lockingCases", - "maintenanceCases", - "metadataTableCases", - "namespaceCases", - "nestedTypeCases", - "partitionEvolutionCases", - "partitionTransformCases", - "procedureCases", - "renameCases", - "scanPlanningCases", - "schemaEvolutionCases", - "snapshotRestoreCases", - "sortOrderCases", - "streamingCases", - "tableEvolutionCompatibilityCases", - "tablePropertyCases", - "timeTravelCases", - "writeDistributionCases", - "writerCompatibilityCases") /** - * Case-ID prefixes that name where a case came from rather than the capability it covers. Every case ID is owned by - * the capability trait that defines it, so none of these appears in the catalog. + * Case-ID prefixes from the old provenance buckets. Every current case ID is owned by the capability trait that + * defines it. */ private val provenanceCaseIdPrefixes = List("fork.", "hazard.", "readerWriter.", "surface.", "interact.") @Test - def orderedCaseCatalogMatchesBaseline(): Unit = { - val caseIds = Plan.caseIds - val actualCatalogSha256 = sha256(caseIds.mkString("\n")) + def everyCaseIdIsUnique(): Unit = { + val caseIds = ScenarioCatalog.caseIds val duplicateCaseIds = caseIds.groupBy(identity).collect { case (caseId, occurrences) if occurrences.size > 1 => caseId }.toList.sorted @@ -71,20 +31,12 @@ final class CaseCatalogTest { assertTrue( duplicateCaseIds.isEmpty, s"case IDs must be unique; duplicates=${duplicateCaseIds.mkString(", ")}") - assertEquals( - expectedCaseCount, - caseIds.size, - s"ordered case catalog changed; count=${caseIds.size}, sha256=$actualCatalogSha256") - assertEquals( - expectedCatalogSha256, - actualCatalogSha256, - s"ordered case catalog changed; count=${caseIds.size}, sha256=$actualCatalogSha256") } @Test def everyCaseIdNamesTheCapabilityItCovers(): Unit = { val provenanceNamedCaseIds = - Plan.caseIds.filter(caseId => provenanceCaseIdPrefixes.exists(caseId.startsWith)) + ScenarioCatalog.caseIds.filter(caseId => provenanceCaseIdPrefixes.exists(caseId.startsWith)) assertTrue( provenanceNamedCaseIds.isEmpty, @@ -93,22 +45,8 @@ final class CaseCatalogTest { } @Test - def theCatalogIntegratesExactlyTheIntendedCapabilities(): Unit = { - val contributionNames = Plan.contributions.map { case (name, _) => name } - - assertEquals( - expectedContributionNames, - contributionNames, - "Plan integrates a different set or order of capabilities than the catalog declares") - assertEquals( - expectedContributionNames.distinct.size, - expectedContributionNames.size, - "the declared capability list names a capability more than once") - } - - @Test - def eachCapabilityContributesExactlyOnceInOrder(): Unit = { - val contributionNames = Plan.contributions.map { case (name, _) => name } + def eachCapabilityContributesExactlyOnceInAlphabeticalOrder(): Unit = { + val contributionNames = ScenarioCatalog.contributions.map { case (name, _) => name } assertEquals( contributionNames.distinct, @@ -118,19 +56,50 @@ final class CaseCatalogTest { contributionNames.sorted, contributionNames, s"capability contributions are integrated in alphabetical order: $contributionNames") - assertEquals( - Plan.contributions.flatMap { case (_, contribution) => contribution.map(_.id) }, - Plan.caseIds, - "the catalog is exactly its named contributions, concatenated in order") assertTrue( - Plan.contributions.forall { case (_, contribution) => contribution.nonEmpty }, + ScenarioCatalog.contributions.forall { case (_, contribution) => contribution.nonEmpty }, "every named contribution supplies at least one case") } - private def sha256(value: String): String = - MessageDigest - .getInstance("SHA-256") - .digest(value.getBytes(StandardCharsets.UTF_8)) - .map(byte => f"$byte%02x") - .mkString + @Test + def theCatalogIsExactlyTheFoundationAndTheExtensionsItNames(): Unit = { + val foundationNames = ScenarioCatalog.foundationContributions.map { case (name, _) => name } + val extensionNames = ScenarioCatalog.extensionContributions.map { case (name, _) => name } + val integratedNames = ScenarioCatalog.contributions.map { case (name, _) => name } + val declaredCases = (ScenarioCatalog.foundationContributions ++ + ScenarioCatalog.extensionContributions).toMap + + assertTrue( + foundationNames.intersect(extensionNames).isEmpty, + "an extension names a contribution the foundation already owns: " + + s"${foundationNames.intersect(extensionNames).mkString(", ")}") + assertEquals( + (foundationNames ++ extensionNames).sorted, + integratedNames.sorted, + "the catalog integrates a contribution that is neither a foundation nor an extension entry") + ScenarioCatalog.contributions.foreach { case (name, contribution) => + assertEquals( + declaredCases(name).map(_.id), + contribution.map(_.id), + s"$name is integrated as something other than the list the capability declares") + } + } + + @Test + def theCatalogIsItsContributionsConcatenatedInOrder(): Unit = { + val contributionOffsets = ScenarioCatalog.contributions + .scanLeft(0) { case (offset, (_, contribution)) => offset + contribution.size } + + assertEquals( + ScenarioCatalog.contributions.map { case (_, contribution) => contribution.size }.sum, + ScenarioCatalog.caseIds.size, + "the catalog holds exactly as many cases as its contributions supply") + ScenarioCatalog.contributions.zip(contributionOffsets).foreach { + case ((name, contribution), offset) => + assertEquals( + contribution.map(_.id), + ScenarioCatalog.caseIds.slice(offset, offset + contribution.size), + s"$name does not occupy the slice of the catalog its position claims") + } + } } diff --git a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala index 4c299bfa6..e7a189c98 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala @@ -177,30 +177,17 @@ final class DmlCaseCatalogTest { @Test def eachLayoutListCrossesItsFormatsWithItsPartitionings(): Unit = { - assertEquals(List("parquet", "orc", "avro"), Scenarios.fileFormats) - assertEquals(List("parquet", "orc"), Scenarios.standardFormats) - assertTrue( - Scenarios.standardFormats.forall(Scenarios.fileFormats.contains), - "the standard formats are drawn from the full file-format list") + assertEquals(List("parquet", "orc"), Scenarios.fileFormats) assertEquals( List( "unpartitioned/parquet", "partitioned/parquet", "unpartitioned/orc", - "partitioned/orc", - "unpartitioned/avro", - "partitioned/avro"), + "partitioned/orc"), Scenarios.layouts.map(_.label)) assertEquals( - List("partitioned/parquet", "partitioned/orc", "partitioned/avro"), + List("partitioned/parquet", "partitioned/orc"), Scenarios.partitionedLayouts.map(_.label)) - assertEquals( - List( - "unpartitioned/parquet", - "partitioned/parquet", - "unpartitioned/orc", - "partitioned/orc"), - Scenarios.parquetAndOrcLayouts.map(_.label)) assertEquals( Scenarios.fileFormats.map(format => s"nested-unpartitioned/$format"), Scenarios.nestedLayouts.map(_.label)) @@ -210,15 +197,13 @@ final class DmlCaseCatalogTest { } @Test - def everyPreparationLabelDrawsItsFormatFromTheStandardLists(): Unit = { - // A preparation label is either a layout path whose last segment is a file format, or one of the two labels for a - // case that owns no core table: `core` for an API-level case and `embedded` for a control-plane case. - val allowedLabelSuffixes = Scenarios.fileFormats ++ List("core", "embedded") - val unknownLabels = Plan.caseIds + def everyPreparationLabelDrawsItsFormatFromTheStandardList(): Unit = { + val unknownLabels = ScenarioCatalog.foundationContributions + .flatMap { case (_, contribution) => contribution.map(_.id) } .map(caseId => caseId.split(" @ ").last) .map(label => label.split("/").last) .distinct - .filterNot(allowedLabelSuffixes.contains) + .filterNot(Scenarios.fileFormats.contains) assertTrue( unknownLabels.isEmpty, diff --git a/integrations/spark/delta-harness/src/test/scala/harness/FoundationCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/FoundationCatalogTest.scala new file mode 100644 index 000000000..cc1bffec2 --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/FoundationCatalogTest.scala @@ -0,0 +1,122 @@ +package harness + +import java.nio.charset.StandardCharsets +import java.security.MessageDigest + +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} +import org.junit.jupiter.api.Test + +/** + * Pins the frozen foundation this branch owns: the eight capabilities in `ScenarioCatalog.foundationContributions`, + * and for each one the exact number of cases it contributes and the fingerprint of the IDs it contributes, in order. + * + * Every assertion reads `foundationContributions` alone, never the complete catalog, so a later layer that fills in + * `extensionContributions` leaves this file untouched and still passing. Each capability is pinned on its own line + * against its own fingerprint, so a change to one of them fails only that line and names it. Uniqueness, ordering and + * the contributions-concatenated rule are pinned once, for any catalog, in CaseCatalogTest. + * + * Reading the catalog is a Spark-free operation. + */ +final class FoundationCatalogTest { + private val expectedFoundationCaseCount = 642 + + /** + * Every capability the frozen foundation is built from, in the order it declares them, with the number of cases it + * contributes and the SHA-256 of its case IDs joined by newlines. This literal fixture makes every foundation + * addition, removal, rename, reorder or resize require an explicit restatement. + */ + private val expectedFoundationContributions = List( + ("dataTypeCases", 10, "e676820cc791e8bbde8921a38d77049f980c55ce78c3a19b7c30a0c5694065e6"), + ("dmlCases", 536, "0346a1b15adda474d19652c99a480228a61dedd376e4a5913b71a1c45e383e6f"), + ("dmlValidationCases", 12, "ce7a969bef2060ff8b64333509f7ab40489a6c8b76b54354884c6468dc33a5ae"), + ("fileFormatCases", 8, "97d22d197425a9156f8c1ef089d494d92f69a8ffcc09d487e5d8278070d98445"), + ("nestedTypeCases", 18, "f2fcdc951e4c17e2d146c26e529f7ef3639534ea22c67dde795b325a7025ea70"), + ("partitionEvolutionCases", 4, "bca457899d4108e353c31619603cec70888b7d3f4c35af67bbb27ebe9b1c4053"), + ("schemaEvolutionCases", 42, "9609fab5bda329357ec4b582fbf225a47678c754e0698a8482c4d9412ddcceb8"), + ("tablePropertyCases", 12, "b97d3f537b8d6740e00a6ae8fb8b88280de442fbc868342e5019fa0284434738")) + + private val expectedKnownBugCaseIds = List( + "nested.deleteByNestedField @ nested-unpartitioned/orc", + "nested.deleteByNestedField @ nested-unpartitioned/parquet", + "prep.ordered:delete.byPartitionPredicate @ partitioned/orc", + "prep.ordered:delete.byPartitionPredicate @ partitioned/parquet", + "prep.ordered:delete.byPartitionPredicate @ unpartitioned/orc", + "prep.ordered:delete.byPartitionPredicate @ unpartitioned/parquet", + "schema.renameColumn @ partitioned/orc", + "schema.renameColumn @ partitioned/parquet", + "schema.renameColumn @ unpartitioned/orc", + "schema.renameColumn @ unpartitioned/parquet") + + @Test + def eachFoundationCapabilityContributesTheCasesItIsPinnedTo(): Unit = { + val actualContributions = + ScenarioCatalog.foundationContributions.map { case (name, contribution) => + (name, contribution.size, sha256(contribution.map(_.id).mkString("\n"))) + } + + assertEquals( + expectedFoundationContributions.map { case (name, _, _) => name }, + actualContributions.map { case (name, _, _) => name }, + "the foundation declares a different set or order of capabilities than it is pinned to") + expectedFoundationContributions.zip(actualContributions).foreach { + case ((name, expectedCount, expectedSha256), (_, actualCount, actualSha256)) => + assertEquals( + (expectedCount, expectedSha256), + (actualCount, actualSha256), + s"$name changed; count=$actualCount, sha256=$actualSha256") + } + } + + @Test + def theFoundationIsTheSizeItIsPinnedTo(): Unit = { + val foundationCaseCount = + ScenarioCatalog.foundationContributions.map { case (_, contribution) => contribution.size }.sum + + assertEquals( + expectedFoundationCaseCount, + foundationCaseCount, + s"foundation case count changed; count=$foundationCaseCount") + assertEquals( + expectedFoundationCaseCount, + expectedFoundationContributions.map { case (_, count, _) => count }.sum, + "the pinned per-capability counts do not add up to the pinned foundation total") + } + + @Test + def everyFoundationCaseRunsOnAColumnarFormatTheFoundationStandardizedOn(): Unit = { + val preparationFormats = ScenarioCatalog.foundationContributions + .flatMap { case (_, contribution) => contribution.map(_.id) } + .map(caseId => caseId.split(" @ ").last) + .map(label => label.split("/").last) + .distinct + + assertEquals(List("parquet", "orc"), Scenarios.fileFormats) + assertEquals( + Scenarios.fileFormats.sorted, + preparationFormats.sorted, + s"a foundation case runs on a format outside the landing matrix: $preparationFormats") + } + + @Test + def theFoundationSkipMetadataIsPinnedToTheKnownProductBugs(): Unit = { + val foundationCases = + ScenarioCatalog.foundationContributions.flatMap { case (_, contribution) => contribution } + + assertEquals( + expectedKnownBugCaseIds, + foundationCases.collect { + case testCase if testCase.knownBugReason.nonEmpty => testCase.id + }.sorted, + "the foundation known-bug cases changed") + assertTrue( + foundationCases.forall(_.embeddedSkipReason.isEmpty), + "every foundation case reaches the embedded catalog") + } + + private def sha256(value: String): String = + MessageDigest + .getInstance("SHA-256") + .digest(value.getBytes(StandardCharsets.UTF_8)) + .map(byte => f"$byte%02x") + .mkString +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/PublicSurfaceCompatibilityTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/PublicSurfaceCompatibilityTest.scala new file mode 100644 index 000000000..333853568 --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/PublicSurfaceCompatibilityTest.scala @@ -0,0 +1,103 @@ +package harness + +import org.junit.jupiter.api.Assertions.{assertEquals, assertSame, assertTrue} +import org.junit.jupiter.api.Test + +/** + * Pins the entry points a consumer outside this module was written against. Splitting the mixin object, the ordered + * catalog and the case type into three names is an internal reorganisation, so a consumer that reads `Plan` and + * `Scenarios` must keep compiling and keep answering the same values. + * + * Every reference below is written the way an external consumer writes it, so this file fails to compile if any of + * those entry points is dropped, renamed or narrowed. The assertions prove the facade and catalog share one state. + * Reading the catalog is a Spark-free operation. + */ +final class PublicSurfaceCompatibilityTest { + + @Test + def planCaseStillNamesAndConstructsTheCaseType(): Unit = { + val constructed: Plan.Case = + Plan.Case("compat.probe", _ => (), knownBugReason = Some("probe")) + + assertEquals("compat.probe", constructed.id) + assertEquals(Some("probe"), constructed.knownBugReason) + assertEquals(None, constructed.embeddedSkipReason) + assertTrue( + (constructed: TestCase).isInstanceOf[TestCase], + "Plan.Case must be the harness case type, not a separate copy of it") + } + + @Test + def planCaseStillMatchesAsAnExtractor(): Unit = { + val matched = (Plan.Case("compat.probe", _ => ()): Plan.Case) match { + case Plan.Case(id, _, None, None) => id + case other => s"unmatched: $other" + } + + assertEquals("compat.probe", matched) + } + + @Test + def planBugReasonStillPhrasesAKnownBugTheWayItAlwaysDid(): Unit = { + val knownBug = Plan.Case("compat.bug", _ => (), knownBugReason = Some("the rewrite crashes")) + val healthy = Plan.Case("compat.healthy", _ => ()) + + assertEquals(Some("bug: the rewrite crashes"), Plan.bugReason(knownBug)) + assertEquals(None, Plan.bugReason(healthy)) + } + + @Test + def planForwardsToTheCatalogRatherThanHoldingItsOwnCopy(): Unit = { + assertEquals(ScenarioCatalog.caseIds, Plan.caseIds) + assertEquals(ScenarioCatalog.cases.map(_.id), Plan.cases.map(_.id)) + assertEquals(Plan.cases.map(_.id), Plan.caseIds) + } + + @Test + def scenariosStillExposesTheConfigurationAConsumerOverrides(): Unit = { + val originalDataSource = Scenarios.dataSource + try { + Scenarios.dataSource = "probe-source" + + assertEquals("probe-source", Scenarios.dataSource) + assertTrue( + Scenarios.layouts.head.create("db.t_probe").contains("USING probe-source"), + "a CREATE statement must follow the data source the consumer set") + } finally { + Scenarios.dataSource = originalDataSource + } + + assertEquals("iceberg", Scenarios.dataSource) + } + + @Test + def scenariosStillExposesTheCapabilityAndPreparationListsAConsumerReads(): Unit = { + assertEquals(List("parquet", "orc"), Scenarios.fileFormats) + assertEquals(4, Scenarios.layouts.size) + assertEquals(4, Scenarios.preparedCoreTables.size) + assertEquals(2, Scenarios.preparedCoreFormats.size) + assertEquals(536, Scenarios.dmlCases.size) + assertEquals(3, Scenarios.standardSeedRowCount) + assertSame( + Scenarios.dmlCases, + ScenarioCatalog.foundationContributions.toMap.apply("dmlCases"), + "the catalog must integrate the very list the capability exposes") + } + + @Test + def everyNamedContributionIsReadableFromTheScenariosObject(): Unit = { + val contributionsFromScenariosObject: List[(String, List[TestCase])] = List( + "dataTypeCases" -> Scenarios.dataTypeCases, + "dmlCases" -> Scenarios.dmlCases, + "dmlValidationCases" -> Scenarios.dmlValidationCases, + "fileFormatCases" -> Scenarios.fileFormatCases, + "nestedTypeCases" -> Scenarios.nestedTypeCases, + "partitionEvolutionCases" -> Scenarios.partitionEvolutionCases, + "schemaEvolutionCases" -> Scenarios.schemaEvolutionCases, + "tablePropertyCases" -> Scenarios.tablePropertyCases) + + assertEquals( + contributionsFromScenariosObject.map { case (name, cases) => (name, cases.map(_.id)) }, + ScenarioCatalog.foundationContributions.map { case (name, cases) => (name, cases.map(_.id)) }) + } +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/SupportContractTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/SupportContractTest.scala new file mode 100644 index 000000000..380f0b09b --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/SupportContractTest.scala @@ -0,0 +1,130 @@ +package harness + +import java.util.concurrent.atomic.AtomicInteger + +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} +import org.junit.jupiter.api.Test + +/** + * Pins the support this branch keeps for later feature layers: the changelog operations, the concurrency primitives, + * and the kit's generic starting-state substrate. These assertions exercise each contract before a feature layer + * integrates it. + * + * These Spark-free assertions inspect changelog data, plain JVM concurrency code, and preparation step lists. + */ +final class SupportContractTest { + private val support = new ChangelogSupport {} + + private val expectedChangelogOperations = List( + ("changelog.append", Map("INSERT" -> 1L)), + ("changelog.overwrite", Map("DELETE" -> 1L)), + ("changelog.delete", Map("DELETE" -> 1L)), + ("changelog.update", Map("DELETE" -> 1L, "INSERT" -> 1L)), + ("changelog.merge", Map("DELETE" -> 1L, "INSERT" -> 2L))) + + @Test + def theChangelogOperationsAreTheOnesALaterLayerCrossesWithItsOwnPreparations(): Unit = { + assertEquals( + expectedChangelogOperations, + support.changelogOperations.map(operation => + (operation.name, operation.expectedChangeCounts))) + } + + @Test + def everyChangelogOperationIsAStatementAgainstTheTableItIsGiven(): Unit = { + support.changelogOperations.foreach { operation => + val statement = operation.statement("db.t_probe") + + assertTrue( + statement.contains("db.t_probe"), + s"${operation.name} does not address the table it is given: $statement") + assertTrue( + operation.expectedChangeCounts.values.forall(_ > 0L), + s"${operation.name} expects a change type it does not produce") + } + } + + @Test + def changelogOperationsCrossWithEveryPreparationTheyAreGiven(): Unit = { + val preparations = support.preparedCoreFormats + val cases = support.changelogOperationCasesFor(preparations) + + assertEquals( + preparations.flatMap(preparation => + support.changelogOperations.map(operation => + s"${preparation.casePrefix}${operation.name} @ ${preparation.label}")), + cases.map(_.id)) + assertTrue( + ScenarioCatalog.foundationContributions + .flatMap { case (_, contribution) => contribution } + .forall(testCase => !testCase.id.startsWith("changelog.")), + "changelog support must not contribute cases to the foundation catalog") + } + + @Test + def runConcurrentlyReleasesEveryFunctionAndReportsNothingWhenTheyAllSucceed(): Unit = { + val completed = new AtomicInteger(0) + val threadErrors = + ConcurrencySupport.runConcurrently(Seq.fill(4)(() => completed.incrementAndGet())) + + assertTrue(threadErrors.isEmpty, s"a function failed unexpectedly: $threadErrors") + assertEquals(4, completed.get) + } + + @Test + def runConcurrentlyReportsTheThrowableEveryFailingFunctionRaised(): Unit = { + val threadErrors = ConcurrencySupport.runConcurrently( + Seq( + () => throw new IllegalStateException("first"), + () => (), + () => throw new IllegalStateException("second"))) + + assertEquals(List("first", "second"), threadErrors.map(_.getMessage).sorted.toList) + } + + @Test + def aTypedCommitConflictIsRecognisedAnywhereInTheCauseChain(): Unit = { + assertTrue( + ConcurrencySupport.isTypedCommitConflict( + new RuntimeException("outer", new CommitFailedProbe("inner"))), + "a commit-conflict class name anywhere in the chain is a typed conflict") + assertTrue( + !ConcurrencySupport.isTypedCommitConflict(new IllegalArgumentException("plain")), + "a failure whose chain names no commit, validation or transport class is untyped") + } + + @Test + def theGenericStartingStatesStayUsableForALaterLayer(): Unit = { + val kit = new KitProbe + + assertTrue( + kit.probeCoreCreate("db.t_probe", "orc").startsWith("CREATE TABLE db.t_probe ("), + "coreCreate must build a CREATE for the table and format it is given") + assertTrue( + kit.probeCoreCreate("db.t_probe", "orc").contains("'write.format.default'='orc'"), + "coreCreate must declare the format it is given") + assertEquals( + "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, false, '2024-01-01-00')", + kit.probeCoreRow(7L, "row-7")) + assertEquals(List("create"), kit.probeEmptyStandardTable("orc").preparation.steps.map(_.label).toList) + assertEquals( + List("create", "insert(3)", "waitForNextSnapshotTimestamp", "insertRowsFourAndFive"), + kit.probeTwoSnapshotTable("parquet").preparation.steps.map(_.label).toList) + } +} + +/** Carries the commit-conflict class-name marker that the harness recognises. */ +private final class CommitFailedProbe(message: String) extends Exception(message) + +/** + * Reads the kit's protected starting-state substrate the way a capability trait reads it, exercising the shared + * contract in the foundation suite. + */ +private final class KitProbe extends ScenarioKit { + def probeCoreCreate(table: String, format: String): String = coreCreate(table, format) + def probeCoreRow(long: Long, tag: String): String = coreRow(long, tag) + def probeEmptyStandardTable(format: String): TablePreparation[CoreTable.type] = + preparedEmptyStandardTable(format) + def probeTwoSnapshotTable(format: String): TablePreparation[CoreTable.type] = + preparedTwoSnapshotTable(format) +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala index f347eec3b..361c1bd51 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala @@ -72,6 +72,6 @@ final class TablePreparationTest { assertEquals(Some("the rewrite crashes on a write-ordered table"), testCase.knownBugReason) assertEquals( Some("bug: the rewrite crashes on a write-ordered table"), - Plan.bugReason(testCase)) + testCase.bugReason) } } From f5e45e2ef91638dbb5a8a7a786e8417e344e88c7 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Tue, 1 Sep 2026 20:40:09 -0700 Subject: [PATCH 15/24] feat(delta-harness): add RTAS matrix Add 264 replacement-specific cases on the 642-case foundation. - run every reusable DML operation across four RTAS preparations - cover schema, partition, policy, history, rename, order, and identity - require precise lineage rejection and typed concurrency outcomes - expose silent narrowing corruption as a known product bug Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/openhouse/ScenarioCatalog.scala | 9 +- .../harness/openhouse/ScenarioRtas.scala | 998 ++++++++++++++++++ .../test/scala/harness/RtasCatalogTest.scala | 222 ++++ 3 files changed, 1226 insertions(+), 3 deletions(-) create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRtas.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala index 9cb046b91..73ee4fe5a 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala @@ -26,6 +26,7 @@ object Scenarios with ScenarioPartitionEvolution with ScenarioSchemaEvolution with ScenarioTableProperty + with ScenarioRtas with ChangelogSupport /** @@ -61,11 +62,13 @@ object ScenarioCatalog { "tablePropertyCases" -> Scenarios.tablePropertyCases) /** - * The capabilities a later layer adds on top of the foundation, named the same way. This branch adds none, so the - * list is empty here and every entry below it in the file stays untouched as layers arrive. + * The capabilities this layer adds on top of the foundation, named the same way. This branch adds the replace-table + * capability, so a sibling layer appends its own entry here and leaves the foundation list and every entry below + * this one untouched. */ def extensionContributions: List[(String, List[TestCase])] = - List.empty + List( + "rtasCases" -> Scenarios.rtasCases) /** Every capability contribution, named once, in the order the catalog integrates them. */ def contributions: List[(String, List[TestCase])] = diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRtas.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRtas.scala new file mode 100644 index 000000000..0816ff25d --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRtas.scala @@ -0,0 +1,998 @@ +package harness + +import java.util.concurrent.ConcurrentHashMap + +import org.apache.iceberg.exceptions.{BadRequestException, ValidationException} +import org.apache.spark.sql.{AnalysisException, SparkSession} +import scala.util.control.NonFatal + +/** + * Replace table as select: what CREATE OR REPLACE TABLE AS SELECT is allowed to do to a table, and what survives it. + * + * A replace re-specifies a table in place and starts a new snapshot lineage under the same catalog identity. That + * makes it the one statement in the harness that can change a table's shape, its partitioning and its content at + * once, so this file owns two separate obligations. The first is that the reusable DML contract still holds on a + * table that reached its starting state through a replace. The second is that everything the catalog governs, and + * everything a reader can ask about history, behaves the way a new lineage requires. + * + * Operations, DML: every reusable operation `ScenarioDml` defines, reused as data. A replaced table runs the same + * statements and the same assertions as a freshly created one, so this file holds one definition of each operation. + * The 51 operations that + * fit any seeded table cross all four replace preparations, the null-string DELETE crosses the four replace + * preparations that carry a null row, and the two partition-scoped writes cross the two date-partitioned replace + * preparations. That is all 54 reusable operations on the preparations each one applies to. + * + * Operations, replace contract: 26 focused families covering the enablement gates, same-shape replacement and the + * write that follows it, the four schema discontinuities, partition replacement and the second replace that is the + * only legal way to repartition afterwards, property override and preservation, retention-policy and column-tag + * preservation, pre-replace time travel with rollback rejection and set-current-snapshot recovery, changelog and + * incremental-read rejection across the replacement boundary, replace crossed with rename in both orders, sort-order + * change and removal after a replace, creator-identity preservation, and a replace racing an append. + * + * Preparation axes: replace lineage is the axis this layer adds. Four replace preparations cross the two columnar + * formats with unpartitioned and date-partitioned tables; each creates a replace-enabled table, seeds the standard + * three rows, re-specifies the same shape through CREATE OR REPLACE TABLE AS SELECT, and refreshes, so the rows a + * case starts from arrived through the replace path. The contract families start from a plain replace-enabled table + * in each format, or from the property, retention or tag table each one needs, and drive the replace themselves so + * they can read the state on both sides of it. + * + * Case families: 264 cases. The DML axis contributes 212 in three families, and the replace contract contributes 52 + * in 26 families, each family running in both columnar formats. + */ +trait ScenarioRtas extends ScenarioKit { this: ScenarioDml with ChangelogSupport => + + /** Every replace case: the reusable DML operations on replaced tables first, then the replace contract. */ + lazy val rtasCases: List[TestCase] = rtasDmlCases ++ rtasContractCases + + /** + * The reusable DML operations on replaced tables, in preparation order: every operation on the four replace + * preparations, the null-string DELETE on their null-string form, then the partition-scoped writes on the two + * date-partitioned replace preparations. + */ + lazy val rtasDmlCases: List[TestCase] = + rtasCoreDmlCases ++ rtasNullStringDmlCases ++ rtasPartitionedDmlCases + + /** Every operation that fits any seeded table, on each of the four replace preparations. */ + lazy val rtasCoreDmlCases: List[TestCase] = + preparedRtasCoreTables.flatMap(preparation => allDmlTestCases.map(_.runOn(preparation))) + + /** The DELETE that selects a null string, on the replace preparations that carry a null row. */ + lazy val rtasNullStringDmlCases: List[TestCase] = + preparedNullStringRtasCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) + + /** The partition-scoped writes, on the two date-partitioned replace preparations. */ + lazy val rtasPartitionedDmlCases: List[TestCase] = + preparedRtasPartitionedCoreTables.flatMap(preparation => + partitionedTableTestCases.map(_.runOn(preparation))) + + /** Every replace-contract case, one file format at a time. */ + lazy val rtasContractCases: List[TestCase] = + fileFormats.flatMap { format => + List( + enablementGateCase(preparedStandardTable(format)), + disabledGateRejectedCase(preparedStandardTable(format)), + replicationGateRejectedCase(preparedStandardTable(format)), + sameShapeReplacementCase(preparedReplaceEnabledTable(format)), + writeAfterReplaceCase(preparedReplaceEnabledTable(format)), + schemaAddColumnCase(preparedReplaceEnabledTable(format)), + schemaDropColumnCase(preparedReplaceEnabledTable(format)), + schemaWidenColumnCase(preparedReplaceEnabledTable(format)), + schemaIncompatibleTypeRejectedCase(preparedReplaceEnabledTable(format)), + partitionSpecReplacedCase(preparedReplaceEnabledTable(format)), + partitionChangeAfterReplaceCase(preparedReplaceEnabledPartitionedTable(format)), + userPropertyPreservedCase(preparedUserPropertyTable(format)), + statementOverridesPropertyCase(preparedUserPropertyTable(format)), + retentionPolicyPreservedCase(preparedRetentionPolicyTable(format)), + columnTagPreservedCase(preparedTaggedTable(format)), + preReplaceTimeTravelCase(preparedReplaceEnabledTable(format)), + rollbackAcrossLineageRejectedCase(preparedReplaceEnabledTable(format)), + setCurrentSnapshotRecoversCase(preparedReplaceEnabledTable(format)), + changelogAcrossBoundaryCase(preparedReplaceEnabledTable(format)), + incrementalReadAcrossBoundaryCase(preparedReplaceEnabledTable(format)), + replaceThenRenameCase(preparedReplaceEnabledTable(format)), + renameThenReplaceCase(preparedReplaceEnabledTable(format)), + sortOrderChangedAfterReplaceCase(preparedReplaceEnabledTable(format)), + sortOrderRemovedAfterReplaceCase(preparedReplaceEnabledTable(format)), + creatorIdentityPreservedCase(preparedReplaceEnabledTable(format)), + replaceVersusAppendCase(preparedReplaceEnabledTable(format))) + } + + // --- the replace preparations the DML axis runs on --- + + /** + * One replace preparation per columnar format and partitioning: the table is created replace-enabled, seeded with + * the standard rows, re-specified in place by a same-shape CREATE OR REPLACE TABLE AS SELECT, then refreshed. The + * result holds the standard seed reached through the replace path, so every reusable DML operation that holds on a + * freshly seeded table must also hold here. + */ + lazy val preparedRtasCoreTables: List[TablePreparation[CoreTable.type]] = + for { + format <- fileFormats + partitioning <- partitionings + } yield TablePreparation( + s"${partitioning.label}/$format", + replaceLineage(partitioning, format), + rtasCasePrefix) + + /** + * One replace preparation per date-partitioned layout, so the partition-scoped writes replace whole partitions of a + * table that reached those partitions through the replace path. + */ + lazy val preparedRtasPartitionedCoreTables: List[TablePreparation[CoreTable.type]] = + fileFormats.map(format => + TablePreparation( + s"${partitionedByDate.label}/$format", + replaceLineage(partitionedByDate, format), + rtasCasePrefix)) + + /** The replace preparations, each carrying one row whose string column is null. */ + lazy val preparedNullStringRtasCoreTables: List[TablePreparation[CoreTable.type]] = + preparedRtasCoreTables.map(withNullStringRow) + + /** The prefix that marks a case ID as running on a table that reached its starting state through a replace. */ + val rtasCasePrefix: String = "prep.rtas:" + + // --- the starting states, shared helpers and case bodies the surface above composes --- + + /** The layout of a table the catalog will let a case replace: the core shape, shaped by `partitioning`. */ + private def replaceEnabledLayout(partitioning: Partitioning, format: String): Layout = + Layout( + s"${partitioning.label}/$format", + table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES ('write.format.default'='$format', 'replace.enabled'='true')") + + /** + * Creates a replace-enabled table, seeds the standard rows, re-specifies the same shape through CREATE OR REPLACE + * TABLE AS SELECT, and refreshes it. The REFRESH is required: the Spark session holds the table state it read + * before the replace, and REFRESH re-reads the committed metadata pointer so later statements in the session + * address the replaced table. + */ + private def replaceLineage( + partitioning: Partitioning, + format: String): TableTest[CoreTable.type] = + create(replaceEnabledLayout(partitioning, format)) + .insert(standardSeedRowCount)() + .sql("prep.rtas")(table => + s"CREATE OR REPLACE TABLE $table USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES ('write.format.default'='$format') AS SELECT * FROM $table")(view => { + assertSeededCoreShape(view, "prep.rtas") + assert( + view.snapshotsAfter == view.snapshotsBefore + 1, + s"prep.rtas commits one snapshot, went from ${view.snapshotsBefore} to " + + s"${view.snapshotsAfter}") + }) + .step("prep.rtas.refresh")((spark, table) => { + val currentSnapshotBefore = currentSnapshotId(spark, table) + val snapshotCountBefore = PreparedTable.snapshotCount(spark, table) + spark.sql(s"REFRESH TABLE $table") + assert( + currentSnapshotId(spark, table) == currentSnapshotBefore, + s"prep.rtas.refresh keeps main on snapshot $currentSnapshotBefore, moved it to " + + s"${currentSnapshotId(spark, table)}") + assert( + PreparedTable.snapshotCount(spark, table) == snapshotCountBefore, + s"prep.rtas.refresh keeps the snapshot count at $snapshotCountBefore") + })(view => { + assertSeededCoreShape(view, "prep.rtas.refresh") + assert( + view.snapshotsAfter == view.snapshotsBefore, + s"prep.rtas.refresh reads committed metadata and commits nothing, went from " + + s"${view.snapshotsBefore} to ${view.snapshotsAfter} snapshots") + }) + + /** + * The state both replace-preparation steps leave behind: the standard seed rows in key order, unchanged by the step, + * under exactly the core columns in their declared order. Both steps assert it, so a replace that loses a row, + * reorders the schema or drops a column fails during preparation, so the DML cases always compare against a known + * baseline. + */ + private def assertSeededCoreShape(view: StepView[CoreTable.type], stepLabel: String): Unit = { + val schemaColumnNames = view.spark.table(view.table).schema.fieldNames.toSeq + + assert( + schemaColumnNames == Core.columnNames, + s"$stepLabel presents the core schema, found $schemaColumnNames") + assert( + view.after == view.before, + s"$stepLabel keeps every row it started from, went from ${view.before} to ${view.after}") + assert( + view.after.size == standardSeedRowCount, + s"$stepLabel holds the $standardSeedRowCount standard seed rows, found ${view.after.size}") + assert( + inKeyOrder(view.after) == view.after, + s"$stepLabel returns the seed rows in key order, found ${view.after}") + assert( + view.after.map(row => Rows.TypedRow(row).get(Core.long0)) == + (1L to standardSeedRowCount.toLong).toList, + s"$stepLabel holds the standard seed keys, found " + + s"${view.after.map(row => Rows.TypedRow(row).get(Core.long0))}") + } + + /** + * The snapshot the table's main branch currently points at, read from the refs metadata table, which names exactly + * one snapshot per branch. A replace starts a second root in the snapshots metadata table, so main is the one + * source that identifies the live snapshot after a replace. + */ + private def currentSnapshotId(spark: SparkSession, table: String): Long = + spark + .sql(s"SELECT snapshot_id FROM $table.refs WHERE name = 'main'") + .collect() + .toSeq + .map(_.getLong(0)) match { + case Seq(snapshotId) => snapshotId + case mainSnapshotIds => + throw new AssertionError(s"main names one snapshot, found $mainSnapshotIds") + } + + /** + * The standard seed in an unpartitioned replace-enabled table in `format`, so a contract case starts from a table + * the catalog will let it replace and drives the replace itself. + */ + private def preparedReplaceEnabledTable(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + create(replaceEnabledLayout(unpartitioned, format)).insert(standardSeedRowCount)()) + + /** The same starting state partitioned by the date column, for the cases that repartition after a replace. */ + private def preparedReplaceEnabledPartitionedTable( + format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + create(replaceEnabledLayout(partitionedByDate, format)).insert(standardSeedRowCount)()) + + /** + * A replace-enabled table in `format` carrying the user property user.key=v1, so a case reads back what a replace + * does to a property the user set. + */ + private def preparedUserPropertyTable(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"'write.format.default'='$format', 'replace.enabled'='true', 'user.key'='v1')")() + .insert(standardSeedRowCount)()) + + /** + * A date-partitioned replace-enabled table in `format` carrying a 30-day retention policy on the date column, so a + * case reads back what a replace does to a policy the catalog stores. + */ + private def preparedRetentionPolicyTable(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + create(replaceEnabledLayout(partitionedByDate, format)) + .insert(standardSeedRowCount)() + .sql("setRetentionPolicy")(table => + s"ALTER TABLE $table SET POLICY " + + s"(RETENTION = 30d ON COLUMN ${Core.date0.columnName} " + + "WHERE pattern = 'yyyy-MM-dd-HH')")()) + + /** + * A replace-enabled table in `format` whose string column carries the PII tag, so a case reads back what a replace + * does to a column classification. + */ + private def preparedTaggedTable(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + create(replaceEnabledLayout(unpartitioned, format)) + .insert(standardSeedRowCount)() + .sql("tagStringColumnAsPii")(table => + s"ALTER TABLE $table MODIFY COLUMN ${Core.string0.columnName} SET TAG = (PII)")()) + + /** The statement that replaces `table` in place with the rows whose key is at most `keyLimit`. */ + private def replaceWithKeysUpTo(table: String, keyLimit: Int): String = + s"CREATE OR REPLACE TABLE $table USING $dataSource " + + s"AS SELECT * FROM $table WHERE ${Core.long0.columnName} <= $keyLimit" + + /** + * The message Iceberg raises when a requested snapshot range starts outside the lineage the table currently follows. + * A replace starts a new lineage, so both the changelog view and the incremental scan report a pre-replace start + * snapshot this way. + */ + private val crossLineageRejectionMessage = "is not a parent ancestor of end snapshot" + + /** The two outcomes a racing writer records: its statement committed, or it hit a typed commit conflict. */ + private val committedOutcome = "committed" + private val conflictedOutcome = "conflicted" + + /** + * The markers a rejection carries when the catalog refuses a column type change as incompatible. The in-place ALTER + * COLUMN TYPE path answers with the Spark analyzer marker, and the catalog answers with a message naming the + * change it refused. + */ + private val incompatibleTypeRejectionMarkers = + List("NOT_SUPPORTED_CHANGE_COLUMN", "incompatible", "cannot be cast", "narrow") + + /** The statement that replaces `table` in place with `projection` selected from it. */ + private def replaceWithProjection(table: String, projection: String): String = + s"CREATE OR REPLACE TABLE $table USING $dataSource AS SELECT $projection FROM $table" + + /** The reserved properties that identify a table to the catalog and must outlive a replace. */ + private val identityPropertyNames = List( + "openhouse.tableUUID", + "openhouse.tableId", + "openhouse.databaseId", + "openhouse.tableCreator") + + /** The values `table` currently reports for the reserved identity properties. */ + private def identityProperties( + table: PreparedTable[CoreTable.type]): Map[String, String] = { + val properties = tableProps(table.spark, table.name) + identityPropertyNames.flatMap(name => properties.get(name).map(name -> _)).toMap + } + + /** The column names the table reports, in the order it reports them. */ + private def columnNamesOf(table: PreparedTable[CoreTable.type], name: String): Seq[String] = + table.spark.sql(s"SELECT * FROM $name LIMIT 0").columns.toSeq + + // --- 1. the gates that decide whether a replace is allowed at all --- + + /** + * With replace.enabled=true, CREATE OR REPLACE TABLE AS SELECT replaces the table's content, leaving exactly the two + * rows the replacement query selected. + */ + private def enablementGateCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.gate.enabled") { table => + table.spark.sql(s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('replace.enabled'='true')") + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "2", + "an enabled replace should leave only the rows its query selected") + } + + /** + * On a table that has left replace.enabled unset, CREATE OR REPLACE TABLE AS SELECT is rejected with a + * BadRequestException naming the disabled feature, and the table keeps the rows it had, so a table opts in before + * anything rewrites it. + */ + private def disabledGateRejectedCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.gate.disabled.rejected") { table => + val stateBefore = table.state + val exception = Check.intercept[BadRequestException]( + table.spark.sql(replaceWithKeysUpTo(table.name, 2))) + + assert( + exception.getMessage.contains("REPLACE TABLE AS SELECT is not enabled"), + s"unexpected message: ${exception.getMessage.take(160)}") + assert(table.state == stateBefore, "a rejected replace should leave the table as it was") + } + + /** + * With replace.enabled=true but a replication policy also set, CREATE OR REPLACE TABLE AS SELECT is rejected with a + * BadRequestException naming replication, so a replicated table keeps the lineage its replicas follow. + */ + private def replicationGateRejectedCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.gate.replicationConflict.rejected") { table => + table.spark.sql(s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('replace.enabled'='true')") + table.spark.sql( + s"ALTER TABLE ${table.name} SET POLICY (REPLICATION = ({destination:'WAR'}))") + val stateBefore = table.state + val exception = Check.intercept[BadRequestException]( + table.spark.sql(replaceWithKeysUpTo(table.name, 2))) + + assert( + exception.getMessage.contains("while replication is enabled"), + s"unexpected message: ${exception.getMessage.take(160)}") + assert(table.state == stateBefore, "a rejected replace should leave the table as it was") + } + + // --- 2. the plainest replace, and the write that follows it --- + + /** + * A same-shape CREATE OR REPLACE TABLE AS SELECT keeps every column in its declared order and every row it selected, + * and commits exactly one snapshot, so replacing a table with itself changes nothing a reader can see except the + * lineage. + */ + private def sameShapeReplacementCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.sameShapeReplacement") { table => + val rowsBefore = table.rows + val snapshotsBefore = table.snapshotCount + table.spark.sql(replaceWithProjection(table.name, columnNameList)) + + assert( + columnNamesOf(table, table.name) == Core.columnNames, + "a same-shape replace should keep the declared columns in order") + assert( + inKeyOrder(table.rows) == inKeyOrder(rowsBefore), + "a same-shape replace should keep every row it selected") + assert( + table.snapshotCount == snapshotsBefore + 1, + s"a replace should commit one snapshot, went from $snapshotsBefore to ${table.snapshotCount}") + } + + /** + * A replaced table accepts an INSERT immediately afterwards and the row lands, so a writer that follows a replace in + * the same session addresses the replaced table, which is the lineage the replace made current. + */ + private def writeAfterReplaceCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.writeAfterReplace") { table => + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + table.spark.sql(s"INSERT INTO ${table.name} VALUES ${coreRow(6L, "row-6")}") + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", + "the replaced table should hold the two selected rows plus the inserted one") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} WHERE ${Core.long0.columnName} = 6") == "1", + "the row inserted after the replace should be readable") + } + + // --- 3. the four schema discontinuities a replace can introduce --- + + /** + * A replace whose projection adds a computed column widens the schema to that column and every row carries its + * value, so a replace is how a caller adds a column with data already in it. + */ + private def schemaAddColumnCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.schema.addColumn") { table => + table.spark.sql( + replaceWithProjection(table.name, s"$columnNameList, CAST(7 AS INT) AS added_col")) + + assert( + columnNamesOf(table, table.name) == Core.columnNames :+ "added_col", + "a replace that projects a new column should add it after the existing ones") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} WHERE added_col = 7") == + standardSeedRowCount.toString, + "every row should carry the value the projection computed") + } + + /** + * A replace whose projection names fewer columns drops the rest while preserving every row, so a replace is how a + * caller removes a column the catalog refuses to drop in place. + */ + private def schemaDropColumnCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.schema.dropColumn") { table => + table.spark.sql( + replaceWithProjection( + table.name, + s"${Core.long0.columnName}, ${Core.string0.columnName}")) + + assert( + columnNamesOf(table, table.name) == + Seq(Core.long0.columnName, Core.string0.columnName), + "a replace that projects two columns should leave exactly those two") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == + standardSeedRowCount.toString, + "dropping a column through a replace should preserve every row") + } + + /** + * A replace that casts the int column to bigint widens it and every value reads back unchanged, so a replace carries + * a widening type change the same way an in-place ALTER COLUMN TYPE does. + */ + private def schemaWidenColumnCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.schema.widenColumn") { table => + val intValuesBefore = table.spark + .sql(s"SELECT ${Core.int0.columnName} FROM ${table.name} ORDER BY ${Core.int0.columnName}") + .collect() + .toSeq + .map(_.getInt(0).toLong) + table.spark.sql( + replaceWithProjection( + table.name, + s"${Core.long0.columnName}, " + + s"CAST(${Core.int0.columnName} AS BIGINT) AS ${Core.int0.columnName}")) + val widenedType = table.spark + .table(table.name) + .schema + .fields + .toList + .collectFirst { case field if field.name == Core.int0.columnName => field.dataType.simpleString } + + assert( + widenedType.contains("bigint"), + s"the replace should widen the int column to bigint, got $widenedType") + assert( + table.spark + .sql(s"SELECT ${Core.int0.columnName} FROM ${table.name} ORDER BY ${Core.int0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) == intValuesBefore, + "widening through a replace should preserve every value") + } + + /** + * A replace narrows the bigint key column to int on a table holding a key wider than an int. A replace defines a + * fresh schema, so it accepts narrowing where the in-place ALTER COLUMN TYPE path refuses it. The contract is that + * the value survives either way: the replace is rejected for the type incompatibility and leaves the bigint column + * and its key intact, or it is accepted and the key still reads back as the key that was written. + * + * The current product accepts the replace, reports success and returns -1294967296 for the key 3000000000. This case + * keeps value preservation as the assertion so the gap stays visible, and is skipped until the product either + * rejects the narrowing or preserves the value. + */ + private def schemaIncompatibleTypeRejectedCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation + .test("rtas.schema.incompatibleType.notSilentlyLossy") { table => + val outOfRangeKey = 3000000000L + table.spark.sql( + s"INSERT INTO ${table.name} VALUES " + + s"(CAST($outOfRangeKey AS BIGINT), 4, 'row-big', 4.5, true, '2024-01-04-03')") + + val rejectionMessage = + try { + table.spark.sql( + replaceWithProjection( + table.name, + s"CAST(${Core.long0.columnName} AS INT) AS ${Core.long0.columnName}, " + + s"${Core.string0.columnName}")) + None + } catch { + case rejection: AnalysisException => Some(rejection.getMessage) + case rejection: BadRequestException => Some(rejection.getMessage) + case rejection: ValidationException => Some(rejection.getMessage) + } + val keyColumnType = table.spark + .table(table.name) + .schema + .fields + .toList + .collectFirst { + case field if field.name == Core.long0.columnName => field.dataType.simpleString + } + val storedKeys = table.spark + .sql( + s"SELECT ${Core.long0.columnName} FROM ${table.name} " + + s"WHERE ${Core.string0.columnName} = 'row-big'") + .collect() + .toSeq + .map(row => row.get(0).asInstanceOf[Number].longValue) + + rejectionMessage match { + case Some(message) => + assert( + incompatibleTypeRejectionMarkers.exists(message.contains), + s"the rejection identifies the type incompatibility, found: ${message.take(200)}") + assert( + keyColumnType.contains("bigint"), + s"a rejected narrowing leaves the key column bigint, found $keyColumnType") + assert( + storedKeys == List(outOfRangeKey), + s"a rejected narrowing leaves the key at $outOfRangeKey, found $storedKeys") + case None => + assert( + storedKeys == List(outOfRangeKey), + s"an accepted narrowing preserves the key $outOfRangeKey, " + + s"found $storedKeys under type $keyColumnType") + } + } + .copy(knownBugReason = Some( + "A replace that narrows bigint to int is accepted and wraps an out-of-range key around " + + "while the contract requires rejection or value preservation. The key 3000000000 reads back as " + + "-1294967296 after the product reports success.")) + + // --- 4. partition discontinuities, and evolving the partitioning the replace installed --- + + /** + * A replace with a new PARTITIONED BY clause installs that partition specification and preserves every row, so a + * replace is the supported repartitioning path after the catalog rejects in-place partition evolution. + */ + private def partitionSpecReplacedCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.partition.specReplaced") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"PARTITIONED BY (${Core.date0.columnName}) AS SELECT * FROM ${table.name}") + val description = table.spark.sql(s"DESCRIBE TABLE ${table.name}").collect().toSeq + + assert( + description.exists(_.getString(0) == "# Partition Information") && + description.count(_.getString(0) == Core.date0.columnName) == 2, + "the replace should install the partition specification it named") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == + standardSeedRowCount.toString, + "repartitioning through a replace should preserve every row") + } + + /** + * After a replace installs a date partition specification, ALTER TABLE DROP PARTITION FIELD is still rejected, and a + * second replace with a different PARTITIONED BY clause is what changes the partitioning. In-place partition + * evolution stays rejected across a replace, so replacing the table again is the one legal way to repartition it, + * which is what the catalog's own rejection message tells a caller to do. + */ + private def partitionChangeAfterReplaceCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.partition.changeAfterReplace") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"PARTITIONED BY (${Core.date0.columnName}) AS SELECT * FROM ${table.name}") + val inPlaceEvolution = Check.intercept[Exception]( + table.spark.sql( + s"ALTER TABLE ${table.name} DROP PARTITION FIELD ${Core.date0.columnName}")) + + assert( + inPlaceEvolution.getMessage.contains("Evolution of table partitioning"), + s"unexpected message: ${inPlaceEvolution.getMessage.take(160)}") + + table.spark.sql(replaceWithProjection(table.name, columnNameList)) + val description = table.spark.sql(s"DESCRIBE TABLE ${table.name}").collect().toSeq + + assert( + !description.exists(_.getString(0) == "# Partition Information"), + "the second replace should leave the table unpartitioned") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == + standardSeedRowCount.toString, + "repartitioning through a second replace should preserve every row") + table.spark.sql(s"INSERT INTO ${table.name} VALUES ${coreRow(6L, "row-6")}") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "4", + "the repartitioned table should stay writable") + } + + // --- 5. what a replace does to the properties a user set --- + + /** + * A replace that omits TBLPROPERTIES preserves the user property and the enablement flag the table carried, keeping + * the existing configuration. + */ + private def userPropertyPreservedCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.property.userPropertyPreserved") { table => + assert( + tableProps(table.spark, table.name).get("user.key").contains("v1"), + "the preparation should set the user property the replace is asked to preserve") + + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + val properties = tableProps(table.spark, table.name) + + assert( + properties.get("user.key").contains("v1"), + s"user.key should survive the replace, got ${properties.get("user.key")}") + assert( + properties.get("replace.enabled").contains("true"), + "replace.enabled should survive the replace") + } + + /** + * A replace whose TBLPROPERTIES clause names an existing property overrides that one and preserves every omitted + * property, so the statement decides exactly what it mentions. + */ + private def statementOverridesPropertyCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.property.statementOverridesProperty") { table => + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + "TBLPROPERTIES ('user.key'='v2') " + + s"AS SELECT * FROM ${table.name} WHERE ${Core.long0.columnName} <= 2") + val properties = tableProps(table.spark, table.name) + + assert( + properties.get("user.key").contains("v2"), + s"the property the statement named should win, got ${properties.get("user.key")}") + assert( + properties.get("replace.enabled").contains("true"), + "a property the statement omits should survive the replace") + } + + // --- 6. what a replace does to the governance the catalog stores --- + + /** + * A replace that also installs a new partition specification preserves the retention policy the catalog stored, so + * replacing a table's content keeps the rule that ages its data out. + */ + private def retentionPolicyPreservedCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.policy.retentionPreserved") { table => + val policiesBefore = tableProps(table.spark, table.name).getOrElse("policies", "") + assert( + policiesBefore.toLowerCase.contains("retention"), + s"the preparation should store the retention policy the replace must preserve: $policiesBefore") + + table.spark.sql( + s"CREATE OR REPLACE TABLE ${table.name} USING $dataSource " + + s"PARTITIONED BY (${Core.date0.columnName}) " + + s"AS SELECT * FROM ${table.name} WHERE ${Core.long0.columnName} <= 2") + + assert( + tableProps(table.spark, table.name).getOrElse("policies", "") == policiesBefore, + "the replace should preserve the retention policy") + } + + /** + * A replace preserves the PII tag the string column carried, so replacing a table's content keeps a column's + * classification. + */ + private def columnTagPreservedCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.policy.columnTagPreserved") { table => + val policiesBefore = tableProps(table.spark, table.name).getOrElse("policies", "") + assert( + policiesBefore.toLowerCase.contains("pii"), + s"the preparation should store the PII tag the replace must preserve: $policiesBefore") + + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + + assert( + tableProps(table.spark, table.name).getOrElse("policies", "") == policiesBefore, + "the replace should preserve the PII column tag") + } + + // --- 7. reading the history a replace retired --- + + /** + * A replace keeps the pre-replace snapshot in history and that snapshot still reads its three rows, so the content a + * replace overwrote stays reachable by time travel. + */ + private def preReplaceTimeTravelCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.history.preReplaceTimeTravel") { table => + val preReplaceSnapshotId = currentSnapshotId(table.spark, table.name) + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}.snapshots") == "2", + "the replace appends to the history it found, leaving the pre-replace snapshot in place") + assert( + countOf( + table.spark, + s"SELECT count(*) FROM ${table.name} VERSION AS OF $preReplaceSnapshotId") == + standardSeedRowCount.toString, + "the pre-replace snapshot should still read the rows it held") + } + + /** + * Rolling back to a snapshot from before the replace is rejected because the replace started a new lineage and the + * earlier snapshot lies outside the current ancestry. + */ + private def rollbackAcrossLineageRejectedCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.history.rollbackRejected") { table => + val preReplaceSnapshotId = currentSnapshotId(table.spark, table.name) + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + + val exception = Check.intercept[ValidationException]( + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"'${catalogRelative(table.name)}', $preReplaceSnapshotId)")) + + assert( + exception.getMessage.contains("not an ancestor"), + s"unexpected message: ${exception.getMessage.take(160)}") + } + + /** + * set_current_snapshot to a pre-replace snapshot recovers the rows the replace overwrote, so the snapshot a rollback + * refuses is still the way back to the content that was there before. + */ + private def setCurrentSnapshotRecoversCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.history.setCurrentSnapshotRecovers") { table => + val preReplaceSnapshotId = currentSnapshotId(table.spark, table.name) + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + table.spark.sql( + "CALL openhouse.system.set_current_snapshot(" + + s"'${catalogRelative(table.name)}', $preReplaceSnapshotId)") + + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == + standardSeedRowCount.toString, + "set_current_snapshot should recover the pre-replace rows") + } + + // --- 8 and 9. asking for a range of changes that crosses the replacement boundary --- + + /** + * A changelog view whose start snapshot sits before the replace is rejected with an IllegalArgumentException naming + * the start snapshot as outside the current lineage, so a reader asking to span the replacement boundary is told + * the range is unanswerable and reads the new lineage's changes only through a range inside it. The append that + * follows the replace comes from ChangelogSupport, so this case and the general changelog cases agree on what the + * operation does. + */ + private def changelogAcrossBoundaryCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.changelog.acrossBoundaryRejected") { table => + val appendOperation = changelogOperations + .find(_.name == "changelog.append") + .getOrElse(throw new AssertionError("ChangelogSupport defines the changelog.append operation")) + val preReplaceSnapshotId = currentSnapshotId(table.spark, table.name) + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + table.spark.sql(appendOperation.statement(table.name)) + val rejection = Check.intercept[IllegalArgumentException]( + changeCounts(table, changelogViewFrom(table, preReplaceSnapshotId))) + + assert( + rejection.getMessage.contains(crossLineageRejectionMessage), + s"the rejection identifies the start snapshot as outside the current lineage, " + + s"found: ${rejection.getMessage.take(200)}") + } + + /** + * An incremental read bounded by a snapshot from before the replace and the snapshot the append after it made + * current is rejected with an IllegalArgumentException naming the start snapshot as outside the current lineage, so + * a scan spans one lineage at a time. + */ + private def incrementalReadAcrossBoundaryCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.incrementalRead.acrossBoundaryRejected") { table => + val preReplaceSnapshotId = currentSnapshotId(table.spark, table.name) + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + table.spark.sql(s"INSERT INTO ${table.name} VALUES ${coreRow(6L, "row-6")}") + val postAppendSnapshotId = currentSnapshotId(table.spark, table.name) + + val rejection = Check.intercept[IllegalArgumentException]( + table.spark.read + .format("iceberg") + .option("start-snapshot-id", preReplaceSnapshotId) + .option("end-snapshot-id", postAppendSnapshotId) + .load(table.name) + .count()) + + assert( + rejection.getMessage.contains(crossLineageRejectionMessage), + s"the rejection identifies the start snapshot as outside the current lineage, " + + s"found: ${rejection.getMessage.take(200)}") + } + + // --- 10. a replace crossed with a rename, in both orders --- + + /** + * A table replaced and then renamed keeps the replaced content under the new name, so a replace leaves a table + * to the name it was replaced under. The rename boundary records the live name after each accepted rename, so a + * failure between the two renames drops the table under the name it currently answers to. + */ + private def replaceThenRenameCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.rename.replaceThenRename") { table => + val renamedTable = s"${table.name}_replaced_then_renamed" + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + + withTrackedRename(table.spark.sql(_), table.name) { renameTo => + renameTo(renamedTable) + + assert( + countOf(table.spark, s"SELECT count(*) FROM $renamedTable") == "2", + "the renamed table should hold the rows the replace left") + renameTo(table.name) + } + } + + /** + * A table renamed and then replaced under its new name accepts the replace and holds the replaced content, so a + * rename keeps a table on the replace path. + */ + private def renameThenReplaceCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.rename.renameThenReplace") { table => + val renamedTable = s"${table.name}_renamed_then_replaced" + + withTrackedRename(table.spark.sql(_), table.name) { renameTo => + renameTo(renamedTable) + table.spark.sql(replaceWithKeysUpTo(renamedTable, 2)) + + assert( + countOf(table.spark, s"SELECT count(*) FROM $renamedTable") == "2", + "the table renamed before the replace should hold the rows the replace left") + renameTo(table.name) + } + } + + // --- 11. evolving the sort order the replaced table starts with --- + + /** + * A replaced table accepts ALTER TABLE WRITE ORDERED BY afterwards, which sets range distribution and leaves the + * table writable, so the write order stays settable after a replace. + */ + private def sortOrderChangedAfterReplaceCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.sortOrder.changedAfterReplace") { table => + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + table.spark.sql(s"ALTER TABLE ${table.name} WRITE ORDERED BY ${Core.long0.columnName}") + val distributionMode = tableProps(table.spark, table.name).get("write.distribution-mode") + + assert( + distributionMode.contains("range"), + s"a write sort order after a replace should set range distribution, got $distributionMode") + table.spark.sql(s"INSERT INTO ${table.name} VALUES ${coreRow(6L, "row-6")}") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", + "the ordered replaced table should stay writable") + } + + /** + * A replaced table that was given a write sort order accepts ALTER TABLE WRITE UNORDERED afterwards, which drops the + * range distribution and leaves the table writable, so a sort order applied after a replace is still removable. + */ + private def sortOrderRemovedAfterReplaceCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.sortOrder.removedAfterReplace") { table => + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + table.spark.sql(s"ALTER TABLE ${table.name} WRITE ORDERED BY ${Core.long0.columnName}") + table.spark.sql(s"ALTER TABLE ${table.name} WRITE UNORDERED") + val distributionMode = tableProps(table.spark, table.name).get("write.distribution-mode") + + assert( + !distributionMode.contains("range"), + s"dropping the sort order should drop range distribution, got $distributionMode") + table.spark.sql(s"INSERT INTO ${table.name} VALUES ${coreRow(6L, "row-6")}") + assert( + countOf(table.spark, s"SELECT count(*) FROM ${table.name}") == "3", + "the unordered replaced table should stay writable") + } + + // --- 12. the identity the catalog governs the table by --- + + /** + * A replace preserves every reserved property that identifies the table, including the creator the catalog recorded, + * so the table the catalog governs after a replace is the same table it governed before, which keeps a replace from + * being a way to take over a table's identity. + */ + private def creatorIdentityPreservedCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.identity.creatorPreserved") { table => + val identityBefore = identityProperties(table) + assert( + identityBefore.contains("openhouse.tableCreator"), + s"the catalog should record a creator before the replace: ${identityBefore.keys.toList.sorted}") + + table.spark.sql(replaceWithKeysUpTo(table.name, 2)) + + assert( + identityProperties(table) == identityBefore, + s"the replace changed the table's identity from $identityBefore to ${identityProperties(table)}") + } + + // --- 13. a replace racing another writer --- + + /** + * A replace racing an INSERT settles at either the two rows the replace selected, where the replace committed last, + * or three rows, where the append landed on the replaced table. Whichever writer loses fails with a typed commit + * conflict, so a caller recognizes every way this race ends. + */ + private def replaceVersusAppendCase(preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("rtas.concurrency.replaceVersusAppend") { table => + val outcomeByWriter = new ConcurrentHashMap[String, String]() + def writer(writerName: String, statement: String): () => Unit = () => + try { + table.spark.sql(statement) + outcomeByWriter.put(writerName, committedOutcome) + } catch { + case NonFatal(conflict) if ConcurrencySupport.isTypedCommitConflict(conflict) => + outcomeByWriter.put(writerName, conflictedOutcome) + } + + val threadErrors = ConcurrencySupport.runConcurrently( + Seq( + writer("replace", replaceWithKeysUpTo(table.name, 2)), + writer("append", s"INSERT INTO ${table.name} VALUES ${coreRow(30L, "row-30")}"))) + assert( + threadErrors.isEmpty, + s"both writers either commit or hit a typed commit conflict, found: $threadErrors") + + table.spark.sql(s"REFRESH TABLE ${table.name}") + val settledKeys = table.spark + .sql(s"SELECT ${Core.long0.columnName} FROM ${table.name}") + .collect() + .toSeq + .map(_.getLong(0)) + .toSet + val raceOutcome = + (outcomeByWriter.get("replace"), outcomeByWriter.get("append")) + + println(s"DIAG rtas.concurrency.replaceVersusAppend: $raceOutcome settled at $settledKeys") + raceOutcome match { + case (`committedOutcome`, `conflictedOutcome`) => + assert( + settledKeys == Set(1L, 2L), + s"a winning replace leaves the keys it selected, found $settledKeys") + case (`conflictedOutcome`, `committedOutcome`) => + assert( + settledKeys == Set(1L, 2L, 3L, 30L), + s"a winning append leaves the seed plus its row, found $settledKeys") + case (`committedOutcome`, `committedOutcome`) => + assert( + settledKeys == Set(1L, 2L) || settledKeys == Set(1L, 2L, 30L), + s"two commits leave the replaced rows, with the append included when it landed " + + s"on the replaced table, found $settledKeys") + case recordedOutcome => + throw new AssertionError( + s"one writer commits when a replace races an append, recorded $recordedOutcome") + } + } + +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala new file mode 100644 index 000000000..8c62c90ad --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala @@ -0,0 +1,222 @@ +package harness + +import java.nio.charset.StandardCharsets +import java.security.MessageDigest + +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} +import org.junit.jupiter.api.Test + +/** + * Pins the replace-table contribution this layer adds: its exact size, the fingerprint of its case IDs, the four + * preparation axes its DML runs on, the reusable DML operations it covers, and the replace contracts it claims to + * prove. + * + * Every assertion reads `rtasCases` and the RTAS families alone, so the frozen foundation keeps its own tests and a + * sibling layer that adds its own contribution leaves this file as it is. The catalog invariants that apply to any + * layer, namely ID uniqueness, alphabetical contribution ordering and the foundation-plus-extensions rule, are pinned + * once in CaseCatalogTest. + * + * Reading the catalog builds the case list only; executing a case and starting Spark stay separate steps. + */ +final class RtasCatalogTest { + private val expectedRtasCaseCount = 264 + private val expectedRtasSha256 = + "90512741431e50ca77d4924dd7c8b789e3fa6512a23d5b746bc397a00e4532d1" + + /** The four replace preparations the reusable DML operations run on, in the order the layer builds them. */ + private val expectedRtasPreparationLabels = List( + "unpartitioned/parquet", + "partitioned/parquet", + "unpartitioned/orc", + "partitioned/orc") + + /** + * Every replace contract this layer claims to prove, named by the case ID that proves it. The list is written out + * here as its own literal, so the layer and this list are independent statements of the same coverage and any + * drop, rename or reorder fails this test until the intended coverage is restated. + */ + private val expectedContractCaseNames = List( + "rtas.gate.enabled", + "rtas.gate.disabled.rejected", + "rtas.gate.replicationConflict.rejected", + "rtas.sameShapeReplacement", + "rtas.writeAfterReplace", + "rtas.schema.addColumn", + "rtas.schema.dropColumn", + "rtas.schema.widenColumn", + "rtas.schema.incompatibleType.notSilentlyLossy", + "rtas.partition.specReplaced", + "rtas.partition.changeAfterReplace", + "rtas.property.userPropertyPreserved", + "rtas.property.statementOverridesProperty", + "rtas.policy.retentionPreserved", + "rtas.policy.columnTagPreserved", + "rtas.history.preReplaceTimeTravel", + "rtas.history.rollbackRejected", + "rtas.history.setCurrentSnapshotRecovers", + "rtas.changelog.acrossBoundaryRejected", + "rtas.incrementalRead.acrossBoundaryRejected", + "rtas.rename.replaceThenRename", + "rtas.rename.renameThenReplace", + "rtas.sortOrder.changedAfterReplace", + "rtas.sortOrder.removedAfterReplace", + "rtas.identity.creatorPreserved", + "rtas.concurrency.replaceVersusAppend") + + @Test + def theReplaceContributionIsTheSizeAndShapeItIsPinnedTo(): Unit = { + val caseIds = Scenarios.rtasCases.map(_.id) + val actualSha256 = sha256(caseIds.mkString("\n")) + + assertEquals( + expectedRtasCaseCount, + caseIds.size, + s"rtasCases changed; count=${caseIds.size}, sha256=$actualSha256") + assertEquals( + expectedRtasSha256, + actualSha256, + s"rtasCases changed; count=${caseIds.size}, sha256=$actualSha256") + assertEquals(caseIds.distinct.size, caseIds.size, "replace case IDs must be unique") + assertEquals( + Scenarios.rtasDmlCases.map(_.id) ++ Scenarios.rtasContractCases.map(_.id), + caseIds, + "rtasCases is the DML axis followed by the replace contract") + } + + @Test + def theCatalogIntegratesTheReplaceContributionExactlyOnce(): Unit = { + val replaceEntries = ScenarioCatalog.extensionContributions.filter { + case (name, _) => name == "rtasCases" + } + + assertEquals( + 1, + replaceEntries.size, + s"rtasCases is integrated once, found ${replaceEntries.size} entries") + assertEquals( + Scenarios.rtasCases.map(_.id), + replaceEntries.head match { case (_, contribution) => contribution.map(_.id) }, + "the catalog integrates the very list the capability exposes") + } + + @Test + def theReplaceDmlAxisIsTheFourReplacePreparations(): Unit = { + assertEquals( + expectedRtasPreparationLabels, + Scenarios.preparedRtasCoreTables.map(_.label)) + assertEquals( + List("partitioned/parquet", "partitioned/orc"), + Scenarios.preparedRtasPartitionedCoreTables.map(_.label)) + assertEquals( + expectedRtasPreparationLabels, + Scenarios.preparedNullStringRtasCoreTables.map(_.label), + "the null-string preparations extend the same four replace preparations") + assertTrue( + (Scenarios.preparedRtasCoreTables ++ Scenarios.preparedRtasPartitionedCoreTables ++ + Scenarios.preparedNullStringRtasCoreTables) + .forall(_.casePrefix == Scenarios.rtasCasePrefix), + "every replace preparation marks its cases as running on a replaced table") + } + + @Test + def everyReplacePreparationReachesItsStartingStateThroughAReplace(): Unit = { + val replaceStepLabels = List("prep.rtas", "prep.rtas.refresh") + + Scenarios.preparedRtasCoreTables.foreach { preparation => + assertEquals( + List("create", s"insert(${Scenarios.standardSeedRowCount})") ++ replaceStepLabels, + preparation.preparation.steps.map(_.label).toList, + s"${preparation.label} creates, seeds, replaces and refreshes in that order") + } + Scenarios.preparedNullStringRtasCoreTables.foreach { preparation => + assertEquals( + List("prep.nullStringRow"), + preparation.preparation.steps.map(_.label).toList.takeRight(1), + s"${preparation.label} ends by adding the null row the null-string operation reads") + } + } + + @Test + def everyReusableDmlOperationRunsOnTheReplacePreparationsItAppliesTo(): Unit = { + val coveredOperationNames = Scenarios.rtasDmlCases + .map(_.id.stripPrefix(Scenarios.rtasCasePrefix).split(" @ ").head) + .distinct + .sorted + val reusableOperationNames = (Scenarios.allDmlTestCases ++ + Scenarios.nullStringRowTestCases ++ + Scenarios.partitionedTableTestCases).map(_.id).distinct.sorted + + assertEquals( + reusableOperationNames, + coveredOperationNames, + "every reusable DML operation runs on a replaced table") + assertEquals(54, reusableOperationNames.size, "the reusable DML operation count changed") + assertEquals( + Scenarios.preparedRtasCoreTables.flatMap(preparation => + Scenarios.allDmlTestCases.map(testCase => + s"${preparation.casePrefix}${testCase.id} @ ${preparation.label}")), + Scenarios.rtasCoreDmlCases.map(_.id), + "the core replace bucket is its preparations crossed with every reusable operation") + assertEquals(204, Scenarios.rtasCoreDmlCases.size) + assertEquals(4, Scenarios.rtasNullStringDmlCases.size) + assertEquals(4, Scenarios.rtasPartitionedDmlCases.size) + assertEquals(212, Scenarios.rtasDmlCases.size) + } + + @Test + def everyReplaceContractHasAtLeastOneCaseInEveryColumnarFormat(): Unit = { + val contractCaseNames = Scenarios.rtasContractCases + .map(_.id.split(" @ ").head) + .distinct + + assertEquals( + expectedContractCaseNames, + contractCaseNames, + "the replace contract families changed") + expectedContractCaseNames.foreach { contractCaseName => + assertEquals( + Scenarios.fileFormats.map(format => s"$contractCaseName @ $format"), + Scenarios.rtasContractCases.map(_.id).filter(_.startsWith(s"$contractCaseName @ ")), + s"$contractCaseName runs in every columnar format") + } + assertEquals( + expectedContractCaseNames.size * Scenarios.fileFormats.size, + Scenarios.rtasContractCases.size) + } + + @Test + def everyReplaceCaseRunsOnAColumnarFormatInTheLandingMatrix(): Unit = { + val preparationFormats = Scenarios.rtasCases + .map(_.id.split(" @ ").last) + .map(label => label.split("/").last) + .distinct + + assertEquals(List("parquet", "orc"), Scenarios.fileFormats) + assertEquals( + Scenarios.fileFormats.sorted, + preparationFormats.sorted, + s"every replace case runs on a landing-matrix format, found $preparationFormats") + } + + @Test + def theReplaceSkipMetadataIsPinnedToTheOneKnownProductBug(): Unit = { + assertEquals( + List( + "rtas.schema.incompatibleType.notSilentlyLossy @ orc", + "rtas.schema.incompatibleType.notSilentlyLossy @ parquet"), + Scenarios.rtasCases.collect { + case testCase if testCase.knownBugReason.nonEmpty => testCase.id + }.sorted, + "the replace known-bug cases changed") + assertTrue( + Scenarios.rtasCases.forall(_.embeddedSkipReason.isEmpty), + "every replace case reaches the embedded catalog") + } + + private def sha256(value: String): String = + MessageDigest + .getInstance("SHA-256") + .digest(value.getBytes(StandardCharsets.UTF_8)) + .map(byte => f"$byte%02x") + .mkString +} From 0c224c04643d7105e24b5fc8bbbc0e36dc729808 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Tue, 1 Sep 2026 22:55:54 -0700 Subject: [PATCH 16/24] test(delta-harness): add merge-on-read matrix Rebuild merge-on-read DML, changelog, delete-file, maintenance, and snapshot-history contracts on the focused RTAS catalog. The scenarios prove exact current file state, procedure effects, and write-mode behavior across Parquet and ORC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/openhouse/ScenarioCatalog.scala | 2 + .../openhouse/ScenarioMergeOnRead.scala | 487 ++++++++++++++++++ .../openhouse/ScenarioMergeOnReadKit.scala | 359 +++++++++++++ .../ScenarioMergeOnReadMaintenance.scala | 444 ++++++++++++++++ .../harness/MergeOnReadCatalogTest.scala | 250 +++++++++ 5 files changed, 1542 insertions(+) create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnRead.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadKit.scala create mode 100644 integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadMaintenance.scala create mode 100644 integrations/spark/delta-harness/src/test/scala/harness/MergeOnReadCatalogTest.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala index 73ee4fe5a..5b8c73761 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala @@ -27,6 +27,7 @@ object Scenarios with ScenarioSchemaEvolution with ScenarioTableProperty with ScenarioRtas + with ScenarioMergeOnRead with ChangelogSupport /** @@ -68,6 +69,7 @@ object ScenarioCatalog { */ def extensionContributions: List[(String, List[TestCase])] = List( + "mergeOnReadCases" -> Scenarios.mergeOnReadCases, "rtasCases" -> Scenarios.rtasCases) /** Every capability contribution, named once, in the order the catalog integrates them. */ diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnRead.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnRead.scala new file mode 100644 index 000000000..a80a0869d --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnRead.scala @@ -0,0 +1,487 @@ +package harness + +/** + * Merge-on-read: what changes when a mutation records a position-delete file beside the data file it matched and + * leaves that data file in place. + * + * The reusable DML contract has to hold identically on both write paths, because a caller writes the same SQL either + * way. Everything else in this file is behavior only a merge-on-read table has: the physical delete file itself, the + * surface a table reaches once data files and a live delete file sit side by side, the metadata that exposes the + * delete, the changelog's ability to decode it, the history that spans it, and the maintenance procedures that fold + * or carry it. Maintenance is large enough to review on its own, so it lives in ScenarioMergeOnReadMaintenance and + * joins the one contribution this layer names. + * + * Operations, DML: the row-mutating operations `ScenarioDml` defines, its null-string DELETE and its reads, reused as + * data. A merge-on-read table runs the same statements and the same row and snapshot delta assertions as a + * copy-on-write one, so this file holds one definition of each preparation and none of each operation. + * + * Operations, merge-on-read contract: 26 focused families. Nineteen live here and cover the physical delete file + * against its copy-on-write counterpart, a mode change applied partway through a table's life, the six operations + * that run once a delete file is live, the position_deletes metadata table, the three changelog operations a scan + * decodes and the two it reports as unsupported, format materialization with a delete file present, the delete-file + * replication property, and time travel and rollback across the delete. Seven more live in + * ScenarioMergeOnReadMaintenance. + * + * Preparation axes: the write mode is the axis this layer adds. Four merge-on-read layouts cross the two columnar + * formats with unpartitioned and date-partitioned tables; two replace-lineage merge-on-read layouts put the same + * mutations on a table that also went through a replace, which is this layer's one dependency on its parent; and two + * verify layouts per write mode seed into a single data file so a strict-subset delete is a partial-file match and + * the physical outcome is deterministic. + * + * Case families: 320 cases. The DML axis contributes 268 in three families, and the merge-on-read contract + * contributes 52 in 26 families: 38 in the 19 families here and 14 in the 7 maintenance families. + */ +trait ScenarioMergeOnRead extends ScenarioMergeOnReadMaintenance { + this: ScenarioDml with ScenarioFileFormat with ChangelogSupport => + + /** Every merge-on-read case: the reusable DML operations on merge-on-read tables, then the write-mode contract. */ + lazy val mergeOnReadCases: List[TestCase] = + mergeOnReadDmlCases ++ mergeOnReadContractCases ++ mergeOnReadMaintenanceCases + + /** + * The reusable DML operations on merge-on-read tables: every row-mutating operation on the four merge-on-read + * preparations and the two replace-lineage ones, the null-string DELETE on their null-string forms, and the reads + * on the preparations that already carry a live position-delete file. + */ + lazy val mergeOnReadDmlCases: List[TestCase] = + mergeOnReadCoreDmlCases ++ replacedMergeOnReadDmlCases ++ deletedMergeOnReadDmlCases + + /** Every row-mutating operation on the merge-on-read preparations, plus the null-string DELETE on their null form. */ + lazy val mergeOnReadCoreDmlCases: List[TestCase] = + preparedMergeOnReadCoreTables.flatMap(preparation => + rowMutationTestCases.map(_.runOn(preparation))) ++ + preparedNullStringMergeOnReadCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) + + /** The same operations on the replace-lineage merge-on-read preparations, so both paths apply at once. */ + lazy val replacedMergeOnReadDmlCases: List[TestCase] = + preparedReplacedMergeOnReadCoreTables.flatMap(preparation => + rowMutationTestCases.map(_.runOn(preparation))) ++ + preparedNullStringReplacedMergeOnReadCoreTables.flatMap(preparation => + nullStringRowTestCases.map(_.runOn(preparation))) + + /** The reads on preparations carrying a live position-delete file, so each read applies one at scan time. */ + lazy val deletedMergeOnReadDmlCases: List[TestCase] = + preparedDeletedMergeOnReadTables.flatMap(preparation => + readTestCases.map(_.runOn(preparation))) + + /** Every merge-on-read contract case outside maintenance, in the order this file introduces them. */ + lazy val mergeOnReadContractCases: List[TestCase] = + deleteFileCases ++ + deleteModeCases ++ + deleteFileCoexistenceCases ++ + mergeOnReadMetadataCases ++ + mergeOnReadChangelogCases ++ + mergeOnReadFileFormatCases ++ + mergeOnReadFileReplicationCases ++ + mergeOnReadHistoryCases + + // --- 1. the physical delete file, and the copy-on-write outcome it is defined against --- + + /** + * A strict-subset DELETE against a single data file, run once on each write mode. Merge-on-read records the removal + * in a position-delete file and keeps the data file; copy-on-write rewrites the data file and leaves no delete + * file. Both remove the same row and commit one snapshot, so the write mode is the only difference. + */ + lazy val deleteFileCases: List[TestCase] = + mergeOnReadVerifyLayouts.map(layout => + TablePreparation(layout.label, singleFileSeed(layout)) + .test("mergeOnRead.deleteFile.writesDeleteFile")(table => + assertSubsetDeleteOutcome(table, expectedDeleteFileCount = 1))) ++ + copyOnWriteVerifyLayouts.map(layout => + TablePreparation(layout.label, singleFileSeed(layout)) + .test("mergeOnRead.deleteFile.copyOnWriteRewritesDataFile")(table => + assertSubsetDeleteOutcome(table, expectedDeleteFileCount = 0))) + + /** + * Runs the strict-subset DELETE and asserts the outcome both write modes share, namely that the matching row is + * gone and exactly one snapshot was committed, together with the delete-file count the mode under test produces. + */ + private def assertSubsetDeleteOutcome( + table: PreparedTable[CoreTable.type], + expectedDeleteFileCount: Long): Unit = { + val before = table.state + + table.spark.sql(s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} < 2") + val after = table.state + + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"the strict-subset delete leaves keys 2 and 3, found ${liveKeys(table.spark, table.name)}") + assert( + currentDeleteFileCount(table.spark, table.name) == expectedDeleteFileCount, + s"the delete leaves $expectedDeleteFileCount delete files, found " + + s"${currentDeleteFileCount(table.spark, table.name)}") + assert( + after.snapshotCount == before.snapshotCount + 1, + s"the delete commits one snapshot, went from ${before.snapshotCount} to ${after.snapshotCount}") + } + + // --- 2. choosing the write mode partway through a table's life --- + + /** + * Switching a copy-on-write table's delete mode to merge-on-read makes the next partial-file DELETE write a + * position-delete file and keep the untouched rows in the data file, so the mode a table carries at commit time is + * the one that decides how the delete is written. + */ + lazy val deleteModeCases: List[TestCase] = + fileFormats.map(format => + preparedSingleFileCopyOnWriteTable(format) + .test("mergeOnRead.deleteMode.alterToMergeOnRead") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')") + table.spark.sql(s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + + assert( + currentDeleteFileCount(table.spark, table.name) == 1, + s"the mode change makes the delete write one delete file, found " + + s"${currentDeleteFileCount(table.spark, table.name)}") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"the delete after the mode change leaves keys 2 and 3, found " + + s"${liveKeys(table.spark, table.name)}") + }) + + // --- 3. the surface a table reaches once a delete file is live beside its data --- + + /** + * The six operations that behave differently once data files and a live position-delete file sit side by side. A + * read or an insert on a delete-free merge-on-read table is identical to copy-on-write, so every family here starts + * from the state where a delete file is already live. + */ + lazy val deleteFileCoexistenceCases: List[TestCase] = + preparedDeletedMergeOnReadTables.flatMap { preparation => + List( + appendOverDeleteFileCase(preparation), + secondDeleteOverDeleteFileCase(preparation), + updateOverDeleteFileCase(preparation), + filteredReadOverDeleteFileCase(preparation), + compactDeletesOverDeleteFileCase(preparation), + mergeOverDeleteFileCase(preparation)) + } + + /** + * Asserts the table persists `propertyName` as merge-on-read before the mutation under test runs. The row + * assertions hold on either write path, so this guard is what ties the case to the merge-on-read path it claims to + * cover. + */ + private def assertConfiguredMergeOnRead( + table: PreparedTable[CoreTable.type], + propertyName: String): Unit = { + val configuredMode = persistedProperty(table.spark, table.name, propertyName) + + assert( + configuredMode.contains("merge-on-read"), + s"the table persists $propertyName as merge-on-read, found $configuredMode") + } + + /** An INSERT over a live position-delete file adds its row and keeps the deleted key out of the live rows. */ + private def appendOverDeleteFileCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.coexistence.append") { table => + table.spark.sql(s"INSERT INTO ${table.name} VALUES ${coreRow(6L, "row-6")}") + + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L, 6L), + s"the append lands beside the live delete, found ${liveKeys(table.spark, table.name)}") + } + + /** A second DELETE over a live position-delete file removes its row and the table still carries delete files. */ + private def secondDeleteOverDeleteFileCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.coexistence.secondDelete") { table => + table.spark.sql(s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 2") + + assert( + liveKeys(table.spark, table.name) == Seq(3L), + s"the second delete leaves key 3, found ${liveKeys(table.spark, table.name)}") + assert( + currentDeleteFileCount(table.spark, table.name) >= 1, + s"the second delete keeps delete files live, found " + + s"${currentDeleteFileCount(table.spark, table.name)}") + } + + /** An UPDATE over a live position-delete file changes its row's value and keeps the live key set. */ + private def updateOverDeleteFileCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.coexistence.update") { table => + assertConfiguredMergeOnRead(table, "write.update.mode") + table.spark.sql( + s"UPDATE ${table.name} SET ${Core.string0.columnName} = 'cx' " + + s"WHERE ${Core.long0.columnName} = 3") + val updatedValue = table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 3") + .collect()(0) + .getString(0) + + assert(updatedValue == "cx", s"the update over a live delete sets the value, found $updatedValue") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"the update keeps the live key set, found ${liveKeys(table.spark, table.name)}") + } + + /** A filtered read over a live position-delete file returns the live rows the filter selects. */ + private def filteredReadOverDeleteFileCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.coexistence.filteredRead") { table => + val selectedKeys = table.spark + .sql( + s"SELECT ${Core.long0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} <= 2 ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) + + assert( + selectedKeys == Seq(2L), + s"the filter applies the position delete, found $selectedKeys") + } + + /** Compacting the position deletes over a live delete file keeps the live rows. */ + private def compactDeletesOverDeleteFileCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.coexistence.compactDeletes") { table => + table.spark.sql( + "CALL openhouse.system.rewrite_position_delete_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"compacting the deletes keeps the live rows, found ${liveKeys(table.spark, table.name)}") + } + + /** A MERGE over a live position-delete file updates its matched row and keeps the live key set. */ + private def mergeOverDeleteFileCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.coexistence.merge") { table => + assertConfiguredMergeOnRead(table, "write.merge.mode") + table.spark.sql( + s"MERGE INTO ${table.name} target " + + "USING (SELECT CAST(3 AS BIGINT) key) source " + + s"ON target.${Core.long0.columnName} = source.key " + + s"WHEN MATCHED THEN UPDATE SET ${Core.string0.columnName} = 'mg'") + val mergedValue = table.spark + .sql( + s"SELECT ${Core.string0.columnName} FROM ${table.name} " + + s"WHERE ${Core.long0.columnName} = 3") + .collect()(0) + .getString(0) + + assert(mergedValue == "mg", s"the merge over a live delete sets the value, found $mergedValue") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"the merge keeps the live key set, found ${liveKeys(table.spark, table.name)}") + } + + // --- 4. the metadata that exposes what the reader will apply --- + + /** + * After a merge-on-read DELETE, the position_deletes metadata table reports exactly the one delete entry the + * mutation created, so what the reader applies at scan time is visible to a caller reading metadata. + */ + lazy val mergeOnReadMetadataCases: List[TestCase] = + fileFormats.map(format => + preparedSingleFileMergeOnReadTable(format) + .test("mergeOnRead.metadata.positionDeletes") { table => + table.spark.sql(s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + val positionDeleteCount = table.spark + .sql(s"SELECT count(*) FROM ${table.name}.position_deletes") + .collect()(0) + .getLong(0) + + assert( + positionDeleteCount == 1, + s"position_deletes exposes the one position delete, found $positionDeleteCount") + }) + + // --- 5. what a changelog scan makes of a merge-on-read history --- + + /** + * The changelog on a merge-on-read table. The append, the INSERT OVERWRITE and the row-level DELETE leave the + * change feed decodable and report exactly the rows they changed, so those three are pinned row by row. The + * UPDATE and the MERGE leave position-delete files that a changelog scan reports as unsupported, so each of those + * is pinned as a rejection. + */ + lazy val mergeOnReadChangelogCases: List[TestCase] = + fileFormats.flatMap { format => + changelogOperations + .filter(operation => decodableChangelogOperationNames.contains(operation.name)) + .map(operation => + decodableChangelogCase(preparedMergeOnReadTable(format), operation)) ++ + changelogOperations + .filterNot(operation => decodableChangelogOperationNames.contains(operation.name)) + .map(operation => + rejectedChangelogCase(preparedMergeOnReadTable(format), operation)) + } + + /** The operations whose merge-on-read change feed a changelog scan decodes, because they leave no delete file. */ + private val decodableChangelogOperationNames = + Set("changelog.append", "changelog.overwrite", "changelog.delete") + + /** The message a changelog scan reports when the range it was asked for spans position-delete files. */ + private val changelogDeleteFileRejectionMessage = "Delete files are currently not supported" + + /** + * The exact change rows each decodable operation reports on a merge-on-read table, as change type followed by the + * core columns in their declared order. Asserting the whole row pins which row the feed attributes each change + * to, so an operation that reported the right number of changes against the wrong row fails here. + */ + private val expectedChangeRowsByOperation: Map[String, List[List[Any]]] = Map( + "changelog.append" -> + List(List("INSERT", 6L, 6, "row-6", 6.5d, true, "2024-01-06-05")), + "changelog.overwrite" -> + List(List("DELETE", 3L, 3, "row-3", 3.5d, false, "2024-01-01-02")), + "changelog.delete" -> + List(List("DELETE", 1L, 1, "row-1", 1.5d, false, "2024-01-01-00"))) + + /** + * On a merge-on-read table, the operation's change feed reports exactly the rows it changed, so the write mode + * leaves the decodable part of the changelog contract as it is. + */ + private def decodableChangelogCase( + preparation: TablePreparation[CoreTable.type], + operation: ChangelogOperation): TestCase = + preparation.test(s"mergeOnRead.${operation.name}") { table => + val expectedChangeRows = expectedChangeRowsByOperation + .getOrElse( + operation.name, + throw new AssertionError(s"${operation.name} declares the change rows it reports")) + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql(operation.statement(table.name)) + val changelogView = changelogViewFrom(table, seedSnapshotId) + val actualChangeRows = table.spark + .sql( + s"SELECT _change_type, $columnNameList FROM $changelogView " + + s"ORDER BY _change_type, ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.toSeq.toList) + + assert( + actualChangeRows == expectedChangeRows, + s"${operation.name} reports $expectedChangeRows on a merge-on-read table, " + + s"found $actualChangeRows") + assert( + changeCounts(table, changelogView) == operation.expectedChangeCounts, + s"${operation.name} agrees with the shared histogram " + + s"${operation.expectedChangeCounts}, found ${changeCounts(table, changelogView)}") + } + + /** + * On a merge-on-read table, reading the operation's change feed reports that delete files are unsupported, so a + * caller learns the range is undecodable and can fall back to a range the scan does decode. + */ + private def rejectedChangelogCase( + preparation: TablePreparation[CoreTable.type], + operation: ChangelogOperation): TestCase = + preparation.test(s"mergeOnRead.${operation.name}.rejected") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql(operation.statement(table.name)) + val rejection = Check.intercept[UnsupportedOperationException] { + val view = changelogViewFrom(table, seedSnapshotId) + table.spark.sql(s"SELECT * FROM $view").collect() + } + + assert( + Exceptions + .causeChain(rejection) + .exists(cause => + Option(cause.getMessage).exists(_.contains(changelogDeleteFileRejectionMessage))), + s"the rejection names delete files as unsupported, found: ${rejection.getMessage.take(200)}") + } + + // --- 6. the properties a merge-on-read write path owns --- + + /** + * Format materialization on a table that already carries a live position-delete file: the data files still carry + * the extension of the declared write.format.default, so a delete file present alongside them leaves the format + * contract as it is. The case body is the foundation's, reused as data. + */ + lazy val mergeOnReadFileFormatCases: List[TestCase] = + layoutFormatCasesFor(preparedDeletedMergeOnReadTables) + + /** + * write.delete-file-replication is the property the delete-file writer resolves into a block replication factor, so + * it applies exactly where a mutation writes a position-delete file. The property round-trips through the catalog, + * survives the DELETE that uses it, and the DELETE physically writes the delete file the property describes. The + * local catalog asserts the property and the delete file; HDFS verifies block replication in its own environment. + */ + lazy val mergeOnReadFileReplicationCases: List[TestCase] = + fileFormats.map(format => + preparedSingleFileMergeOnReadTable(format) + .test("mergeOnRead.fileReplication.deleteFileProperty") { table => + table.spark.sql( + s"ALTER TABLE ${table.name} SET TBLPROPERTIES ('write.delete-file-replication'='2')") + + assert( + tableProps(table.spark, table.name).get("write.delete-file-replication").contains("2"), + s"the delete-file replication property round-trips, found " + + s"${tableProps(table.spark, table.name).get("write.delete-file-replication")}") + + table.spark.sql(s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + + assert( + currentDeleteFileCount(table.spark, table.name) == 1, + s"the delete writes the position-delete file the property describes, found " + + s"${currentDeleteFileCount(table.spark, table.name)}") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"the delete leaves keys 2 and 3, found ${liveKeys(table.spark, table.name)}") + assert( + tableProps(table.spark, table.name).get("write.delete-file-replication").contains("2"), + "the delete-file replication property survives the delete that used it") + }) + + // --- 7. reading the history a position delete sits in --- + + /** + * Snapshot history over a live position-delete file. The delete is a commit like any other, so the snapshot before + * it still reads the removed row, and a rollback to that snapshot brings the row back into the live set. + */ + lazy val mergeOnReadHistoryCases: List[TestCase] = + preparedDeletedMergeOnReadTables.flatMap { preparation => + List( + timeTravelBeforeDeleteCase(preparation), + rollbackUndoesDeleteCase(preparation)) + } + + /** The current read applies the delete, while the snapshot before it still reads the removed row. */ + private def timeTravelBeforeDeleteCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.history.timeTravelBeforeDelete") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + val preDeleteKeys = table.spark + .sql( + s"SELECT ${Core.long0.columnName} FROM ${table.name} VERSION AS OF $seedSnapshotId " + + s"ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) + + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"the current read applies the delete, found ${liveKeys(table.spark, table.name)}") + assert( + preDeleteKeys == Seq(1L, 2L, 3L), + s"the snapshot before the delete reads the removed row, found $preDeleteKeys") + } + + /** A rollback to the snapshot before the delete brings the removed row back into the live set. */ + private def rollbackUndoesDeleteCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.history.rollbackUndoesDelete") { table => + val seedSnapshotId = snapshotIds(table.spark, table.name).head + table.spark.sql( + "CALL openhouse.system.rollback_to_snapshot(" + + s"table => '${catalogRelative(table.name)}', " + + s"snapshot_id => ${seedSnapshotId}L)") + + assert( + liveKeys(table.spark, table.name) == Seq(1L, 2L, 3L), + s"the rollback restores the position-deleted row, found ${liveKeys(table.spark, table.name)}") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadKit.scala new file mode 100644 index 000000000..fdee1636a --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadKit.scala @@ -0,0 +1,359 @@ +package harness + +/** + * The merge-on-read starting states. + * + * A merge-on-read table is format version 2 whose delete, update and merge modes are merge-on-read, so a mutation + * records position-delete files beside the data files it matched and leaves those data files in place. Copy-on-write + * rewrites the matched data file instead. That physical difference is the whole subject of this layer, so this kit + * supplies the starting states that put a table on one write path or the other and leaves the operations to the + * foundation. + * + * Several families need a table whose delete is a partial-file match, because a delete aligned with a whole data file + * is satisfied by dropping that file on either write path and the two modes become indistinguishable. The verify + * layouts seed through a single write task so all three rows land in one data file, which makes a strict-subset + * delete a partial-file match and the physical outcome deterministic in both formats. + * + * The members are lazy so they initialize on first read, after every trait mixed into `object Scenarios` has been + * constructed. + */ +trait ScenarioMergeOnReadKit extends ScenarioKit { + + /** Every merge-on-read layout: each file format crossed with each partitioning. */ + lazy val mergeOnReadLayouts: List[Layout] = + for { + format <- fileFormats + partitioning <- partitionings + } yield mergeOnReadLayout(partitioning, format) + + /** + * One merge-on-read layout per file format that pins how a mutation is written physically. It carries all three + * merge-on-read modes, so an UPDATE and a MERGE take the same write path a DELETE does, and it sets + * write.distribution-mode to none while staying unpartitioned, so a single seed INSERT lands every row in one data + * file and a strict-subset mutation is a partial-file match that Iceberg satisfies with a position delete. + */ + lazy val mergeOnReadVerifyLayouts: List[Layout] = + fileFormats.map(format => + Layout( + s"mor-verify/$format", + table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"${mergeOnReadProperties(format)}, 'write.distribution-mode'='none')")) + + /** + * The copy-on-write counterpart of `mergeOnReadVerifyLayouts`, identical except that all three modes are + * copy-on-write, so the pair isolates the write mode as the only difference between the two physical outcomes. + */ + lazy val copyOnWriteVerifyLayouts: List[Layout] = + fileFormats.map(format => + Layout( + s"cow-verify/$format", + table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"${copyOnWriteProperties(format)}, 'write.distribution-mode'='none')")) + + /** One preparation per merge-on-read layout: created, then seeded with the standard rows. */ + lazy val preparedMergeOnReadCoreTables: List[TablePreparation[CoreTable.type]] = + mergeOnReadLayouts.map(layout => + TablePreparation(layout.label, create(layout).insert(standardSeedRowCount)(), mergeOnReadCasePrefix)) + + /** The merge-on-read core preparations, each carrying one row whose string column is null. */ + lazy val preparedNullStringMergeOnReadCoreTables: List[TablePreparation[CoreTable.type]] = + preparedMergeOnReadCoreTables.map(withNullStringRow) + + /** + * One replace-lineage merge-on-read preparation per file format: the standard seed in an unpartitioned table, + * re-specified in place by CREATE OR REPLACE TABLE AS SELECT that restates the merge-on-read modes, then refreshed. + * A mutation on the result runs on replace lineage and the merge-on-read write path at once, which is the one + * direct dependency this layer has on its parent. + */ + lazy val preparedReplacedMergeOnReadCoreTables: List[TablePreparation[CoreTable.type]] = + fileFormats.map(format => + TablePreparation( + s"mor-${unpartitioned.label}/$format", + replaceLineageMergeOnRead(unpartitioned, format), + replacedMergeOnReadCasePrefix)) + + /** The replace-lineage merge-on-read preparations, each carrying one row whose string column is null. */ + lazy val preparedNullStringReplacedMergeOnReadCoreTables: List[TablePreparation[CoreTable.type]] = + preparedReplacedMergeOnReadCoreTables.map(withNullStringRow) + + /** + * One preparation per merge-on-read verify layout: three seed rows in one data file, then the row with key 1 + * deleted merge-on-read, so keys 2 and 3 remain behind a live position-delete file the reader applies at scan time. + */ + lazy val preparedDeletedMergeOnReadTables: List[TablePreparation[CoreTable.type]] = + preparedDeletedTables(mergeOnReadVerifyLayouts, deletedMergeOnReadCasePrefix) + + /** The prefix that marks a case ID as running on a merge-on-read table. */ + val mergeOnReadCasePrefix: String = "prep.mor:" + + /** The prefix that marks a case ID as running on a merge-on-read table reached through a replace. */ + val replacedMergeOnReadCasePrefix: String = "prep.rtasMor:" + + /** The prefix that marks a case ID as running on a table that already carries a live position-delete file. */ + val deletedMergeOnReadCasePrefix: String = "prep.morRead:" + + // --- the layouts, seeds and starting states the merge-on-read families build on --- + + /** + * One merge-on-read layout: a format-version 2 table whose delete, update and merge modes are merge-on-read, so a + * mutation records its change in position-delete files and leaves the untouched data files in place. + */ + private def mergeOnReadLayout(partitioning: Partitioning, format: String): Layout = + Layout( + s"mor-${partitioning.label}/$format", + table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES (${mergeOnReadProperties(format)})") + + /** + * The merge-on-read table property fragment for `format`: format-version 2 with the delete, update and merge modes + * all set to merge-on-read. + */ + protected def mergeOnReadProperties(format: String): String = + s"'write.format.default'='$format', 'format-version'='2', " + + "'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read', " + + "'write.merge.mode'='merge-on-read'" + + /** + * The copy-on-write table property fragment for `format`: format-version 2 with the delete, update and merge modes + * all set to copy-on-write, so a mutation rewrites the data file it matched. + */ + protected def copyOnWriteProperties(format: String): String = + s"'write.format.default'='$format', 'format-version'='2', " + + "'write.delete.mode'='copy-on-write', 'write.update.mode'='copy-on-write', " + + "'write.merge.mode'='copy-on-write'" + + /** The three properties that decide which write path a mutation takes. */ + val writeModePropertyNames: List[String] = + List("write.delete.mode", "write.update.mode", "write.merge.mode") + + /** + * Creates the table under `layout`, then seeds the standard rows through a single write task so they land in one + * data file. The COALESCE(1) hint is what forces the single file, which keeps a strict-subset delete a partial-file + * match: merge-on-read writes a position delete for it, and copy-on-write rewrites the data file. + */ + protected def singleFileSeed(layout: Layout): TableTest[CoreTable.type] = + create(layout) + .sql(s"seed($standardSeedRowCount, one-file)")(table => + s"INSERT INTO $table SELECT /*+ COALESCE(1) */ * FROM " + + s"(${RowGenerator.valuesClause(Core, standardSeedRowCount)}) AS seed")(view => + assert( + view.after.size == standardSeedRowCount, + s"the single-file seed lands $standardSeedRowCount rows, found ${view.after.size}")) + + /** + * Seeds the standard rows into one data file, then deletes the row with key 1. The table holds keys 2 and 3 behind + * a live position-delete file, which is the state every coexistence family starts from and the one a delete file + * makes reachable. + */ + protected def deletedMergeOnReadLineage(layout: Layout): TableTest[CoreTable.type] = + singleFileSeed(layout) + .step("prep.morDelete")((spark, table) => + spark.sql(s"DELETE FROM $table WHERE ${Core.long0.columnName} = 1"))(view => { + assert( + view.after.size == standardSeedRowCount - 1, + s"the preparation delete leaves ${standardSeedRowCount - 1} rows, found ${view.after.size}") + assert( + currentDeleteFileCount(view.spark, view.table) == 1, + s"the preparation leaves one live position-delete file, found " + + s"${currentDeleteFileCount(view.spark, view.table)}") + }) + + /** One preparation per layout given: three seed rows in one data file, with key 1 deleted merge-on-read. */ + protected def preparedDeletedTables( + layouts: List[Layout], + casePrefix: String): List[TablePreparation[CoreTable.type]] = + layouts.map(layout => + TablePreparation(layout.label, deletedMergeOnReadLineage(layout), casePrefix)) + + /** + * The number of delete files the table's current snapshot references, which is what a reader applies at scan time. + * Every assertion about the table as it stands now reads this. + */ + protected def currentDeleteFileCount( + spark: org.apache.spark.sql.SparkSession, + table: String): Long = + spark.sql(s"SELECT count(*) FROM $table.delete_files").collect()(0).getLong(0) + + /** + * The snapshot the table's main branch currently reads from, read from the refs metadata table, which names exactly + * one snapshot per branch. + */ + protected def currentSnapshotId( + spark: org.apache.spark.sql.SparkSession, + table: String): Long = + spark + .sql(s"SELECT snapshot_id FROM $table.refs WHERE name = 'main'") + .collect() + .toSeq + .map(_.getLong(0)) match { + case Seq(snapshotId) => snapshotId + case mainSnapshotIds => + throw new AssertionError(s"main names one snapshot, found $mainSnapshotIds") + } + + /** The snapshot IDs the table still retains. */ + protected def retainedSnapshotIds( + spark: org.apache.spark.sql.SparkSession, + table: String): Seq[Long] = + spark + .sql(s"SELECT snapshot_id FROM $table.snapshots") + .collect() + .toSeq + .map(_.getLong(0)) + + /** The manifest paths the table's current snapshot references, for the given manifest content code. */ + protected def currentManifestPaths( + spark: org.apache.spark.sql.SparkSession, + table: String, + manifestContent: Int): Set[String] = + spark + .sql(s"SELECT path FROM $table.manifests WHERE content = $manifestContent") + .collect() + .toSeq + .map(_.getString(0)) + .toSet + + /** The manifest content code for the manifests that list data files. */ + protected val dataManifestContent: Int = 0 + + /** The manifest content code for the manifests that list delete files. */ + protected val deleteManifestContent: Int = 1 + + /** + * The data-file paths the table's current snapshot references. The `files` metadata table lists delete files + * alongside data files, so the content code selects the data files on their own. + */ + protected def currentDataFilePaths( + spark: org.apache.spark.sql.SparkSession, + table: String): Set[String] = + spark + .sql(s"SELECT file_path FROM $table.files WHERE content = $dataFileContent") + .collect() + .toSeq + .map(_.getString(0)) + .toSet + + /** The number of data files the table's current snapshot references. */ + protected def currentDataFileCount( + spark: org.apache.spark.sql.SparkSession, + table: String): Long = + currentDataFilePaths(spark, table).size.toLong + + /** The file content code for a data file, as the `files` metadata table reports it. */ + protected val dataFileContent: Int = 0 + + /** The persisted value of `propertyName`, which is what the table is actually configured with. */ + protected def persistedProperty( + spark: org.apache.spark.sql.SparkSession, + table: String, + propertyName: String): Option[String] = + tableProps(spark, table).get(propertyName) + + /** The live keys the table reads back, in key order, with every position delete applied. */ + protected def liveKeys(spark: org.apache.spark.sql.SparkSession, table: String): Seq[Long] = + spark + .sql(s"SELECT ${Core.long0.columnName} FROM $table ORDER BY ${Core.long0.columnName}") + .collect() + .toSeq + .map(_.getLong(0)) + + /** The standard seed written as one data file in a merge-on-read table in `format`. */ + protected def preparedSingleFileMergeOnReadTable( + format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + singleFileSeed( + Layout( + format, + table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"${mergeOnReadProperties(format)})"))) + + /** The standard seed written as one data file in a copy-on-write table in `format`. */ + protected def preparedSingleFileCopyOnWriteTable( + format: String): TablePreparation[CoreTable.type] = + TablePreparation( + format, + singleFileSeed( + Layout( + format, + table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource TBLPROPERTIES (" + + s"${copyOnWriteProperties(format)})"))) + + /** The standard seed in a merge-on-read table in `format`, labelled so its IDs name the write mode they ran on. */ + protected def preparedMergeOnReadTable(format: String): TablePreparation[CoreTable.type] = + TablePreparation( + s"mor/$format", + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource " + + s"TBLPROPERTIES (${mergeOnReadProperties(format)})")() + .insert(standardSeedRowCount)()) + + /** + * Creates a replace-lineage merge-on-read table: the standard seed, re-specified in place by CREATE OR REPLACE + * TABLE AS SELECT restating the merge-on-read modes, then refreshed so the Spark session reads the committed + * metadata pointer. Each step validates the state it leaves, so a mutation case that runs on the result starts from + * a known baseline. + */ + private def replaceLineageMergeOnRead( + partitioning: Partitioning, + format: String): TableTest[CoreTable.type] = + TableTest(Core) + .sql("create")(table => + s"CREATE TABLE $table ($columnDefinitions) USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES (${mergeOnReadProperties(format)}, 'replace.enabled'='true')")() + .insert(standardSeedRowCount)() + .sql("prep.rtasMor")(table => + s"CREATE OR REPLACE TABLE $table USING $dataSource ${partitioning.clause} " + + s"TBLPROPERTIES (${mergeOnReadProperties(format)}) AS SELECT * FROM $table")(view => { + assertSeededMergeOnReadShape(view, "prep.rtasMor") + assert( + view.snapshotsAfter == view.snapshotsBefore + 1, + s"prep.rtasMor commits one snapshot, went from ${view.snapshotsBefore} to " + + s"${view.snapshotsAfter}") + }) + .sql("prep.rtasMor.refresh")(table => s"REFRESH TABLE $table")(view => { + assertSeededMergeOnReadShape(view, "prep.rtasMor.refresh") + assert( + view.snapshotsAfter == view.snapshotsBefore, + s"prep.rtasMor.refresh reads committed metadata and commits nothing, went from " + + s"${view.snapshotsBefore} to ${view.snapshotsAfter} snapshots") + }) + + /** + * The state both replace-lineage steps leave behind: the standard seed rows in key order, unchanged by the step, + * under exactly the core columns, on a table still configured merge-on-read. Asserting it here means a mutation + * case always compares against a known baseline. + */ + private def assertSeededMergeOnReadShape( + view: StepView[CoreTable.type], + stepLabel: String): Unit = { + val schemaColumnNames = view.spark.table(view.table).schema.fieldNames.toSeq + val configuredWriteModes = writeModePropertyNames.map(propertyName => + propertyName -> persistedProperty(view.spark, view.table, propertyName)) + + assert( + schemaColumnNames == Core.columnNames, + s"$stepLabel presents the core schema, found $schemaColumnNames") + assert( + view.after == view.before, + s"$stepLabel keeps every row it started from, went from ${view.before} to ${view.after}") + assert( + view.after.size == standardSeedRowCount, + s"$stepLabel holds the $standardSeedRowCount standard seed rows, found ${view.after.size}") + assert( + view.after.map(row => Rows.TypedRow(row).get(Core.long0)) == + (1L to standardSeedRowCount.toLong).toList, + s"$stepLabel holds the standard seed keys, found " + + s"${view.after.map(row => Rows.TypedRow(row).get(Core.long0))}") + assert( + configuredWriteModes == writeModePropertyNames.map(_ -> Some("merge-on-read")), + s"$stepLabel keeps every write mode on the merge-on-read path, found $configuredWriteModes") + } + +} diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadMaintenance.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadMaintenance.scala new file mode 100644 index 000000000..9821a0162 --- /dev/null +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadMaintenance.scala @@ -0,0 +1,444 @@ +package harness + +import java.time.LocalDateTime +import java.time.format.DateTimeFormatter +import java.util.concurrent.TimeUnit + +import org.apache.hadoop.fs.{FileSystem, Path} + +/** + * Maintenance over a live position-delete file. + * + * Each maintenance procedure has to decide whether to fold the delete into the data it rewrites, carry it forward, or + * leave it in place. All of them keep the row the delete removed out of the live row set, so a table that accumulates + * delete files stays maintainable and a maintenance run stays safe to schedule. + * + * Every family reads the procedure's own report of what it rewrote, expired or removed, so each case proves the + * effect it is named for and not only that the rows survived it. + * + * Operations: rewrite_data_files, rewrite_position_delete_files, expire_snapshots, rewrite_manifests, + * remove_orphan_files against an orphan the case plants and backdates itself, and a compaction followed by an + * expiration whose two effects are proven separately. + * + * Preparation axes: the two merge-on-read verify layouts, each seeded into one data file with key 1 deleted, for the + * six families that start from a live delete; and the single-file merge-on-read table in each format for the family + * that writes its own delete first. + * + * Case families: seven families contributing 14 cases, 12 on the two deleted preparations and 2 on the self-deleting + * one. + */ +trait ScenarioMergeOnReadMaintenance extends ScenarioMergeOnReadKit { + + /** Every merge-on-read maintenance case, one deleted preparation at a time, then the self-deleting family. */ + lazy val mergeOnReadMaintenanceCases: List[TestCase] = + preparedDeletedMergeOnReadTables.flatMap { preparation => + List( + rewriteDataFilesLeavesDanglingDeleteCase(preparation), + rewritePositionDeleteFilesFoldsDanglingDeleteCase(preparation), + expireSnapshotsKeepsDeleteCase(preparation), + rewriteManifestsKeepsDeleteCase(preparation), + removeOrphanFilesKeepsDeleteCase(preparation), + compactThenExpireKeepsDeleteCase(preparation)) + } ++ fileFormats.map(format => + rewritePositionDeleteFilesCompactsCase(preparedSingleFileMergeOnReadTable(format))) + + // --- the case bodies the surface above composes --- + + /** + * rewrite_data_files folds the live delete into the data it compacts, so the deleted key stays gone and the two live + * rows read back, while the position-delete file it superseded stays referenced until a later procedure clears it. + */ + private def rewriteDataFilesLeavesDanglingDeleteCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.maintenance.rewriteDataFilesFoldsDelete") { table => + assert( + currentDeleteFileCount(table.spark, table.name) == 1, + s"the preparation leaves the delete this compaction folds, found " + + s"${currentDeleteFileCount(table.spark, table.name)}") + + val dataFilePathsBefore = currentDataFilePaths(table.spark, table.name) + val rewriteReport = table.spark + .sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + .collect()(0) + val dataFilePathsAfter = currentDataFilePaths(table.spark, table.name) + + assert( + rewriteReport.getInt(0) == dataFilePathsBefore.size, + s"the compaction rewrites the ${dataFilePathsBefore.size} data files it started from, " + + s"rewrote ${rewriteReport.getInt(0)}") + assert( + rewriteReport.getInt(1) == dataFilePathsAfter.size, + s"the compaction adds the ${dataFilePathsAfter.size} data files it left behind, added " + + s"${rewriteReport.getInt(1)}") + assert( + dataFilePathsAfter.intersect(dataFilePathsBefore).isEmpty, + s"every data file the compaction rewrote leaves the current set, " + + s"${dataFilePathsAfter.intersect(dataFilePathsBefore)} stayed") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"compaction folds the delete and keeps the live rows, found " + + s"${liveKeys(table.spark, table.name)}") + } + + /** + * rewrite_position_delete_files after a compaction clears the position-delete file the compaction superseded, and + * the live row set stays as it was, so the two procedures together return the table to a delete-free state. + */ + private def rewritePositionDeleteFilesFoldsDanglingDeleteCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.maintenance.rewritePositionDeletesClearsDangling") { table => + table.spark.sql( + "CALL openhouse.system.rewrite_data_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + val danglingDeleteFileCount = currentDeleteFileCount(table.spark, table.name) + val rewriteReport = table.spark + .sql( + "CALL openhouse.system.rewrite_position_delete_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + .collect()(0) + + assert( + danglingDeleteFileCount >= 1, + s"the compaction leaves the delete file this call clears, found $danglingDeleteFileCount") + assert( + rewriteReport.getInt(0) == danglingDeleteFileCount, + s"the call rewrites the $danglingDeleteFileCount delete files it found, rewrote " + + s"${rewriteReport.getInt(0)}") + assert( + rewriteReport.getInt(1) == currentDeleteFileCount(table.spark, table.name), + s"the call adds the ${currentDeleteFileCount(table.spark, table.name)} delete files it " + + s"left behind, added ${rewriteReport.getInt(1)}") + assert( + currentDeleteFileCount(table.spark, table.name) == 0, + s"the folded delete is cleared, found " + + s"${currentDeleteFileCount(table.spark, table.name)} delete files") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"clearing the folded delete keeps the live rows, found " + + s"${liveKeys(table.spark, table.name)}") + } + + /** + * expire_snapshots drops every snapshot the table no longer needs to retain and keeps the one it currently reads + * from, so the history shrinks to the retained snapshot while the live rows and the delete file the reader applies + * stay exactly as they were. + */ + private def expireSnapshotsKeepsDeleteCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.maintenance.expireSnapshotsKeepsDelete") { table => + val snapshotIdsBefore = retainedSnapshotIds(table.spark, table.name) + val currentSnapshotIdBefore = currentSnapshotId(table.spark, table.name) + val deleteFileCountBefore = currentDeleteFileCount(table.spark, table.name) + + assert( + snapshotIdsBefore.size >= 2, + s"the preparation leaves history for the expiration to drop, found $snapshotIdsBefore") + + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + s"older_than => TIMESTAMP '$expirationCutoff', " + + "retain_last => 1)") + val snapshotIdsAfter = retainedSnapshotIds(table.spark, table.name) + + assert( + snapshotIdsAfter == Seq(currentSnapshotIdBefore), + s"the expiration retains the snapshot the table reads from and drops the rest, " + + s"went from $snapshotIdsBefore to $snapshotIdsAfter") + assert( + snapshotIdsBefore.filterNot(_ == currentSnapshotIdBefore).forall(expiredSnapshotId => + !snapshotIdsAfter.contains(expiredSnapshotId)), + s"every superseded snapshot is gone, found $snapshotIdsAfter") + assert( + currentSnapshotId(table.spark, table.name) == currentSnapshotIdBefore, + "the expiration leaves the table reading from the snapshot it was already on") + assert( + currentDeleteFileCount(table.spark, table.name) == deleteFileCountBefore, + s"the expiration keeps the $deleteFileCountBefore delete files the reader applies, found " + + s"${currentDeleteFileCount(table.spark, table.name)}") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"expiration keeps the delete applied, found ${liveKeys(table.spark, table.name)}") + } + + /** + * rewrite_manifests replaces the manifests the table references with the ones it wrote, and the data files, delete + * files and live rows the manifests point at stay exactly as they were. + */ + private def rewriteManifestsKeepsDeleteCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.maintenance.rewriteManifestsKeepsDelete") { table => + // Each append commits its own data manifest, so the rewrite has several to merge into one. + val appendedKeys = List(4L, 5L, 6L) + appendedKeys.foreach(key => + table.spark.sql(s"INSERT INTO ${table.name} VALUES ${coreRow(key, s"row-$key")}")) + + // rewrite_manifests rewrites the data manifests, so the data manifests present beforehand are the eligible set + // and the delete manifests are the part it is expected to leave alone. + val dataManifestPathsBefore = + currentManifestPaths(table.spark, table.name, dataManifestContent) + val deleteManifestPathsBefore = + currentManifestPaths(table.spark, table.name, deleteManifestContent) + val dataFileCountBefore = currentDataFileCount(table.spark, table.name) + val deleteFileCountBefore = currentDeleteFileCount(table.spark, table.name) + val rewriteReport = table.spark + .sql( + "CALL openhouse.system.rewrite_manifests(" + + s"table => '${catalogRelative(table.name)}')") + .collect()(0) + val dataManifestPathsAfter = + currentManifestPaths(table.spark, table.name, dataManifestContent) + + assert( + dataManifestPathsBefore.size >= 2, + s"the appends leave several data manifests for the rewrite to merge, found " + + s"${dataManifestPathsBefore.size}") + assert( + rewriteReport.getInt(0) == dataManifestPathsBefore.size, + s"the call rewrites the ${dataManifestPathsBefore.size} data manifests it started from, " + + s"rewrote ${rewriteReport.getInt(0)}") + assert( + rewriteReport.getInt(1) == dataManifestPathsAfter.size, + s"the call adds the ${dataManifestPathsAfter.size} data manifests it left behind, added " + + s"${rewriteReport.getInt(1)}") + assert( + dataManifestPathsAfter.intersect(dataManifestPathsBefore).isEmpty, + s"every data manifest the rewrite merged leaves the current set, " + + s"${dataManifestPathsAfter.intersect(dataManifestPathsBefore)} stayed") + assert( + currentManifestPaths(table.spark, table.name, deleteManifestContent) == + deleteManifestPathsBefore, + s"the rewrite leaves the delete manifests as they were, found " + + s"${currentManifestPaths(table.spark, table.name, deleteManifestContent)}") + assert( + currentDataFileCount(table.spark, table.name) == dataFileCountBefore, + s"manifest rewriting keeps the $dataFileCountBefore data files, found " + + s"${currentDataFileCount(table.spark, table.name)}") + assert( + currentDeleteFileCount(table.spark, table.name) == deleteFileCountBefore, + s"manifest rewriting keeps the $deleteFileCountBefore delete files, found " + + s"${currentDeleteFileCount(table.spark, table.name)}") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L) ++ appendedKeys, + s"manifest rewriting keeps the delete applied, found ${liveKeys(table.spark, table.name)}") + } + + /** + * remove_orphan_files removes exactly the unreferenced file the case plants under the table's data directory and + * leaves every referenced data file, delete file and row in place. + * + * The case owns the orphan end to end: it writes the file itself, backdates its modification time behind the + * cutoff through the Hadoop FileSystem the table's own path resolves to, asserts the procedure reports that one + * location, and deletes the orphan on the way out if the procedure left it. The table's referenced files were + * written moments ago, so they sit ahead of the cutoff and are outside the removal window, which is what makes + * "exactly the orphan" a real assertion. + * + * Locations are compared as fully qualified paths resolved through the same filesystem, so the scheme and + * authority the procedure reports line up with the planted path on local storage and on HDFS alike. + */ + private def removeOrphanFilesKeepsDeleteCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.maintenance.removeOrphanFilesRemovesTheOrphan") { table => + val dataFileCountBefore = currentDataFileCount(table.spark, table.name) + val deleteFileCountBefore = currentDeleteFileCount(table.spark, table.name) + val fileSystem = tableFileSystem(table) + val orphanPath = plantBackdatedOrphanFile(table, fileSystem) + + OwnedTableLifecycle.withCleanup( + if (fileSystem.exists(orphanPath)) { + assert( + fileSystem.delete(orphanPath, false), + s"the case removes the orphan it planted at $orphanPath") + }) { + val removedPaths = table.spark + .sql( + "CALL openhouse.system.remove_orphan_files(" + + s"table => '${catalogRelative(table.name)}', " + + s"older_than => TIMESTAMP '${orphanRemovalCutoffTimestamp()}')") + .collect() + .toSeq + .map(row => qualified(fileSystem, new Path(row.getString(0)))) + + assert( + removedPaths == Seq(orphanPath), + s"the call removes exactly the planted orphan $orphanPath, removed $removedPaths") + assert( + !fileSystem.exists(orphanPath), + s"the removed orphan is gone from storage, $orphanPath is still there") + assert( + currentDataFileCount(table.spark, table.name) == dataFileCountBefore, + s"orphan removal keeps the $dataFileCountBefore referenced data files, found " + + s"${currentDataFileCount(table.spark, table.name)}") + assert( + currentDeleteFileCount(table.spark, table.name) == deleteFileCountBefore, + s"orphan removal keeps the $deleteFileCountBefore referenced delete files, found " + + s"${currentDeleteFileCount(table.spark, table.name)}") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"orphan removal keeps the delete applied, found ${liveKeys(table.spark, table.name)}") + } + } + + /** The filesystem the table's own data files resolve through, which is the one the procedure reports against. */ + private def tableFileSystem(table: PreparedTable[CoreTable.type]): FileSystem = + referencedDataFilePath(table).getFileSystem(table.spark.sessionState.newHadoopConf()) + + /** One data-file path the table's current snapshot references. */ + private def referencedDataFilePath(table: PreparedTable[CoreTable.type]): Path = + new Path( + table.spark + .sql(s"SELECT file_path FROM ${table.name}.files LIMIT 1") + .collect()(0) + .getString(0)) + + /** + * The fully qualified form of `path` on `fileSystem`, carrying its scheme and authority. Comparing qualified paths + * keeps the assertion correct whether the procedure reports a bare path, a `file:` URI or an `hdfs:` URI. + */ + private def qualified(fileSystem: FileSystem, path: Path): Path = + path.makeQualified(fileSystem.getUri, fileSystem.getWorkingDirectory) + + /** + * Writes an unreferenced file beside the table's data files and backdates it well behind the removal cutoff, then + * returns its qualified path. Placing it beside a referenced data file puts it inside the directory tree the + * procedure scans, and it is unreferenced because no manifest names it. + */ + private def plantBackdatedOrphanFile( + table: PreparedTable[CoreTable.type], + fileSystem: FileSystem): Path = { + val orphanPath = qualified( + fileSystem, + new Path(referencedDataFilePath(table).getParent, "harness-planted-orphan.parquet")) + val backdatedModificationTime = + System.currentTimeMillis() - TimeUnit.DAYS.toMillis(orphanAgeDays) + + fileSystem.create(orphanPath, true).close() + fileSystem.setTimes(orphanPath, backdatedModificationTime, -1L) + + assert( + fileSystem.getFileStatus(orphanPath).getModificationTime == backdatedModificationTime, + s"the planted orphan carries the backdated modification time the cutoff is measured " + + s"against, found ${fileSystem.getFileStatus(orphanPath).getModificationTime} for " + + s"$backdatedModificationTime") + orphanPath + } + + /** How far behind the present the planted orphan's modification time sits. */ + private val orphanAgeDays = 30L + + /** How far behind the present the removal cutoff sits, which the procedure requires to exceed 24 hours. */ + private val orphanRemovalCutoffDays = 7L + + /** + * The older_than cutoff for orphan removal: far enough back that the procedure accepts it and the table's + * just-written files sit ahead of it, and recent enough that the planted orphan sits behind it. + */ + private def orphanRemovalCutoffTimestamp(): String = + LocalDateTime + .now() + .minusDays(orphanRemovalCutoffDays) + .format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")) + + /** + * The pair a scheduled maintenance run issues together, with each half proven on its own: the compaction rewrites + * the delete files it found, then the expiration drops the history that compaction superseded and leaves the table + * reading from one snapshot. The live rows are the same at the end as at the start. + */ + private def compactThenExpireKeepsDeleteCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.maintenance.compactThenExpireKeepsDelete") { table => + val deleteFileCountBefore = currentDeleteFileCount(table.spark, table.name) + val compactionReport = table.spark + .sql( + "CALL openhouse.system.rewrite_position_delete_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + .collect()(0) + + assert( + deleteFileCountBefore >= 1, + s"the preparation leaves the delete files this compaction rewrites, found $deleteFileCountBefore") + assert( + compactionReport.getInt(0) == deleteFileCountBefore, + s"the compaction rewrites the $deleteFileCountBefore delete files it found, rewrote " + + s"${compactionReport.getInt(0)}") + assert( + compactionReport.getInt(1) == currentDeleteFileCount(table.spark, table.name), + s"the compaction adds the ${currentDeleteFileCount(table.spark, table.name)} delete " + + s"files it left behind, added ${compactionReport.getInt(1)}") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"the compaction keeps the delete applied, found ${liveKeys(table.spark, table.name)}") + + val snapshotIdsBeforeExpiration = retainedSnapshotIds(table.spark, table.name) + val currentSnapshotIdBeforeExpiration = currentSnapshotId(table.spark, table.name) + table.spark.sql( + "CALL openhouse.system.expire_snapshots(" + + s"table => '${catalogRelative(table.name)}', " + + s"older_than => TIMESTAMP '$expirationCutoff', " + + "retain_last => 1)") + val snapshotIdsAfterExpiration = retainedSnapshotIds(table.spark, table.name) + + assert( + snapshotIdsBeforeExpiration.size >= 2, + s"the compaction leaves history for the expiration to drop, found $snapshotIdsBeforeExpiration") + assert( + snapshotIdsAfterExpiration == Seq(currentSnapshotIdBeforeExpiration), + s"the expiration retains the compacted snapshot and drops the rest, went from " + + s"$snapshotIdsBeforeExpiration to $snapshotIdsAfterExpiration") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"compaction followed by expiration keeps the delete applied, found " + + s"${liveKeys(table.spark, table.name)}") + } + + /** + * The older_than cutoff for snapshot expiration. It is far ahead of any snapshot the harness commits, so every + * snapshot outside the retained one is inside the expiration window and the call has real work to do. + */ + private val expirationCutoff = "2999-01-01 00:00:00" + + /** + * A merge-on-read DELETE writes one position-delete file, and rewrite_position_delete_files compacts it while the + * two surviving rows stay readable, so the procedure is available to a table that accumulates delete files. + */ + private def rewritePositionDeleteFilesCompactsCase( + preparation: TablePreparation[CoreTable.type]): TestCase = + preparation.test("mergeOnRead.maintenance.rewritePositionDeleteFiles") { table => + assert( + persistedProperty(table.spark, table.name, "write.delete.mode").contains("merge-on-read"), + "the table persists write.delete.mode as merge-on-read before the delete under test") + + table.spark.sql(s"DELETE FROM ${table.name} WHERE ${Core.long0.columnName} = 1") + + assert( + currentDeleteFileCount(table.spark, table.name) == 1, + s"the merge-on-read delete writes one position-delete file, found " + + s"${currentDeleteFileCount(table.spark, table.name)}") + + val rewriteReport = table.spark + .sql( + "CALL openhouse.system.rewrite_position_delete_files(" + + s"table => '${catalogRelative(table.name)}', " + + "options => map('rewrite-all', 'true'))") + .collect()(0) + + assert( + rewriteReport.getInt(0) == 1, + s"the call rewrites the one delete file it found, rewrote ${rewriteReport.getInt(0)}") + assert( + rewriteReport.getInt(1) == currentDeleteFileCount(table.spark, table.name), + s"the call adds the ${currentDeleteFileCount(table.spark, table.name)} delete files it " + + s"left behind, added ${rewriteReport.getInt(1)}") + assert( + liveKeys(table.spark, table.name) == Seq(2L, 3L), + s"compacting the position deletes keeps the live rows, found " + + s"${liveKeys(table.spark, table.name)}") + } + +} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/MergeOnReadCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/MergeOnReadCatalogTest.scala new file mode 100644 index 000000000..c0077e061 --- /dev/null +++ b/integrations/spark/delta-harness/src/test/scala/harness/MergeOnReadCatalogTest.scala @@ -0,0 +1,250 @@ +package harness + +import java.nio.charset.StandardCharsets +import java.security.MessageDigest + +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} +import org.junit.jupiter.api.Test + +/** + * Pins the merge-on-read contribution this layer adds: its exact size, the fingerprint of its case IDs, the + * preparation axes its DML runs on, the reusable operations it covers, the merge-on-read contracts it claims to + * prove, and its skip metadata. + * + * Every assertion reads `mergeOnReadCases` and the merge-on-read families alone, so the frozen foundation and the + * replace layer keep their own tests and a sibling layer that adds its own contribution leaves this file as it is. + * The catalog invariants that apply to any layer, namely ID uniqueness, alphabetical contribution ordering and the + * foundation-plus-extensions rule, are pinned once in CaseCatalogTest. + * + * Reading the catalog builds the case list only; executing a case and starting Spark stay separate steps. + */ +final class MergeOnReadCatalogTest { + private val expectedMergeOnReadCaseCount = 320 + private val expectedMergeOnReadSha256 = + "cf7b880c9c8a92bd9a7e6489224e9a80e46080bc20457fa1dc1499d589346a2b" + + /** The merge-on-read preparations the row-mutating operations run on, in the order the layer builds them. */ + private val expectedMergeOnReadPreparationLabels = List( + "mor-unpartitioned/parquet", + "mor-partitioned/parquet", + "mor-unpartitioned/orc", + "mor-partitioned/orc") + + /** The replace-lineage merge-on-read preparations, which are this layer's one dependency on the replace layer. */ + private val expectedReplacedPreparationLabels = List( + "mor-unpartitioned/parquet", + "mor-unpartitioned/orc") + + /** The preparations that already carry a live position-delete file. */ + private val expectedDeletedPreparationLabels = List("mor-verify/parquet", "mor-verify/orc") + + /** + * Every merge-on-read contract this layer claims to prove, named by the case ID that proves it, in the order the + * layer integrates them. The list is written out here as its own literal, so the layer and this list are + * independent statements of the same coverage and any drop, rename or reorder fails this test until the intended + * coverage is restated. + */ + private val expectedContractCaseNames = List( + "mergeOnRead.deleteFile.writesDeleteFile", + "mergeOnRead.deleteFile.copyOnWriteRewritesDataFile", + "mergeOnRead.deleteMode.alterToMergeOnRead", + "mergeOnRead.coexistence.append", + "mergeOnRead.coexistence.secondDelete", + "mergeOnRead.coexistence.update", + "mergeOnRead.coexistence.filteredRead", + "mergeOnRead.coexistence.compactDeletes", + "mergeOnRead.coexistence.merge", + "mergeOnRead.metadata.positionDeletes", + "mergeOnRead.changelog.append", + "mergeOnRead.changelog.overwrite", + "mergeOnRead.changelog.delete", + "mergeOnRead.changelog.update.rejected", + "mergeOnRead.changelog.merge.rejected", + "format.materialization", + "mergeOnRead.fileReplication.deleteFileProperty", + "mergeOnRead.history.timeTravelBeforeDelete", + "mergeOnRead.history.rollbackUndoesDelete") + + /** Every maintenance contract, named by the case ID that proves it. */ + private val expectedMaintenanceCaseNames = List( + "mergeOnRead.maintenance.rewriteDataFilesFoldsDelete", + "mergeOnRead.maintenance.rewritePositionDeletesClearsDangling", + "mergeOnRead.maintenance.expireSnapshotsKeepsDelete", + "mergeOnRead.maintenance.rewriteManifestsKeepsDelete", + "mergeOnRead.maintenance.removeOrphanFilesRemovesTheOrphan", + "mergeOnRead.maintenance.compactThenExpireKeepsDelete", + "mergeOnRead.maintenance.rewritePositionDeleteFiles") + + @Test + def theMergeOnReadContributionIsTheSizeAndShapeItIsPinnedTo(): Unit = { + val caseIds = Scenarios.mergeOnReadCases.map(_.id) + val actualSha256 = sha256(caseIds.mkString("\n")) + + assertEquals( + expectedMergeOnReadCaseCount, + caseIds.size, + s"mergeOnReadCases changed; count=${caseIds.size}, sha256=$actualSha256") + assertEquals( + expectedMergeOnReadSha256, + actualSha256, + s"mergeOnReadCases changed; count=${caseIds.size}, sha256=$actualSha256") + assertEquals(caseIds.distinct.size, caseIds.size, "merge-on-read case IDs are unique") + assertEquals( + Scenarios.mergeOnReadDmlCases.map(_.id) ++ + Scenarios.mergeOnReadContractCases.map(_.id) ++ + Scenarios.mergeOnReadMaintenanceCases.map(_.id), + caseIds, + "mergeOnReadCases is the DML axis, then the contract, then maintenance") + } + + @Test + def theCatalogIntegratesTheMergeOnReadContributionExactlyOnce(): Unit = { + val mergeOnReadEntries = ScenarioCatalog.extensionContributions.filter { + case (name, _) => name == "mergeOnReadCases" + } + + assertEquals( + 1, + mergeOnReadEntries.size, + s"mergeOnReadCases is integrated once, found ${mergeOnReadEntries.size} entries") + assertEquals( + Scenarios.mergeOnReadCases.map(_.id), + mergeOnReadEntries.head match { case (_, contribution) => contribution.map(_.id) }, + "the catalog integrates the very list the capability exposes") + } + + @Test + def theMergeOnReadDmlAxisIsTheWriteModePreparations(): Unit = { + assertEquals( + expectedMergeOnReadPreparationLabels, + Scenarios.preparedMergeOnReadCoreTables.map(_.label)) + assertEquals( + expectedMergeOnReadPreparationLabels, + Scenarios.preparedNullStringMergeOnReadCoreTables.map(_.label), + "the null-string preparations extend the same merge-on-read preparations") + assertEquals( + expectedReplacedPreparationLabels, + Scenarios.preparedReplacedMergeOnReadCoreTables.map(_.label)) + assertEquals( + expectedDeletedPreparationLabels, + Scenarios.preparedDeletedMergeOnReadTables.map(_.label)) + assertTrue( + Scenarios.preparedMergeOnReadCoreTables.forall( + _.casePrefix == Scenarios.mergeOnReadCasePrefix), + "every merge-on-read preparation marks its cases as running on the merge-on-read write path") + assertTrue( + Scenarios.preparedReplacedMergeOnReadCoreTables.forall( + _.casePrefix == Scenarios.replacedMergeOnReadCasePrefix), + "every replace-lineage preparation marks its cases as running on a replaced table") + assertTrue( + Scenarios.preparedDeletedMergeOnReadTables.forall( + _.casePrefix == Scenarios.deletedMergeOnReadCasePrefix), + "every deleted preparation marks its cases as running behind a live delete file") + } + + @Test + def everyDeletedPreparationReachesItsStartingStateThroughAPositionDelete(): Unit = { + Scenarios.preparedDeletedMergeOnReadTables.foreach { preparation => + assertEquals( + List("create", s"seed(${Scenarios.standardSeedRowCount}, one-file)", "prep.morDelete"), + preparation.preparation.steps.map(_.label).toList, + s"${preparation.label} creates, seeds into one file, then deletes a strict subset") + } + Scenarios.preparedReplacedMergeOnReadCoreTables.foreach { preparation => + assertEquals( + List( + "create", + s"insert(${Scenarios.standardSeedRowCount})", + "prep.rtasMor", + "prep.rtasMor.refresh"), + preparation.preparation.steps.map(_.label).toList, + s"${preparation.label} creates, seeds, replaces and refreshes in that order") + } + } + + @Test + def everyReusableOperationTheWriteModeChangesRunsOnAMergeOnReadPreparation(): Unit = { + val coveredOperationNames = Scenarios.mergeOnReadDmlCases + .map(caseId => + caseId.id + .stripPrefix(Scenarios.mergeOnReadCasePrefix) + .stripPrefix(Scenarios.replacedMergeOnReadCasePrefix) + .stripPrefix(Scenarios.deletedMergeOnReadCasePrefix) + .split(" @ ") + .head) + .distinct + .sorted + val reusableOperationNames = (Scenarios.rowMutationTestCases ++ + Scenarios.nullStringRowTestCases ++ + Scenarios.readTestCases).map(_.id).distinct.sorted + + assertEquals( + reusableOperationNames, + coveredOperationNames, + "every reusable operation the write mode changes runs on a merge-on-read table") + assertEquals(46, reusableOperationNames.size, "the reusable operation count changed") + assertEquals(176, Scenarios.mergeOnReadCoreDmlCases.size) + assertEquals(88, Scenarios.replacedMergeOnReadDmlCases.size) + assertEquals(4, Scenarios.deletedMergeOnReadDmlCases.size) + assertEquals(268, Scenarios.mergeOnReadDmlCases.size) + } + + @Test + def everyMergeOnReadContractHasACaseInEveryColumnarFormat(): Unit = { + val contractCaseNames = Scenarios.mergeOnReadContractCases + .map(_.id.split(" @ ").head.split(":").last) + .distinct + val maintenanceCaseNames = Scenarios.mergeOnReadMaintenanceCases + .map(_.id.split(" @ ").head.split(":").last) + .distinct + + assertEquals(expectedContractCaseNames, contractCaseNames, "the contract families changed") + assertEquals( + expectedMaintenanceCaseNames, + maintenanceCaseNames, + "the maintenance families changed") + assertEquals(38, Scenarios.mergeOnReadContractCases.size) + assertEquals(14, Scenarios.mergeOnReadMaintenanceCases.size) + (expectedContractCaseNames ++ expectedMaintenanceCaseNames).foreach { contractCaseName => + assertEquals( + 2, + (Scenarios.mergeOnReadContractCases ++ Scenarios.mergeOnReadMaintenanceCases) + .count(_.id.split(" @ ").head.split(":").last == contractCaseName), + s"$contractCaseName runs in both columnar formats") + } + } + + @Test + def everyMergeOnReadCaseRunsOnAColumnarFormatInTheLandingMatrix(): Unit = { + val preparationFormats = Scenarios.mergeOnReadCases + .map(_.id.split(" @ ").last) + .map(label => label.split("/").last) + .distinct + + assertEquals(List("parquet", "orc"), Scenarios.fileFormats) + assertEquals( + Scenarios.fileFormats.sorted, + preparationFormats.sorted, + s"every merge-on-read case runs on a landing-matrix format, found $preparationFormats") + } + + @Test + def theMergeOnReadSkipMetadataIsEmpty(): Unit = { + assertEquals( + List.empty[String], + Scenarios.mergeOnReadCases.collect { + case testCase if testCase.knownBugReason.nonEmpty => testCase.id + }, + "every merge-on-read case is expected to pass") + assertTrue( + Scenarios.mergeOnReadCases.forall(_.embeddedSkipReason.isEmpty), + "every merge-on-read case reaches the embedded catalog") + } + + private def sha256(value: String): String = + MessageDigest + .getInstance("SHA-256") + .digest(value.getBytes(StandardCharsets.UTF_8)) + .map(byte => f"$byte%02x") + .mkString +} From b05d4662f4fd3d73b459970f101a3a465c766045 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Tue, 1 Sep 2026 23:14:25 -0700 Subject: [PATCH 17/24] test(delta-harness): mark RTAS race bug Keep the replace-versus-append assertion limited to serializable outcomes while recording the rare lost-replace result as a known bug. Both format cases remain available for re-enablement when concurrent RTAS commits report conflicts or preserve the replace. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/openhouse/ScenarioRtas.scala | 102 +++++++++--------- .../test/scala/harness/RtasCatalogTest.scala | 4 +- 2 files changed, 56 insertions(+), 50 deletions(-) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRtas.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRtas.scala index 0816ff25d..623e452b7 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRtas.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRtas.scala @@ -940,59 +940,63 @@ trait ScenarioRtas extends ScenarioKit { this: ScenarioDml with ChangelogSupport // --- 13. a replace racing another writer --- /** - * A replace racing an INSERT settles at either the two rows the replace selected, where the replace committed last, - * or three rows, where the append landed on the replaced table. Whichever writer loses fails with a typed commit - * conflict, so a caller recognizes every way this race ends. + * A serializable replace and INSERT race settles at either the two rows the replace selected, where the replace + * committed last, or three rows, where the append landed on the replaced table. Whichever writer loses fails with a + * typed commit conflict, so a caller recognizes every valid way this race ends. */ private def replaceVersusAppendCase(preparation: TablePreparation[CoreTable.type]): TestCase = - preparation.test("rtas.concurrency.replaceVersusAppend") { table => - val outcomeByWriter = new ConcurrentHashMap[String, String]() - def writer(writerName: String, statement: String): () => Unit = () => - try { - table.spark.sql(statement) - outcomeByWriter.put(writerName, committedOutcome) - } catch { - case NonFatal(conflict) if ConcurrencySupport.isTypedCommitConflict(conflict) => - outcomeByWriter.put(writerName, conflictedOutcome) - } + preparation + .test("rtas.concurrency.replaceVersusAppend") { table => + val outcomeByWriter = new ConcurrentHashMap[String, String]() + def writer(writerName: String, statement: String): () => Unit = () => + try { + table.spark.sql(statement) + outcomeByWriter.put(writerName, committedOutcome) + } catch { + case NonFatal(conflict) if ConcurrencySupport.isTypedCommitConflict(conflict) => + outcomeByWriter.put(writerName, conflictedOutcome) + } - val threadErrors = ConcurrencySupport.runConcurrently( - Seq( - writer("replace", replaceWithKeysUpTo(table.name, 2)), - writer("append", s"INSERT INTO ${table.name} VALUES ${coreRow(30L, "row-30")}"))) - assert( - threadErrors.isEmpty, - s"both writers either commit or hit a typed commit conflict, found: $threadErrors") + val threadErrors = ConcurrencySupport.runConcurrently( + Seq( + writer("replace", replaceWithKeysUpTo(table.name, 2)), + writer("append", s"INSERT INTO ${table.name} VALUES ${coreRow(30L, "row-30")}"))) + assert( + threadErrors.isEmpty, + s"both writers either commit or hit a typed commit conflict, found: $threadErrors") - table.spark.sql(s"REFRESH TABLE ${table.name}") - val settledKeys = table.spark - .sql(s"SELECT ${Core.long0.columnName} FROM ${table.name}") - .collect() - .toSeq - .map(_.getLong(0)) - .toSet - val raceOutcome = - (outcomeByWriter.get("replace"), outcomeByWriter.get("append")) - - println(s"DIAG rtas.concurrency.replaceVersusAppend: $raceOutcome settled at $settledKeys") - raceOutcome match { - case (`committedOutcome`, `conflictedOutcome`) => - assert( - settledKeys == Set(1L, 2L), - s"a winning replace leaves the keys it selected, found $settledKeys") - case (`conflictedOutcome`, `committedOutcome`) => - assert( - settledKeys == Set(1L, 2L, 3L, 30L), - s"a winning append leaves the seed plus its row, found $settledKeys") - case (`committedOutcome`, `committedOutcome`) => - assert( - settledKeys == Set(1L, 2L) || settledKeys == Set(1L, 2L, 30L), - s"two commits leave the replaced rows, with the append included when it landed " + - s"on the replaced table, found $settledKeys") - case recordedOutcome => - throw new AssertionError( - s"one writer commits when a replace races an append, recorded $recordedOutcome") + table.spark.sql(s"REFRESH TABLE ${table.name}") + val settledKeys = table.spark + .sql(s"SELECT ${Core.long0.columnName} FROM ${table.name}") + .collect() + .toSeq + .map(_.getLong(0)) + .toSet + val raceOutcome = + (outcomeByWriter.get("replace"), outcomeByWriter.get("append")) + + println(s"DIAG rtas.concurrency.replaceVersusAppend: $raceOutcome settled at $settledKeys") + raceOutcome match { + case (`committedOutcome`, `conflictedOutcome`) => + assert( + settledKeys == Set(1L, 2L), + s"a winning replace leaves the keys it selected, found $settledKeys") + case (`conflictedOutcome`, `committedOutcome`) => + assert( + settledKeys == Set(1L, 2L, 3L, 30L), + s"a winning append leaves the seed plus its row, found $settledKeys") + case (`committedOutcome`, `committedOutcome`) => + assert( + settledKeys == Set(1L, 2L) || settledKeys == Set(1L, 2L, 30L), + s"two commits leave the replaced rows, with the append included when it landed " + + s"on the replaced table, found $settledKeys") + case recordedOutcome => + throw new AssertionError( + s"one writer commits when a replace races an append, recorded $recordedOutcome") + } } - } + .copy(knownBugReason = Some( + "A replace and append can both report successful commits while the append's snapshot wins and loses the " + + "replace, leaving the seed rows plus the appended row.")) } diff --git a/integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala index 8c62c90ad..2ac0bcf3e 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala @@ -199,9 +199,11 @@ final class RtasCatalogTest { } @Test - def theReplaceSkipMetadataIsPinnedToTheOneKnownProductBug(): Unit = { + def theReplaceSkipMetadataIsPinnedToTheKnownProductBugs(): Unit = { assertEquals( List( + "rtas.concurrency.replaceVersusAppend @ orc", + "rtas.concurrency.replaceVersusAppend @ parquet", "rtas.schema.incompatibleType.notSilentlyLossy @ orc", "rtas.schema.incompatibleType.notSilentlyLossy @ parquet"), Scenarios.rtasCases.collect { From d2aee5bb8283baf054f3dbb644a723b1bfe1cd54 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Wed, 2 Sep 2026 12:40:46 -0700 Subject: [PATCH 18/24] test(delta-harness): remove tautological checks Keep only generated ID uniqueness and the ownership and cleanup failure behaviors that can fail independently. Scenario definitions are validated by running their behavior cases instead of restating source declarations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/openhouse/ScenarioCatalog.scala | 5 +- .../test/scala/harness/CaseCatalogTest.scala | 100 +----- .../scala/harness/DmlCaseCatalogTest.scala | 289 ------------------ .../scala/harness/FoundationCatalogTest.scala | 122 -------- .../PublicSurfaceCompatibilityTest.scala | 103 ------- .../scala/harness/SupportContractTest.scala | 130 -------- .../scala/harness/TableLifecycleTest.scala | 267 ++-------------- .../scala/harness/TablePreparationTest.scala | 77 ----- .../test/scala/harness/TableTestTest.scala | 86 ------ 9 files changed, 28 insertions(+), 1151 deletions(-) delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/FoundationCatalogTest.scala delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/PublicSurfaceCompatibilityTest.scala delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/SupportContractTest.scala delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala index 9cb046b91..c2e8a5d1c 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala @@ -45,10 +45,7 @@ object Scenarios */ object ScenarioCatalog { - /** - * The frozen foundation: the reusable DDL and DML capabilities this branch landed, named once, in alphabetical - * order. FoundationCatalogTest pins this list, so a later layer adds to `extensionContributions` instead. - */ + /** The reusable DDL and DML capabilities in the foundation, named once in alphabetical order. */ def foundationContributions: List[(String, List[TestCase])] = List( "dataTypeCases" -> Scenarios.dataTypeCases, diff --git a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala index 9ff3ef5b1..751fa1530 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala @@ -1,105 +1,21 @@ package harness -import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} +import org.junit.jupiter.api.Assertions.assertTrue import org.junit.jupiter.api.Test -/** - * Pins the rules every integrated scenario set obeys: IDs are unique, the catalog is exactly - * the foundation plus the extensions it names, it is those contributions concatenated in order, contributions are - * named once and integrated alphabetically, every contribution supplies cases, and every case ID names its capability. - * - * This extension-stable test holds structural invariants while FoundationCatalogTest pins the exact set, size and - * fingerprint of this branch's frozen foundation. Each later layer pins its contributions in a focused test. Reading - * the catalog is a Spark-free operation. - */ final class CaseCatalogTest { - - /** - * Case-ID prefixes from the old provenance buckets. Every current case ID is owned by the capability trait that - * defines it. - */ - private val provenanceCaseIdPrefixes = - List("fork.", "hazard.", "readerWriter.", "surface.", "interact.") - @Test def everyCaseIdIsUnique(): Unit = { - val caseIds = ScenarioCatalog.caseIds - val duplicateCaseIds = caseIds.groupBy(identity).collect { - case (caseId, occurrences) if occurrences.size > 1 => caseId - }.toList.sorted + val duplicateCaseIds = ScenarioCatalog.caseIds + .groupBy(identity) + .collect { + case (caseId, occurrences) if occurrences.size > 1 => caseId + } + .toList + .sorted assertTrue( duplicateCaseIds.isEmpty, s"case IDs must be unique; duplicates=${duplicateCaseIds.mkString(", ")}") } - - @Test - def everyCaseIdNamesTheCapabilityItCovers(): Unit = { - val provenanceNamedCaseIds = - ScenarioCatalog.caseIds.filter(caseId => provenanceCaseIdPrefixes.exists(caseId.startsWith)) - - assertTrue( - provenanceNamedCaseIds.isEmpty, - "a case ID names a capability, not the bucket it came from; " + - s"offenders=${provenanceNamedCaseIds.mkString(", ")}") - } - - @Test - def eachCapabilityContributesExactlyOnceInAlphabeticalOrder(): Unit = { - val contributionNames = ScenarioCatalog.contributions.map { case (name, _) => name } - - assertEquals( - contributionNames.distinct, - contributionNames, - s"a capability contribution is integrated more than once: $contributionNames") - assertEquals( - contributionNames.sorted, - contributionNames, - s"capability contributions are integrated in alphabetical order: $contributionNames") - assertTrue( - ScenarioCatalog.contributions.forall { case (_, contribution) => contribution.nonEmpty }, - "every named contribution supplies at least one case") - } - - @Test - def theCatalogIsExactlyTheFoundationAndTheExtensionsItNames(): Unit = { - val foundationNames = ScenarioCatalog.foundationContributions.map { case (name, _) => name } - val extensionNames = ScenarioCatalog.extensionContributions.map { case (name, _) => name } - val integratedNames = ScenarioCatalog.contributions.map { case (name, _) => name } - val declaredCases = (ScenarioCatalog.foundationContributions ++ - ScenarioCatalog.extensionContributions).toMap - - assertTrue( - foundationNames.intersect(extensionNames).isEmpty, - "an extension names a contribution the foundation already owns: " + - s"${foundationNames.intersect(extensionNames).mkString(", ")}") - assertEquals( - (foundationNames ++ extensionNames).sorted, - integratedNames.sorted, - "the catalog integrates a contribution that is neither a foundation nor an extension entry") - ScenarioCatalog.contributions.foreach { case (name, contribution) => - assertEquals( - declaredCases(name).map(_.id), - contribution.map(_.id), - s"$name is integrated as something other than the list the capability declares") - } - } - - @Test - def theCatalogIsItsContributionsConcatenatedInOrder(): Unit = { - val contributionOffsets = ScenarioCatalog.contributions - .scanLeft(0) { case (offset, (_, contribution)) => offset + contribution.size } - - assertEquals( - ScenarioCatalog.contributions.map { case (_, contribution) => contribution.size }.sum, - ScenarioCatalog.caseIds.size, - "the catalog holds exactly as many cases as its contributions supply") - ScenarioCatalog.contributions.zip(contributionOffsets).foreach { - case ((name, contribution), offset) => - assertEquals( - contribution.map(_.id), - ScenarioCatalog.caseIds.slice(offset, offset + contribution.size), - s"$name does not occupy the slice of the catalog its position claims") - } - } } diff --git a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala deleted file mode 100644 index e7a189c98..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/DmlCaseCatalogTest.scala +++ /dev/null @@ -1,289 +0,0 @@ -package harness - -import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} -import org.junit.jupiter.api.Test - -/** - * Pins the shape the standard DML tests are written in: one list of test cases, one list of preparations, and a bucket - * that is the cross of the two. Each feature layer pins its own buckets in its own test. Reading these lists does not - * execute a case or start Spark. - */ -final class DmlCaseCatalogTest { - private val expectedReadTestCaseIds = List("read.projection", "read.filter") - - private val expectedDeleteTestCaseIds = List( - "delete.byPredicate", - "delete.byInList", - "delete.byInSubquery", - "delete.byNotInSubquery", - "delete.byExistsSubquery", - "delete.byNotExistsSubquery", - "delete.byScalarSubquery", - "delete.all", - "delete.none", - "delete.byPartitionPredicate", - "delete.withAlias", - "delete.whereFalse.noSnapshot", - "delete.truncate", - "delete.atSnapshot.rejected") - - private val expectedUpdateTestCaseIds = List( - "update.byPredicate", - "update.withoutCondition", - "update.noMatch", - "update.byInSubquery", - "update.byNotInSubquery", - "update.byExistsSubquery", - "update.byNotExistsSubquery", - "update.byScalarSubquery", - "update.withAlias", - "update.multipleColumns", - "update.byExpression", - "update.movePartition", - "update.nullAssignment") - - private val expectedMergeTestCaseIds = List( - "merge.insertNotMatched", - "merge.updateMatched", - "merge.deleteMatched", - "merge.upsert", - "merge.deleteNotMatchedBySource", - "merge.conditionalUpdate", - "merge.multipleMatchedClauses", - "merge.conditionalInsert", - "merge.allClauses", - "merge.updateStar", - "merge.insertExplicitColumns", - "merge.sourceCTE", - "merge.sourceSetOp", - "merge.intoEmptyTarget", - "merge.nullJoinKey", - "merge.resolveByName") - - private val expectedDmlTestCaseIds = List( - "read.projection", - "read.filter", - "delete.byPredicate", - "delete.byInList", - "delete.byInSubquery", - "delete.byNotInSubquery", - "delete.byExistsSubquery", - "delete.byNotExistsSubquery", - "delete.byScalarSubquery", - "delete.all", - "delete.none", - "delete.byPartitionPredicate", - "delete.withAlias", - "delete.whereFalse.noSnapshot", - "delete.truncate", - "delete.atSnapshot.rejected", - "update.byPredicate", - "update.withoutCondition", - "update.noMatch", - "update.byInSubquery", - "update.byNotInSubquery", - "update.byExistsSubquery", - "update.byNotExistsSubquery", - "update.byScalarSubquery", - "update.withAlias", - "update.multipleColumns", - "update.byExpression", - "update.movePartition", - "update.nullAssignment", - "merge.insertNotMatched", - "merge.updateMatched", - "merge.deleteMatched", - "merge.upsert", - "merge.deleteNotMatchedBySource", - "merge.conditionalUpdate", - "merge.multipleMatchedClauses", - "merge.conditionalInsert", - "merge.allClauses", - "merge.updateStar", - "merge.insertExplicitColumns", - "merge.sourceCTE", - "merge.sourceSetOp", - "merge.intoEmptyTarget", - "merge.nullJoinKey", - "merge.resolveByName", - "insert.into", - "insert.explicitColumns", - "insert.intoSelect", - "append.dataFrame", - "insert.overwrite", - "overwrite.dataFrame") - - @Test - def everyDmlTestCaseIsListedOnceInOrder(): Unit = { - val caseIds = Scenarios.allDmlTestCases.map(_.id) - - assertEquals(expectedDmlTestCaseIds, caseIds) - assertEquals(caseIds.distinct.size, caseIds.size, s"duplicate DML case id in $caseIds") - } - - @Test - def eachCompatibilityListNamesTheOperationsItsStartingStateSupports(): Unit = { - assertEquals( - expectedDeleteTestCaseIds ++ expectedUpdateTestCaseIds ++ expectedMergeTestCaseIds, - Scenarios.rowMutationTestCases.map(_.id)) - assertEquals( - expectedReadTestCaseIds ++ expectedDeleteTestCaseIds ++ expectedUpdateTestCaseIds, - Scenarios.testCasesCompatibleWithAnAddedColumn.map(_.id)) - assertEquals(expectedReadTestCaseIds, Scenarios.readTestCases.map(_.id)) - assertEquals(List("delete.byNullCondition"), Scenarios.nullStringRowTestCases.map(_.id)) - } - - @Test - def orderedPreparationMarksItsKnownFailingMatrixCellExplicitly(): Unit = { - assertEquals( - List("delete.byPartitionPredicate"), - Scenarios.orderedDmlTestCases.collect { - case testCase if testCase.knownBugReason.nonEmpty => testCase.id - }) - } - - @Test - def theNullStringPreparationsExtendTheCorePreparations(): Unit = { - assertEquals( - Scenarios.preparedCoreTables.map(preparation => (preparation.casePrefix, preparation.label)), - Scenarios.preparedNullStringCoreTables.map(preparation => - (preparation.casePrefix, preparation.label))) - assertEquals( - Scenarios.preparedCoreTables.map(_.preparation.steps.size + 1), - Scenarios.preparedNullStringCoreTables.map(_.preparation.steps.size)) - assertEquals( - List("prep.nullStringRow"), - Scenarios.preparedNullStringCoreTables.head.preparation.steps.map(_.label).toList.takeRight(1)) - } - - @Test - def everyDmlCaseIdNamesItsOperationAndItsPreparation(): Unit = { - val describedBuckets = List( - Scenarios.coreDmlCases, - Scenarios.orderedDmlCases, - Scenarios.evolvedDmlCases, - Scenarios.partitionedDmlCases, - Scenarios.fileFormatCases).flatten - val caseIds = describedBuckets.map(_.id) - - caseIds.foreach { caseId => - assertEquals( - 2, - caseId.split(" @ ").length, - s"$caseId must be an operation name, then ' @ ', then a preparation label") - } - assertEquals(caseIds.distinct.size, caseIds.size, "DML case IDs must be unique") - } - - @Test - def eachLayoutListCrossesItsFormatsWithItsPartitionings(): Unit = { - assertEquals(List("parquet", "orc"), Scenarios.fileFormats) - assertEquals( - List( - "unpartitioned/parquet", - "partitioned/parquet", - "unpartitioned/orc", - "partitioned/orc"), - Scenarios.layouts.map(_.label)) - assertEquals( - List("partitioned/parquet", "partitioned/orc"), - Scenarios.partitionedLayouts.map(_.label)) - assertEquals( - Scenarios.fileFormats.map(format => s"nested-unpartitioned/$format"), - Scenarios.nestedLayouts.map(_.label)) - assertEquals( - Scenarios.fileFormats.map(format => s"types-unpartitioned/$format"), - Scenarios.typesLayouts.map(_.label)) - } - - @Test - def everyPreparationLabelDrawsItsFormatFromTheStandardList(): Unit = { - val unknownLabels = ScenarioCatalog.foundationContributions - .flatMap { case (_, contribution) => contribution.map(_.id) } - .map(caseId => caseId.split(" @ ").last) - .map(label => label.split("/").last) - .distinct - .filterNot(Scenarios.fileFormats.contains) - - assertTrue( - unknownLabels.isEmpty, - "every preparation label names a format from ScenarioKit.fileFormats; " + - s"offenders=${unknownLabels.mkString(", ")}") - } - - @Test - def dmlCasesIsTheFourDmlBucketsInPreparationOrder(): Unit = { - assertEquals( - (Scenarios.coreDmlCases ++ - Scenarios.partitionedDmlCases ++ - Scenarios.orderedDmlCases ++ - Scenarios.evolvedDmlCases).map(_.id), - Scenarios.dmlCases.map(_.id)) - } - - @Test - def formatMaterializationIsNotADmlOperation(): Unit = { - assertTrue( - !Scenarios.allDmlTestCases.map(_.id).contains("format.materialization"), - "format.materialization describes the preparation, not an operation run against it") - assertEquals( - caseIds(Scenarios.layoutFormatPreparations, "format.materialization"), - Scenarios.fileFormatCases.map(_.id)) - } - - @Test - def eachBucketIsThePreparationListCrossedWithItsTestCaseList(): Unit = { - val noNullStringPreparations = List.empty[TablePreparation[CoreTable.type]] - val buckets = List( - ( - "coreDmlCases", - Scenarios.coreDmlCases, - Scenarios.preparedCoreTables, - Scenarios.allDmlTestCases, - Scenarios.preparedNullStringCoreTables), - ( - "orderedDmlCases", - Scenarios.orderedDmlCases, - Scenarios.preparedOrderedCoreTables, - Scenarios.allDmlTestCases, - Scenarios.preparedNullStringOrderedCoreTables), - ( - "evolvedDmlCases", - Scenarios.evolvedDmlCases, - Scenarios.preparedEvolvedCoreTables, - Scenarios.testCasesCompatibleWithAnAddedColumn, - noNullStringPreparations), - ( - "partitionedDmlCases", - Scenarios.partitionedDmlCases, - Scenarios.preparedPartitionedCoreTables, - Scenarios.partitionedTableTestCases, - noNullStringPreparations)) - - buckets.foreach { case (bucketName, bucket, preparations, testCases, nullStringPreparations) => - val expectedIds = - caseIds(preparations, testCases) ++ - caseIds(nullStringPreparations, Scenarios.nullStringRowTestCases) - - assertEquals( - expectedIds, - bucket.map(_.id), - s"$bucketName is not its named preparations crossed with its named test cases") - } - } - - private def caseIds( - preparations: List[TablePreparation[CoreTable.type]], - testCases: List[DmlTestCase[CoreTable.type]] - ): List[String] = - preparations.flatMap(preparation => - testCases.map(testCase => - s"${preparation.casePrefix}${testCase.id} @ ${preparation.label}")) - - private def caseIds( - preparations: List[TablePreparation[CoreTable.type]], - testCaseId: String - ): List[String] = - preparations.map(preparation => - s"${preparation.casePrefix}$testCaseId @ ${preparation.label}") -} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/FoundationCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/FoundationCatalogTest.scala deleted file mode 100644 index cc1bffec2..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/FoundationCatalogTest.scala +++ /dev/null @@ -1,122 +0,0 @@ -package harness - -import java.nio.charset.StandardCharsets -import java.security.MessageDigest - -import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} -import org.junit.jupiter.api.Test - -/** - * Pins the frozen foundation this branch owns: the eight capabilities in `ScenarioCatalog.foundationContributions`, - * and for each one the exact number of cases it contributes and the fingerprint of the IDs it contributes, in order. - * - * Every assertion reads `foundationContributions` alone, never the complete catalog, so a later layer that fills in - * `extensionContributions` leaves this file untouched and still passing. Each capability is pinned on its own line - * against its own fingerprint, so a change to one of them fails only that line and names it. Uniqueness, ordering and - * the contributions-concatenated rule are pinned once, for any catalog, in CaseCatalogTest. - * - * Reading the catalog is a Spark-free operation. - */ -final class FoundationCatalogTest { - private val expectedFoundationCaseCount = 642 - - /** - * Every capability the frozen foundation is built from, in the order it declares them, with the number of cases it - * contributes and the SHA-256 of its case IDs joined by newlines. This literal fixture makes every foundation - * addition, removal, rename, reorder or resize require an explicit restatement. - */ - private val expectedFoundationContributions = List( - ("dataTypeCases", 10, "e676820cc791e8bbde8921a38d77049f980c55ce78c3a19b7c30a0c5694065e6"), - ("dmlCases", 536, "0346a1b15adda474d19652c99a480228a61dedd376e4a5913b71a1c45e383e6f"), - ("dmlValidationCases", 12, "ce7a969bef2060ff8b64333509f7ab40489a6c8b76b54354884c6468dc33a5ae"), - ("fileFormatCases", 8, "97d22d197425a9156f8c1ef089d494d92f69a8ffcc09d487e5d8278070d98445"), - ("nestedTypeCases", 18, "f2fcdc951e4c17e2d146c26e529f7ef3639534ea22c67dde795b325a7025ea70"), - ("partitionEvolutionCases", 4, "bca457899d4108e353c31619603cec70888b7d3f4c35af67bbb27ebe9b1c4053"), - ("schemaEvolutionCases", 42, "9609fab5bda329357ec4b582fbf225a47678c754e0698a8482c4d9412ddcceb8"), - ("tablePropertyCases", 12, "b97d3f537b8d6740e00a6ae8fb8b88280de442fbc868342e5019fa0284434738")) - - private val expectedKnownBugCaseIds = List( - "nested.deleteByNestedField @ nested-unpartitioned/orc", - "nested.deleteByNestedField @ nested-unpartitioned/parquet", - "prep.ordered:delete.byPartitionPredicate @ partitioned/orc", - "prep.ordered:delete.byPartitionPredicate @ partitioned/parquet", - "prep.ordered:delete.byPartitionPredicate @ unpartitioned/orc", - "prep.ordered:delete.byPartitionPredicate @ unpartitioned/parquet", - "schema.renameColumn @ partitioned/orc", - "schema.renameColumn @ partitioned/parquet", - "schema.renameColumn @ unpartitioned/orc", - "schema.renameColumn @ unpartitioned/parquet") - - @Test - def eachFoundationCapabilityContributesTheCasesItIsPinnedTo(): Unit = { - val actualContributions = - ScenarioCatalog.foundationContributions.map { case (name, contribution) => - (name, contribution.size, sha256(contribution.map(_.id).mkString("\n"))) - } - - assertEquals( - expectedFoundationContributions.map { case (name, _, _) => name }, - actualContributions.map { case (name, _, _) => name }, - "the foundation declares a different set or order of capabilities than it is pinned to") - expectedFoundationContributions.zip(actualContributions).foreach { - case ((name, expectedCount, expectedSha256), (_, actualCount, actualSha256)) => - assertEquals( - (expectedCount, expectedSha256), - (actualCount, actualSha256), - s"$name changed; count=$actualCount, sha256=$actualSha256") - } - } - - @Test - def theFoundationIsTheSizeItIsPinnedTo(): Unit = { - val foundationCaseCount = - ScenarioCatalog.foundationContributions.map { case (_, contribution) => contribution.size }.sum - - assertEquals( - expectedFoundationCaseCount, - foundationCaseCount, - s"foundation case count changed; count=$foundationCaseCount") - assertEquals( - expectedFoundationCaseCount, - expectedFoundationContributions.map { case (_, count, _) => count }.sum, - "the pinned per-capability counts do not add up to the pinned foundation total") - } - - @Test - def everyFoundationCaseRunsOnAColumnarFormatTheFoundationStandardizedOn(): Unit = { - val preparationFormats = ScenarioCatalog.foundationContributions - .flatMap { case (_, contribution) => contribution.map(_.id) } - .map(caseId => caseId.split(" @ ").last) - .map(label => label.split("/").last) - .distinct - - assertEquals(List("parquet", "orc"), Scenarios.fileFormats) - assertEquals( - Scenarios.fileFormats.sorted, - preparationFormats.sorted, - s"a foundation case runs on a format outside the landing matrix: $preparationFormats") - } - - @Test - def theFoundationSkipMetadataIsPinnedToTheKnownProductBugs(): Unit = { - val foundationCases = - ScenarioCatalog.foundationContributions.flatMap { case (_, contribution) => contribution } - - assertEquals( - expectedKnownBugCaseIds, - foundationCases.collect { - case testCase if testCase.knownBugReason.nonEmpty => testCase.id - }.sorted, - "the foundation known-bug cases changed") - assertTrue( - foundationCases.forall(_.embeddedSkipReason.isEmpty), - "every foundation case reaches the embedded catalog") - } - - private def sha256(value: String): String = - MessageDigest - .getInstance("SHA-256") - .digest(value.getBytes(StandardCharsets.UTF_8)) - .map(byte => f"$byte%02x") - .mkString -} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/PublicSurfaceCompatibilityTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/PublicSurfaceCompatibilityTest.scala deleted file mode 100644 index 333853568..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/PublicSurfaceCompatibilityTest.scala +++ /dev/null @@ -1,103 +0,0 @@ -package harness - -import org.junit.jupiter.api.Assertions.{assertEquals, assertSame, assertTrue} -import org.junit.jupiter.api.Test - -/** - * Pins the entry points a consumer outside this module was written against. Splitting the mixin object, the ordered - * catalog and the case type into three names is an internal reorganisation, so a consumer that reads `Plan` and - * `Scenarios` must keep compiling and keep answering the same values. - * - * Every reference below is written the way an external consumer writes it, so this file fails to compile if any of - * those entry points is dropped, renamed or narrowed. The assertions prove the facade and catalog share one state. - * Reading the catalog is a Spark-free operation. - */ -final class PublicSurfaceCompatibilityTest { - - @Test - def planCaseStillNamesAndConstructsTheCaseType(): Unit = { - val constructed: Plan.Case = - Plan.Case("compat.probe", _ => (), knownBugReason = Some("probe")) - - assertEquals("compat.probe", constructed.id) - assertEquals(Some("probe"), constructed.knownBugReason) - assertEquals(None, constructed.embeddedSkipReason) - assertTrue( - (constructed: TestCase).isInstanceOf[TestCase], - "Plan.Case must be the harness case type, not a separate copy of it") - } - - @Test - def planCaseStillMatchesAsAnExtractor(): Unit = { - val matched = (Plan.Case("compat.probe", _ => ()): Plan.Case) match { - case Plan.Case(id, _, None, None) => id - case other => s"unmatched: $other" - } - - assertEquals("compat.probe", matched) - } - - @Test - def planBugReasonStillPhrasesAKnownBugTheWayItAlwaysDid(): Unit = { - val knownBug = Plan.Case("compat.bug", _ => (), knownBugReason = Some("the rewrite crashes")) - val healthy = Plan.Case("compat.healthy", _ => ()) - - assertEquals(Some("bug: the rewrite crashes"), Plan.bugReason(knownBug)) - assertEquals(None, Plan.bugReason(healthy)) - } - - @Test - def planForwardsToTheCatalogRatherThanHoldingItsOwnCopy(): Unit = { - assertEquals(ScenarioCatalog.caseIds, Plan.caseIds) - assertEquals(ScenarioCatalog.cases.map(_.id), Plan.cases.map(_.id)) - assertEquals(Plan.cases.map(_.id), Plan.caseIds) - } - - @Test - def scenariosStillExposesTheConfigurationAConsumerOverrides(): Unit = { - val originalDataSource = Scenarios.dataSource - try { - Scenarios.dataSource = "probe-source" - - assertEquals("probe-source", Scenarios.dataSource) - assertTrue( - Scenarios.layouts.head.create("db.t_probe").contains("USING probe-source"), - "a CREATE statement must follow the data source the consumer set") - } finally { - Scenarios.dataSource = originalDataSource - } - - assertEquals("iceberg", Scenarios.dataSource) - } - - @Test - def scenariosStillExposesTheCapabilityAndPreparationListsAConsumerReads(): Unit = { - assertEquals(List("parquet", "orc"), Scenarios.fileFormats) - assertEquals(4, Scenarios.layouts.size) - assertEquals(4, Scenarios.preparedCoreTables.size) - assertEquals(2, Scenarios.preparedCoreFormats.size) - assertEquals(536, Scenarios.dmlCases.size) - assertEquals(3, Scenarios.standardSeedRowCount) - assertSame( - Scenarios.dmlCases, - ScenarioCatalog.foundationContributions.toMap.apply("dmlCases"), - "the catalog must integrate the very list the capability exposes") - } - - @Test - def everyNamedContributionIsReadableFromTheScenariosObject(): Unit = { - val contributionsFromScenariosObject: List[(String, List[TestCase])] = List( - "dataTypeCases" -> Scenarios.dataTypeCases, - "dmlCases" -> Scenarios.dmlCases, - "dmlValidationCases" -> Scenarios.dmlValidationCases, - "fileFormatCases" -> Scenarios.fileFormatCases, - "nestedTypeCases" -> Scenarios.nestedTypeCases, - "partitionEvolutionCases" -> Scenarios.partitionEvolutionCases, - "schemaEvolutionCases" -> Scenarios.schemaEvolutionCases, - "tablePropertyCases" -> Scenarios.tablePropertyCases) - - assertEquals( - contributionsFromScenariosObject.map { case (name, cases) => (name, cases.map(_.id)) }, - ScenarioCatalog.foundationContributions.map { case (name, cases) => (name, cases.map(_.id)) }) - } -} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/SupportContractTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/SupportContractTest.scala deleted file mode 100644 index 380f0b09b..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/SupportContractTest.scala +++ /dev/null @@ -1,130 +0,0 @@ -package harness - -import java.util.concurrent.atomic.AtomicInteger - -import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} -import org.junit.jupiter.api.Test - -/** - * Pins the support this branch keeps for later feature layers: the changelog operations, the concurrency primitives, - * and the kit's generic starting-state substrate. These assertions exercise each contract before a feature layer - * integrates it. - * - * These Spark-free assertions inspect changelog data, plain JVM concurrency code, and preparation step lists. - */ -final class SupportContractTest { - private val support = new ChangelogSupport {} - - private val expectedChangelogOperations = List( - ("changelog.append", Map("INSERT" -> 1L)), - ("changelog.overwrite", Map("DELETE" -> 1L)), - ("changelog.delete", Map("DELETE" -> 1L)), - ("changelog.update", Map("DELETE" -> 1L, "INSERT" -> 1L)), - ("changelog.merge", Map("DELETE" -> 1L, "INSERT" -> 2L))) - - @Test - def theChangelogOperationsAreTheOnesALaterLayerCrossesWithItsOwnPreparations(): Unit = { - assertEquals( - expectedChangelogOperations, - support.changelogOperations.map(operation => - (operation.name, operation.expectedChangeCounts))) - } - - @Test - def everyChangelogOperationIsAStatementAgainstTheTableItIsGiven(): Unit = { - support.changelogOperations.foreach { operation => - val statement = operation.statement("db.t_probe") - - assertTrue( - statement.contains("db.t_probe"), - s"${operation.name} does not address the table it is given: $statement") - assertTrue( - operation.expectedChangeCounts.values.forall(_ > 0L), - s"${operation.name} expects a change type it does not produce") - } - } - - @Test - def changelogOperationsCrossWithEveryPreparationTheyAreGiven(): Unit = { - val preparations = support.preparedCoreFormats - val cases = support.changelogOperationCasesFor(preparations) - - assertEquals( - preparations.flatMap(preparation => - support.changelogOperations.map(operation => - s"${preparation.casePrefix}${operation.name} @ ${preparation.label}")), - cases.map(_.id)) - assertTrue( - ScenarioCatalog.foundationContributions - .flatMap { case (_, contribution) => contribution } - .forall(testCase => !testCase.id.startsWith("changelog.")), - "changelog support must not contribute cases to the foundation catalog") - } - - @Test - def runConcurrentlyReleasesEveryFunctionAndReportsNothingWhenTheyAllSucceed(): Unit = { - val completed = new AtomicInteger(0) - val threadErrors = - ConcurrencySupport.runConcurrently(Seq.fill(4)(() => completed.incrementAndGet())) - - assertTrue(threadErrors.isEmpty, s"a function failed unexpectedly: $threadErrors") - assertEquals(4, completed.get) - } - - @Test - def runConcurrentlyReportsTheThrowableEveryFailingFunctionRaised(): Unit = { - val threadErrors = ConcurrencySupport.runConcurrently( - Seq( - () => throw new IllegalStateException("first"), - () => (), - () => throw new IllegalStateException("second"))) - - assertEquals(List("first", "second"), threadErrors.map(_.getMessage).sorted.toList) - } - - @Test - def aTypedCommitConflictIsRecognisedAnywhereInTheCauseChain(): Unit = { - assertTrue( - ConcurrencySupport.isTypedCommitConflict( - new RuntimeException("outer", new CommitFailedProbe("inner"))), - "a commit-conflict class name anywhere in the chain is a typed conflict") - assertTrue( - !ConcurrencySupport.isTypedCommitConflict(new IllegalArgumentException("plain")), - "a failure whose chain names no commit, validation or transport class is untyped") - } - - @Test - def theGenericStartingStatesStayUsableForALaterLayer(): Unit = { - val kit = new KitProbe - - assertTrue( - kit.probeCoreCreate("db.t_probe", "orc").startsWith("CREATE TABLE db.t_probe ("), - "coreCreate must build a CREATE for the table and format it is given") - assertTrue( - kit.probeCoreCreate("db.t_probe", "orc").contains("'write.format.default'='orc'"), - "coreCreate must declare the format it is given") - assertEquals( - "(CAST(7 AS BIGINT), 7, 'row-7', 7.5, false, '2024-01-01-00')", - kit.probeCoreRow(7L, "row-7")) - assertEquals(List("create"), kit.probeEmptyStandardTable("orc").preparation.steps.map(_.label).toList) - assertEquals( - List("create", "insert(3)", "waitForNextSnapshotTimestamp", "insertRowsFourAndFive"), - kit.probeTwoSnapshotTable("parquet").preparation.steps.map(_.label).toList) - } -} - -/** Carries the commit-conflict class-name marker that the harness recognises. */ -private final class CommitFailedProbe(message: String) extends Exception(message) - -/** - * Reads the kit's protected starting-state substrate the way a capability trait reads it, exercising the shared - * contract in the foundation suite. - */ -private final class KitProbe extends ScenarioKit { - def probeCoreCreate(table: String, format: String): String = coreCreate(table, format) - def probeCoreRow(long: Long, tag: String): String = coreRow(long, tag) - def probeEmptyStandardTable(format: String): TablePreparation[CoreTable.type] = - preparedEmptyStandardTable(format) - def probeTwoSnapshotTable(format: String): TablePreparation[CoreTable.type] = - preparedTwoSnapshotTable(format) -} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TableLifecycleTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TableLifecycleTest.scala index 7ccd5181d..5da2938c8 100644 --- a/integrations/spark/delta-harness/src/test/scala/harness/TableLifecycleTest.scala +++ b/integrations/spark/delta-harness/src/test/scala/harness/TableLifecycleTest.scala @@ -1,272 +1,43 @@ package harness -import org.junit.jupiter.api.Assertions.{assertEquals, assertSame, assertThrows, assertTrue} +import org.junit.jupiter.api.Assertions.{assertEquals, assertSame, assertThrows} import org.junit.jupiter.api.Test -import scala.collection.mutable.ListBuffer -/** - * A `ScenarioKit` whose catalog statements are recorded instead of executed, so a test drives the real lifecycle - * boundaries without a Spark session. `failingStatements` names the substrings whose statements throw, which is how a - * test injects a create, rename or cleanup failure. - */ -private final class RecordingScenarioKit extends ScenarioKit { - val statements = ListBuffer.empty[String] - var failingStatements: List[String] = Nil - - val runStatement: String => Unit = statement => { - statements += statement - failingStatements - .find(statement.contains) - .foreach(failing => throw new IllegalStateException(s"statement rejected: $failing")) - } - - def ownedTable(table: String)(create: => Unit)(use: => Unit): Unit = - withOwnedTable(runStatement, table)(create)(use) - - def cleanupStatement(statement: String)(use: => Unit): Unit = - withCleanupStatement(runStatement, statement)(use) - - def trackedRename(originalTable: String)(use: (String => Unit) => Unit): Unit = - withTrackedRename(runStatement, originalTable)(use) - - def heldLock(lock: () => (Int, String), unlock: () => (Int, String))( - use: (() => Unit) => Unit): Unit = - withTableLock(lock, unlock)(use) -} - -/** - * Pins the lifecycle boundaries a case uses for an artifact it builds for itself: the owned table, the unconditional - * cleanup around a rejected create, the rename tracker, and the lock. Every test drives the boundary in `ScenarioKit` - * itself and injects the failure it is about, so a boundary that stopped cleaning up, cleaned up the wrong artifact, - * or swallowed a failure is caught here. - */ final class TableLifecycleTest { - private val ok: () => (Int, String) = () => (200, "") - - @Test - def anOwnedTableIsDroppedWhenItsCreateSucceeds(): Unit = { - val kit = new RecordingScenarioKit - - kit.ownedTable("db.t_owned")(kit.runStatement("CREATE TABLE db.t_owned"))( - kit.runStatement("SELECT 1")) - - assertEquals( - List("CREATE TABLE db.t_owned", "SELECT 1", "DROP TABLE IF EXISTS db.t_owned"), - kit.statements.toList) - } - - @Test - def anOwnedTableIsNotDroppedWhenItsCreateFails(): Unit = { - val kit = new RecordingScenarioKit - kit.failingStatements = List("CREATE TABLE db.t_conflict") - - val thrown = assertThrows( - classOf[IllegalStateException], - () => - kit.ownedTable("db.t_conflict")(kit.runStatement("CREATE TABLE db.t_conflict"))( - kit.runStatement("SELECT 1"))) - - assertTrue(thrown.getMessage.contains("CREATE TABLE db.t_conflict")) - assertEquals(List("CREATE TABLE db.t_conflict"), kit.statements.toList) - } - @Test - def anOwnedTableBodyFailureStaysPrimaryWhenItsDropAlsoFails(): Unit = { - val kit = new RecordingScenarioKit - kit.failingStatements = List("DROP TABLE IF EXISTS db.t_owned") - val bodyFailure = new Exception("body failed") + def cleanupRunsOnlyForAnOwnedTable(): Unit = { + var cleanupCount = 0 + val createFailure = new Exception("table already exists") val thrown = assertThrows( classOf[Exception], () => - kit.ownedTable("db.t_owned")(kit.runStatement("CREATE TABLE db.t_owned"))(throw bodyFailure)) + OwnedTableLifecycle.withOwnership(cleanupCount += 1)(_ => + throw createFailure)) - assertSame(bodyFailure, thrown) - assertEquals(1, thrown.getSuppressed.length) - assertTrue(thrown.getSuppressed.head.getMessage.contains("DROP TABLE IF EXISTS db.t_owned")) - assertEquals( - List("CREATE TABLE db.t_owned", "DROP TABLE IF EXISTS db.t_owned"), - kit.statements.toList) - } + assertSame(createFailure, thrown) + assertEquals(0, cleanupCount) - @Test - def anOwnedTableDropFailureSurfacesWhenTheBodySucceeds(): Unit = { - val kit = new RecordingScenarioKit - kit.failingStatements = List("DROP TABLE IF EXISTS db.t_owned") - - val thrown = assertThrows( - classOf[IllegalStateException], - () => kit.ownedTable("db.t_owned")(kit.runStatement("CREATE TABLE db.t_owned"))(())) + OwnedTableLifecycle.withOwnership(cleanupCount += 1)(markTableCreated => + markTableCreated()) - assertTrue(thrown.getMessage.contains("DROP TABLE IF EXISTS db.t_owned")) + assertEquals(1, cleanupCount) } @Test - def nestedOwnedTablesEachDropOnlyTheTableTheyCreated(): Unit = { - val kit = new RecordingScenarioKit - kit.failingStatements = List("CREATE TABLE db.t_inner") - val outerCreate = "CREATE TABLE db.t_outer" + def cleanupFailureIsSuppressedBehindTheBodyFailure(): Unit = { + val bodyFailure = new Exception("test failed") + val cleanupFailure = new Exception("cleanup failed") val thrown = assertThrows( - classOf[IllegalStateException], - () => - kit.ownedTable("db.t_outer")(kit.runStatement(outerCreate)) { - kit.ownedTable("db.t_inner")(kit.runStatement("CREATE TABLE db.t_inner"))(()) - }) - - assertTrue(thrown.getMessage.contains("CREATE TABLE db.t_inner")) - assertEquals( - List(outerCreate, "CREATE TABLE db.t_inner", "DROP TABLE IF EXISTS db.t_outer"), - kit.statements.toList) - } - - @Test - def aRejectedCreateIsCleanedUpWhateverTheRejectionDid(): Unit = { - val scratchDrop = "DROP TABLE IF EXISTS db.t_scratch" - - // The rejection arrives as expected. - val expectedKit = new RecordingScenarioKit - expectedKit.failingStatements = List("CREATE TABLE db.t_scratch") - expectedKit.cleanupStatement(scratchDrop) { - Check.intercept[IllegalStateException]( - expectedKit.runStatement("CREATE TABLE db.t_scratch")) - } - assertEquals(List("CREATE TABLE db.t_scratch", scratchDrop), expectedKit.statements.toList) - - // The create unexpectedly succeeds, so the interception fails and the scratch table still goes. - val unexpectedSuccessKit = new RecordingScenarioKit - val successThrown = assertThrows( - classOf[AssertionError], - () => - unexpectedSuccessKit.cleanupStatement(scratchDrop) { - Check.intercept[IllegalStateException]( - unexpectedSuccessKit.runStatement("CREATE TABLE db.t_scratch")) - }) - assertTrue(successThrown.getMessage.contains("to be thrown")) - assertEquals( - List("CREATE TABLE db.t_scratch", scratchDrop), - unexpectedSuccessKit.statements.toList) - - // The create throws a different type, so the interception fails and the scratch table still goes. - val wrongTypeKit = new RecordingScenarioKit - val wrongTypeThrown = assertThrows( - classOf[AssertionError], - () => - wrongTypeKit.cleanupStatement(scratchDrop) { - Check.intercept[IllegalArgumentException](throw new IllegalStateException("other")) - }) - assertTrue(wrongTypeThrown.getMessage.contains("but got")) - assertEquals(List(scratchDrop), wrongTypeKit.statements.toList) - - // The assertion after the interception fails, and its failure stays primary over the cleanup failure. - val assertionKit = new RecordingScenarioKit - assertionKit.failingStatements = List(scratchDrop) - val assertionFailure = new Exception("message assertion failed") - val assertionThrown = assertThrows( - classOf[Exception], - () => assertionKit.cleanupStatement(scratchDrop)(throw assertionFailure)) - assertSame(assertionFailure, assertionThrown) - assertEquals(1, assertionThrown.getSuppressed.length) - assertEquals(List(scratchDrop), assertionKit.statements.toList) - } - - @Test - def aTrackedRenameLeavesNothingBehindUnderTheNameItLastAccepted(): Unit = { - // The two names share no prefix, so an injected failure names exactly one of the two renames. - val originalTable = "db.t_alpha" - val renamedTable = "db.t_beta" - val renameAway = s"ALTER TABLE $originalTable RENAME TO $renamedTable" - val renameBack = s"ALTER TABLE $renamedTable RENAME TO $originalTable" - - // The case renames away and back, so the table ends under its original name and nothing is dropped. - val kit = new RecordingScenarioKit - kit.trackedRename(originalTable) { renameTo => - renameTo(renamedTable) - renameTo(originalTable) - } - assertEquals(List(renameAway, renameBack), kit.statements.toList) - - // An assertion between the two renames fails, so the live name is the renamed one and that is what goes. - val assertionKit = new RecordingScenarioKit - val assertionFailure = new Exception("row count assertion failed") - val assertionThrown = assertThrows( classOf[Exception], () => - assertionKit.trackedRename(originalTable) { renameTo => - renameTo(renamedTable) - throw assertionFailure - }) - assertSame(assertionFailure, assertionThrown) - assertEquals( - List(renameAway, s"DROP TABLE IF EXISTS $renamedTable"), - assertionKit.statements.toList) - - // The rename back fails, so the table is still live under the renamed name and that is what goes. - val renameBackKit = new RecordingScenarioKit - renameBackKit.failingStatements = List(renameBack) - val renameBackThrown = assertThrows( - classOf[IllegalStateException], - () => - renameBackKit.trackedRename(originalTable) { renameTo => - renameTo(renamedTable) - renameTo(originalTable) + OwnedTableLifecycle.withOwnership(throw cleanupFailure) { markTableCreated => + markTableCreated() + throw bodyFailure }) - assertTrue(renameBackThrown.getMessage.contains(renameBack)) - assertEquals( - List(renameAway, renameBack, s"DROP TABLE IF EXISTS $renamedTable"), - renameBackKit.statements.toList) - - // The first rename fails, so the table never left its original name and the boundary drops nothing. - val renameAwayKit = new RecordingScenarioKit - renameAwayKit.failingStatements = List(renameAway) - val renameAwayThrown = assertThrows( - classOf[IllegalStateException], - () => renameAwayKit.trackedRename(originalTable)(renameTo => renameTo(renamedTable))) - assertTrue(renameAwayThrown.getMessage.contains(renameAway)) - assertEquals(List(renameAway), renameAwayKit.statements.toList) - } - - @Test - def aHeldLockIsReleasedOnceAndItsResponsesAreChecked(): Unit = { - // The case releases the lock itself, so the boundary does not release it again. - val releases = ListBuffer.empty[String] - new RecordingScenarioKit().heldLock(ok, () => { releases += "release"; (200, "") }) { release => - release() - } - assertEquals(List("release"), releases.toList) - // The case leaves the lock held, so the boundary releases it. - releases.clear() - new RecordingScenarioKit().heldLock(ok, () => { releases += "release"; (200, "") })(_ => ()) - assertEquals(List("release"), releases.toList) - - // A rejected lock request fails the case before the body runs. - var bodyRan = false - val lockThrown = assertThrows( - classOf[AssertionError], - () => - new RecordingScenarioKit() - .heldLock(() => (503, "unavailable"), ok)(_ => bodyRan = true)) - assertTrue(lockThrown.getMessage.contains("lock request failed: 503")) - assertTrue(!bodyRan, "the body should not run when the lock was refused") - - // A rejected release fails the case. - val releaseThrown = assertThrows( - classOf[AssertionError], - () => new RecordingScenarioKit().heldLock(ok, () => (500, "boom"))(_ => ())) - assertTrue(releaseThrown.getMessage.contains("unlock request failed: 500")) - - // A release failure rides along behind a body failure, and the boundary tries the release exactly once. - releases.clear() - val bodyFailure = new Exception("locked-write assertion failed") - val bodyThrown = assertThrows( - classOf[Exception], - () => - new RecordingScenarioKit() - .heldLock(ok, () => { releases += "release"; (500, "boom") })(_ => throw bodyFailure)) - assertSame(bodyFailure, bodyThrown) - assertEquals(1, bodyThrown.getSuppressed.length) - assertTrue(bodyThrown.getSuppressed.head.getMessage.contains("unlock request failed: 500")) - assertEquals(List("release"), releases.toList) + assertSame(bodyFailure, thrown) + assertEquals(List(cleanupFailure), thrown.getSuppressed.toList) } } diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala deleted file mode 100644 index 361c1bd51..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/TablePreparationTest.scala +++ /dev/null @@ -1,77 +0,0 @@ -package harness - -import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} -import org.junit.jupiter.api.Test -import scala.collection.mutable.ListBuffer - -/** - * Pins how a preparation turns a test body into a catalog case: the ID it builds, the post-test hook every case from - * that preparation runs, and the known-bug reason a DML test case carries into its cases. Building a case runs no SQL, - * so these assertions need no Spark session. - */ -final class TablePreparationTest { - private val emptyPreparation = TableTest(CoreTable) - - @Test - def formatsCaseIdFromPrefixNameAndLabel(): Unit = { - val preparation = TablePreparation("partitioned/orc", emptyPreparation, "prep.evolved:") - - val testCase = preparation.test("delete.byPredicate")(_ => ()) - - assertEquals("prep.evolved:delete.byPredicate @ partitioned/orc", testCase.id) - } - - @Test - def formatsCaseIdWithoutAPrefixWhenThePreparationDeclaresNone(): Unit = { - val preparation = TablePreparation("unpartitioned/parquet", emptyPreparation) - - assertEquals( - "insert.into @ unpartitioned/parquet", - preparation.test("insert.into")(_ => ()).id) - } - - @Test - def buildsACaseWithoutRunningItsBodyOrItsPostTestHook(): Unit = { - val calls = ListBuffer.empty[String] - val preparation = TablePreparation[CoreTable.type]( - "unpartitioned/parquet", - emptyPreparation, - afterTest = _ => calls += "afterTest") - - preparation.test("insert.into")(_ => calls += "body") - - assertTrue(calls.isEmpty, s"building a case ran $calls") - } - - @Test - def runsADmlTestCaseUnderTheIdOfThePreparationItIsGiven(): Unit = { - val calls = ListBuffer.empty[String] - val preparation = TablePreparation("unpartitioned/parquet", emptyPreparation) - val dmlTestCase = DmlTestCase( - "insert.into", - (_: PreparedTable[CoreTable.type]) => calls += "insert.into") - - val testCase = dmlTestCase.runOn(preparation) - - assertEquals("insert.into @ unpartitioned/parquet", testCase.id) - assertEquals(None, testCase.knownBugReason) - assertTrue(calls.isEmpty, s"runOn ran the operation: $calls") - } - - @Test - def carriesTheKnownBugReasonOfADmlTestCaseIntoItsCase(): Unit = { - val preparation = TablePreparation("partitioned/orc", emptyPreparation, "prep.ordered:") - val dmlTestCase = DmlTestCase( - "delete.byPartitionPredicate", - (_: PreparedTable[CoreTable.type]) => (), - knownBugReason = Some("the rewrite crashes on a write-ordered table")) - - val testCase = dmlTestCase.runOn(preparation) - - assertEquals("prep.ordered:delete.byPartitionPredicate @ partitioned/orc", testCase.id) - assertEquals(Some("the rewrite crashes on a write-ordered table"), testCase.knownBugReason) - assertEquals( - Some("bug: the rewrite crashes on a write-ordered table"), - testCase.bugReason) - } -} diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala deleted file mode 100644 index 177085b82..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/TableTestTest.scala +++ /dev/null @@ -1,86 +0,0 @@ -package harness - -import org.junit.jupiter.api.Assertions.{ - assertEquals, - assertFalse, - assertNotEquals, - assertSame, - assertThrows, - assertTrue -} -import org.junit.jupiter.api.Test -import scala.collection.mutable.ListBuffer - -/** - * Pins fresh table identity and ownership cleanup: generated names stay namespace-scoped and unique across counter - * resets, cleanup starts after the ownership mark, and a cleanup failure is suppressed behind the primary test failure. - */ -final class TableTestTest { - @Test - def generatedTableNamesAreDistinctAndNamespaceScoped(): Unit = { - TableTest.seedCounter(0) - val firstTable = TableTest.nextQualifiedTableName("test_namespace") - TableTest.seedCounter(0) - val secondTable = TableTest.nextQualifiedTableName("test_namespace") - - assertTrue(firstTable.startsWith("test_namespace.t_")) - assertTrue(secondTable.startsWith("test_namespace.t_")) - assertNotEquals(firstTable, secondTable) - } - - @Test - def failureBeforeOwnershipSkipsCleanup(): Unit = { - val createFailure = new Exception("table already exists") - var cleanupCalled = false - - val thrown = assertThrows( - classOf[Exception], - () => - OwnedTableLifecycle.withOwnership(cleanupCalled = true)(_ => - throw createFailure)) - - assertSame(createFailure, thrown) - assertFalse(cleanupCalled, "a failed create must leave the conflicting table intact") - } - - @Test - def successfulOwnershipRunsCleanupOnce(): Unit = { - var cleanupCount = 0 - - OwnedTableLifecycle.withOwnership(cleanupCount += 1)(markTableCreated => - markTableCreated()) - - assertEquals(1, cleanupCount) - } - - @Test - def cleanupFailureIsPrimaryAfterTheBodySucceeds(): Unit = { - val cleanupFailure = new Exception("cleanup failed") - - val thrown = assertThrows( - classOf[Exception], - () => - OwnedTableLifecycle.withOwnership(throw cleanupFailure)( - markTableCreated => markTableCreated())) - - assertSame(cleanupFailure, thrown) - } - - @Test - def cleanupFailureIsSuppressedOnThePrimaryFailure(): Unit = { - val testFailure = new Exception("test failed") - val cleanupFailure = new Exception("cleanup failed") - - val thrown = assertThrows( - classOf[Exception], - () => - OwnedTableLifecycle.withOwnership(throw cleanupFailure) { markTableCreated => - markTableCreated() - throw testFailure - }) - - assertSame(testFailure, thrown) - assertEquals(List(cleanupFailure), thrown.getSuppressed.toList) - } - -} From 77ca6b95f67ab4248b3dac6693fe338888d0938c Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Wed, 2 Sep 2026 12:43:14 -0700 Subject: [PATCH 19/24] test(delta-harness): remove RTAS catalog pin Validate replace-table behavior by running its scenarios instead of restating their generated catalog entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../test/scala/harness/RtasCatalogTest.scala | 224 ------------------ 1 file changed, 224 deletions(-) delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala diff --git a/integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala deleted file mode 100644 index 2ac0bcf3e..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/RtasCatalogTest.scala +++ /dev/null @@ -1,224 +0,0 @@ -package harness - -import java.nio.charset.StandardCharsets -import java.security.MessageDigest - -import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} -import org.junit.jupiter.api.Test - -/** - * Pins the replace-table contribution this layer adds: its exact size, the fingerprint of its case IDs, the four - * preparation axes its DML runs on, the reusable DML operations it covers, and the replace contracts it claims to - * prove. - * - * Every assertion reads `rtasCases` and the RTAS families alone, so the frozen foundation keeps its own tests and a - * sibling layer that adds its own contribution leaves this file as it is. The catalog invariants that apply to any - * layer, namely ID uniqueness, alphabetical contribution ordering and the foundation-plus-extensions rule, are pinned - * once in CaseCatalogTest. - * - * Reading the catalog builds the case list only; executing a case and starting Spark stay separate steps. - */ -final class RtasCatalogTest { - private val expectedRtasCaseCount = 264 - private val expectedRtasSha256 = - "90512741431e50ca77d4924dd7c8b789e3fa6512a23d5b746bc397a00e4532d1" - - /** The four replace preparations the reusable DML operations run on, in the order the layer builds them. */ - private val expectedRtasPreparationLabels = List( - "unpartitioned/parquet", - "partitioned/parquet", - "unpartitioned/orc", - "partitioned/orc") - - /** - * Every replace contract this layer claims to prove, named by the case ID that proves it. The list is written out - * here as its own literal, so the layer and this list are independent statements of the same coverage and any - * drop, rename or reorder fails this test until the intended coverage is restated. - */ - private val expectedContractCaseNames = List( - "rtas.gate.enabled", - "rtas.gate.disabled.rejected", - "rtas.gate.replicationConflict.rejected", - "rtas.sameShapeReplacement", - "rtas.writeAfterReplace", - "rtas.schema.addColumn", - "rtas.schema.dropColumn", - "rtas.schema.widenColumn", - "rtas.schema.incompatibleType.notSilentlyLossy", - "rtas.partition.specReplaced", - "rtas.partition.changeAfterReplace", - "rtas.property.userPropertyPreserved", - "rtas.property.statementOverridesProperty", - "rtas.policy.retentionPreserved", - "rtas.policy.columnTagPreserved", - "rtas.history.preReplaceTimeTravel", - "rtas.history.rollbackRejected", - "rtas.history.setCurrentSnapshotRecovers", - "rtas.changelog.acrossBoundaryRejected", - "rtas.incrementalRead.acrossBoundaryRejected", - "rtas.rename.replaceThenRename", - "rtas.rename.renameThenReplace", - "rtas.sortOrder.changedAfterReplace", - "rtas.sortOrder.removedAfterReplace", - "rtas.identity.creatorPreserved", - "rtas.concurrency.replaceVersusAppend") - - @Test - def theReplaceContributionIsTheSizeAndShapeItIsPinnedTo(): Unit = { - val caseIds = Scenarios.rtasCases.map(_.id) - val actualSha256 = sha256(caseIds.mkString("\n")) - - assertEquals( - expectedRtasCaseCount, - caseIds.size, - s"rtasCases changed; count=${caseIds.size}, sha256=$actualSha256") - assertEquals( - expectedRtasSha256, - actualSha256, - s"rtasCases changed; count=${caseIds.size}, sha256=$actualSha256") - assertEquals(caseIds.distinct.size, caseIds.size, "replace case IDs must be unique") - assertEquals( - Scenarios.rtasDmlCases.map(_.id) ++ Scenarios.rtasContractCases.map(_.id), - caseIds, - "rtasCases is the DML axis followed by the replace contract") - } - - @Test - def theCatalogIntegratesTheReplaceContributionExactlyOnce(): Unit = { - val replaceEntries = ScenarioCatalog.extensionContributions.filter { - case (name, _) => name == "rtasCases" - } - - assertEquals( - 1, - replaceEntries.size, - s"rtasCases is integrated once, found ${replaceEntries.size} entries") - assertEquals( - Scenarios.rtasCases.map(_.id), - replaceEntries.head match { case (_, contribution) => contribution.map(_.id) }, - "the catalog integrates the very list the capability exposes") - } - - @Test - def theReplaceDmlAxisIsTheFourReplacePreparations(): Unit = { - assertEquals( - expectedRtasPreparationLabels, - Scenarios.preparedRtasCoreTables.map(_.label)) - assertEquals( - List("partitioned/parquet", "partitioned/orc"), - Scenarios.preparedRtasPartitionedCoreTables.map(_.label)) - assertEquals( - expectedRtasPreparationLabels, - Scenarios.preparedNullStringRtasCoreTables.map(_.label), - "the null-string preparations extend the same four replace preparations") - assertTrue( - (Scenarios.preparedRtasCoreTables ++ Scenarios.preparedRtasPartitionedCoreTables ++ - Scenarios.preparedNullStringRtasCoreTables) - .forall(_.casePrefix == Scenarios.rtasCasePrefix), - "every replace preparation marks its cases as running on a replaced table") - } - - @Test - def everyReplacePreparationReachesItsStartingStateThroughAReplace(): Unit = { - val replaceStepLabels = List("prep.rtas", "prep.rtas.refresh") - - Scenarios.preparedRtasCoreTables.foreach { preparation => - assertEquals( - List("create", s"insert(${Scenarios.standardSeedRowCount})") ++ replaceStepLabels, - preparation.preparation.steps.map(_.label).toList, - s"${preparation.label} creates, seeds, replaces and refreshes in that order") - } - Scenarios.preparedNullStringRtasCoreTables.foreach { preparation => - assertEquals( - List("prep.nullStringRow"), - preparation.preparation.steps.map(_.label).toList.takeRight(1), - s"${preparation.label} ends by adding the null row the null-string operation reads") - } - } - - @Test - def everyReusableDmlOperationRunsOnTheReplacePreparationsItAppliesTo(): Unit = { - val coveredOperationNames = Scenarios.rtasDmlCases - .map(_.id.stripPrefix(Scenarios.rtasCasePrefix).split(" @ ").head) - .distinct - .sorted - val reusableOperationNames = (Scenarios.allDmlTestCases ++ - Scenarios.nullStringRowTestCases ++ - Scenarios.partitionedTableTestCases).map(_.id).distinct.sorted - - assertEquals( - reusableOperationNames, - coveredOperationNames, - "every reusable DML operation runs on a replaced table") - assertEquals(54, reusableOperationNames.size, "the reusable DML operation count changed") - assertEquals( - Scenarios.preparedRtasCoreTables.flatMap(preparation => - Scenarios.allDmlTestCases.map(testCase => - s"${preparation.casePrefix}${testCase.id} @ ${preparation.label}")), - Scenarios.rtasCoreDmlCases.map(_.id), - "the core replace bucket is its preparations crossed with every reusable operation") - assertEquals(204, Scenarios.rtasCoreDmlCases.size) - assertEquals(4, Scenarios.rtasNullStringDmlCases.size) - assertEquals(4, Scenarios.rtasPartitionedDmlCases.size) - assertEquals(212, Scenarios.rtasDmlCases.size) - } - - @Test - def everyReplaceContractHasAtLeastOneCaseInEveryColumnarFormat(): Unit = { - val contractCaseNames = Scenarios.rtasContractCases - .map(_.id.split(" @ ").head) - .distinct - - assertEquals( - expectedContractCaseNames, - contractCaseNames, - "the replace contract families changed") - expectedContractCaseNames.foreach { contractCaseName => - assertEquals( - Scenarios.fileFormats.map(format => s"$contractCaseName @ $format"), - Scenarios.rtasContractCases.map(_.id).filter(_.startsWith(s"$contractCaseName @ ")), - s"$contractCaseName runs in every columnar format") - } - assertEquals( - expectedContractCaseNames.size * Scenarios.fileFormats.size, - Scenarios.rtasContractCases.size) - } - - @Test - def everyReplaceCaseRunsOnAColumnarFormatInTheLandingMatrix(): Unit = { - val preparationFormats = Scenarios.rtasCases - .map(_.id.split(" @ ").last) - .map(label => label.split("/").last) - .distinct - - assertEquals(List("parquet", "orc"), Scenarios.fileFormats) - assertEquals( - Scenarios.fileFormats.sorted, - preparationFormats.sorted, - s"every replace case runs on a landing-matrix format, found $preparationFormats") - } - - @Test - def theReplaceSkipMetadataIsPinnedToTheKnownProductBugs(): Unit = { - assertEquals( - List( - "rtas.concurrency.replaceVersusAppend @ orc", - "rtas.concurrency.replaceVersusAppend @ parquet", - "rtas.schema.incompatibleType.notSilentlyLossy @ orc", - "rtas.schema.incompatibleType.notSilentlyLossy @ parquet"), - Scenarios.rtasCases.collect { - case testCase if testCase.knownBugReason.nonEmpty => testCase.id - }.sorted, - "the replace known-bug cases changed") - assertTrue( - Scenarios.rtasCases.forall(_.embeddedSkipReason.isEmpty), - "every replace case reaches the embedded catalog") - } - - private def sha256(value: String): String = - MessageDigest - .getInstance("SHA-256") - .digest(value.getBytes(StandardCharsets.UTF_8)) - .map(byte => f"$byte%02x") - .mkString -} From 3ce11dfc50f2d8d2732112c4d0241844467f5882 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Wed, 2 Sep 2026 12:43:54 -0700 Subject: [PATCH 20/24] test(delta-harness): remove merge-on-read catalog pin Validate merge-on-read behavior by running its scenarios instead of restating its generated catalog entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/MergeOnReadCatalogTest.scala | 250 ------------------ 1 file changed, 250 deletions(-) delete mode 100644 integrations/spark/delta-harness/src/test/scala/harness/MergeOnReadCatalogTest.scala diff --git a/integrations/spark/delta-harness/src/test/scala/harness/MergeOnReadCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/MergeOnReadCatalogTest.scala deleted file mode 100644 index c0077e061..000000000 --- a/integrations/spark/delta-harness/src/test/scala/harness/MergeOnReadCatalogTest.scala +++ /dev/null @@ -1,250 +0,0 @@ -package harness - -import java.nio.charset.StandardCharsets -import java.security.MessageDigest - -import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} -import org.junit.jupiter.api.Test - -/** - * Pins the merge-on-read contribution this layer adds: its exact size, the fingerprint of its case IDs, the - * preparation axes its DML runs on, the reusable operations it covers, the merge-on-read contracts it claims to - * prove, and its skip metadata. - * - * Every assertion reads `mergeOnReadCases` and the merge-on-read families alone, so the frozen foundation and the - * replace layer keep their own tests and a sibling layer that adds its own contribution leaves this file as it is. - * The catalog invariants that apply to any layer, namely ID uniqueness, alphabetical contribution ordering and the - * foundation-plus-extensions rule, are pinned once in CaseCatalogTest. - * - * Reading the catalog builds the case list only; executing a case and starting Spark stay separate steps. - */ -final class MergeOnReadCatalogTest { - private val expectedMergeOnReadCaseCount = 320 - private val expectedMergeOnReadSha256 = - "cf7b880c9c8a92bd9a7e6489224e9a80e46080bc20457fa1dc1499d589346a2b" - - /** The merge-on-read preparations the row-mutating operations run on, in the order the layer builds them. */ - private val expectedMergeOnReadPreparationLabels = List( - "mor-unpartitioned/parquet", - "mor-partitioned/parquet", - "mor-unpartitioned/orc", - "mor-partitioned/orc") - - /** The replace-lineage merge-on-read preparations, which are this layer's one dependency on the replace layer. */ - private val expectedReplacedPreparationLabels = List( - "mor-unpartitioned/parquet", - "mor-unpartitioned/orc") - - /** The preparations that already carry a live position-delete file. */ - private val expectedDeletedPreparationLabels = List("mor-verify/parquet", "mor-verify/orc") - - /** - * Every merge-on-read contract this layer claims to prove, named by the case ID that proves it, in the order the - * layer integrates them. The list is written out here as its own literal, so the layer and this list are - * independent statements of the same coverage and any drop, rename or reorder fails this test until the intended - * coverage is restated. - */ - private val expectedContractCaseNames = List( - "mergeOnRead.deleteFile.writesDeleteFile", - "mergeOnRead.deleteFile.copyOnWriteRewritesDataFile", - "mergeOnRead.deleteMode.alterToMergeOnRead", - "mergeOnRead.coexistence.append", - "mergeOnRead.coexistence.secondDelete", - "mergeOnRead.coexistence.update", - "mergeOnRead.coexistence.filteredRead", - "mergeOnRead.coexistence.compactDeletes", - "mergeOnRead.coexistence.merge", - "mergeOnRead.metadata.positionDeletes", - "mergeOnRead.changelog.append", - "mergeOnRead.changelog.overwrite", - "mergeOnRead.changelog.delete", - "mergeOnRead.changelog.update.rejected", - "mergeOnRead.changelog.merge.rejected", - "format.materialization", - "mergeOnRead.fileReplication.deleteFileProperty", - "mergeOnRead.history.timeTravelBeforeDelete", - "mergeOnRead.history.rollbackUndoesDelete") - - /** Every maintenance contract, named by the case ID that proves it. */ - private val expectedMaintenanceCaseNames = List( - "mergeOnRead.maintenance.rewriteDataFilesFoldsDelete", - "mergeOnRead.maintenance.rewritePositionDeletesClearsDangling", - "mergeOnRead.maintenance.expireSnapshotsKeepsDelete", - "mergeOnRead.maintenance.rewriteManifestsKeepsDelete", - "mergeOnRead.maintenance.removeOrphanFilesRemovesTheOrphan", - "mergeOnRead.maintenance.compactThenExpireKeepsDelete", - "mergeOnRead.maintenance.rewritePositionDeleteFiles") - - @Test - def theMergeOnReadContributionIsTheSizeAndShapeItIsPinnedTo(): Unit = { - val caseIds = Scenarios.mergeOnReadCases.map(_.id) - val actualSha256 = sha256(caseIds.mkString("\n")) - - assertEquals( - expectedMergeOnReadCaseCount, - caseIds.size, - s"mergeOnReadCases changed; count=${caseIds.size}, sha256=$actualSha256") - assertEquals( - expectedMergeOnReadSha256, - actualSha256, - s"mergeOnReadCases changed; count=${caseIds.size}, sha256=$actualSha256") - assertEquals(caseIds.distinct.size, caseIds.size, "merge-on-read case IDs are unique") - assertEquals( - Scenarios.mergeOnReadDmlCases.map(_.id) ++ - Scenarios.mergeOnReadContractCases.map(_.id) ++ - Scenarios.mergeOnReadMaintenanceCases.map(_.id), - caseIds, - "mergeOnReadCases is the DML axis, then the contract, then maintenance") - } - - @Test - def theCatalogIntegratesTheMergeOnReadContributionExactlyOnce(): Unit = { - val mergeOnReadEntries = ScenarioCatalog.extensionContributions.filter { - case (name, _) => name == "mergeOnReadCases" - } - - assertEquals( - 1, - mergeOnReadEntries.size, - s"mergeOnReadCases is integrated once, found ${mergeOnReadEntries.size} entries") - assertEquals( - Scenarios.mergeOnReadCases.map(_.id), - mergeOnReadEntries.head match { case (_, contribution) => contribution.map(_.id) }, - "the catalog integrates the very list the capability exposes") - } - - @Test - def theMergeOnReadDmlAxisIsTheWriteModePreparations(): Unit = { - assertEquals( - expectedMergeOnReadPreparationLabels, - Scenarios.preparedMergeOnReadCoreTables.map(_.label)) - assertEquals( - expectedMergeOnReadPreparationLabels, - Scenarios.preparedNullStringMergeOnReadCoreTables.map(_.label), - "the null-string preparations extend the same merge-on-read preparations") - assertEquals( - expectedReplacedPreparationLabels, - Scenarios.preparedReplacedMergeOnReadCoreTables.map(_.label)) - assertEquals( - expectedDeletedPreparationLabels, - Scenarios.preparedDeletedMergeOnReadTables.map(_.label)) - assertTrue( - Scenarios.preparedMergeOnReadCoreTables.forall( - _.casePrefix == Scenarios.mergeOnReadCasePrefix), - "every merge-on-read preparation marks its cases as running on the merge-on-read write path") - assertTrue( - Scenarios.preparedReplacedMergeOnReadCoreTables.forall( - _.casePrefix == Scenarios.replacedMergeOnReadCasePrefix), - "every replace-lineage preparation marks its cases as running on a replaced table") - assertTrue( - Scenarios.preparedDeletedMergeOnReadTables.forall( - _.casePrefix == Scenarios.deletedMergeOnReadCasePrefix), - "every deleted preparation marks its cases as running behind a live delete file") - } - - @Test - def everyDeletedPreparationReachesItsStartingStateThroughAPositionDelete(): Unit = { - Scenarios.preparedDeletedMergeOnReadTables.foreach { preparation => - assertEquals( - List("create", s"seed(${Scenarios.standardSeedRowCount}, one-file)", "prep.morDelete"), - preparation.preparation.steps.map(_.label).toList, - s"${preparation.label} creates, seeds into one file, then deletes a strict subset") - } - Scenarios.preparedReplacedMergeOnReadCoreTables.foreach { preparation => - assertEquals( - List( - "create", - s"insert(${Scenarios.standardSeedRowCount})", - "prep.rtasMor", - "prep.rtasMor.refresh"), - preparation.preparation.steps.map(_.label).toList, - s"${preparation.label} creates, seeds, replaces and refreshes in that order") - } - } - - @Test - def everyReusableOperationTheWriteModeChangesRunsOnAMergeOnReadPreparation(): Unit = { - val coveredOperationNames = Scenarios.mergeOnReadDmlCases - .map(caseId => - caseId.id - .stripPrefix(Scenarios.mergeOnReadCasePrefix) - .stripPrefix(Scenarios.replacedMergeOnReadCasePrefix) - .stripPrefix(Scenarios.deletedMergeOnReadCasePrefix) - .split(" @ ") - .head) - .distinct - .sorted - val reusableOperationNames = (Scenarios.rowMutationTestCases ++ - Scenarios.nullStringRowTestCases ++ - Scenarios.readTestCases).map(_.id).distinct.sorted - - assertEquals( - reusableOperationNames, - coveredOperationNames, - "every reusable operation the write mode changes runs on a merge-on-read table") - assertEquals(46, reusableOperationNames.size, "the reusable operation count changed") - assertEquals(176, Scenarios.mergeOnReadCoreDmlCases.size) - assertEquals(88, Scenarios.replacedMergeOnReadDmlCases.size) - assertEquals(4, Scenarios.deletedMergeOnReadDmlCases.size) - assertEquals(268, Scenarios.mergeOnReadDmlCases.size) - } - - @Test - def everyMergeOnReadContractHasACaseInEveryColumnarFormat(): Unit = { - val contractCaseNames = Scenarios.mergeOnReadContractCases - .map(_.id.split(" @ ").head.split(":").last) - .distinct - val maintenanceCaseNames = Scenarios.mergeOnReadMaintenanceCases - .map(_.id.split(" @ ").head.split(":").last) - .distinct - - assertEquals(expectedContractCaseNames, contractCaseNames, "the contract families changed") - assertEquals( - expectedMaintenanceCaseNames, - maintenanceCaseNames, - "the maintenance families changed") - assertEquals(38, Scenarios.mergeOnReadContractCases.size) - assertEquals(14, Scenarios.mergeOnReadMaintenanceCases.size) - (expectedContractCaseNames ++ expectedMaintenanceCaseNames).foreach { contractCaseName => - assertEquals( - 2, - (Scenarios.mergeOnReadContractCases ++ Scenarios.mergeOnReadMaintenanceCases) - .count(_.id.split(" @ ").head.split(":").last == contractCaseName), - s"$contractCaseName runs in both columnar formats") - } - } - - @Test - def everyMergeOnReadCaseRunsOnAColumnarFormatInTheLandingMatrix(): Unit = { - val preparationFormats = Scenarios.mergeOnReadCases - .map(_.id.split(" @ ").last) - .map(label => label.split("/").last) - .distinct - - assertEquals(List("parquet", "orc"), Scenarios.fileFormats) - assertEquals( - Scenarios.fileFormats.sorted, - preparationFormats.sorted, - s"every merge-on-read case runs on a landing-matrix format, found $preparationFormats") - } - - @Test - def theMergeOnReadSkipMetadataIsEmpty(): Unit = { - assertEquals( - List.empty[String], - Scenarios.mergeOnReadCases.collect { - case testCase if testCase.knownBugReason.nonEmpty => testCase.id - }, - "every merge-on-read case is expected to pass") - assertTrue( - Scenarios.mergeOnReadCases.forall(_.embeddedSkipReason.isEmpty), - "every merge-on-read case reaches the embedded catalog") - } - - private def sha256(value: String): String = - MessageDigest - .getInstance("SHA-256") - .digest(value.getBytes(StandardCharsets.UTF_8)) - .map(byte => f"$byte%02x") - .mkString -} From ac7ae9db5f946526d47089d90697bfc366d985a1 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Wed, 2 Sep 2026 16:42:13 -0700 Subject: [PATCH 21/24] refactor(delta-harness): separate scenario sources Place scenario definitions, suite composition, and scenario-facing tests in dedicated directories while preserving the harness package and public class names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/openhouse/{ => scenarios}/ChangelogSupport.scala | 0 .../harness/openhouse/{ => scenarios}/ConcurrencySupport.scala | 0 .../scala/harness/openhouse/{ => scenarios}/ScenarioCatalog.scala | 0 .../harness/openhouse/{ => scenarios}/ScenarioDataType.scala | 0 .../scala/harness/openhouse/{ => scenarios}/ScenarioDml.scala | 0 .../harness/openhouse/{ => scenarios}/ScenarioDmlValidation.scala | 0 .../harness/openhouse/{ => scenarios}/ScenarioFileFormat.scala | 0 .../scala/harness/openhouse/{ => scenarios}/ScenarioKit.scala | 0 .../harness/openhouse/{ => scenarios}/ScenarioNestedType.scala | 0 .../openhouse/{ => scenarios}/ScenarioPartitionEvolution.scala | 0 .../openhouse/{ => scenarios}/ScenarioSchemaEvolution.scala | 0 .../harness/openhouse/{ => scenarios}/ScenarioTableProperty.scala | 0 .../test/scala/harness/{ => framework}/TableLifecycleTest.scala | 0 .../src/test/scala/harness/{ => scenarios}/CaseCatalogTest.scala | 0 14 files changed, 0 insertions(+), 0 deletions(-) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ChangelogSupport.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ConcurrencySupport.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioCatalog.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioDataType.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioDml.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioDmlValidation.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioFileFormat.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioKit.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioNestedType.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioPartitionEvolution.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioSchemaEvolution.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioTableProperty.scala (100%) rename integrations/spark/delta-harness/src/test/scala/harness/{ => framework}/TableLifecycleTest.scala (100%) rename integrations/spark/delta-harness/src/test/scala/harness/{ => scenarios}/CaseCatalogTest.scala (100%) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogSupport.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ChangelogSupport.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ChangelogSupport.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ChangelogSupport.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencySupport.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ConcurrencySupport.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ConcurrencySupport.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ConcurrencySupport.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioCatalog.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioCatalog.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioCatalog.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDataType.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioDataType.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDataType.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioDataType.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDml.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioDml.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDml.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioDml.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDmlValidation.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioDmlValidation.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioDmlValidation.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioDmlValidation.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileFormat.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioFileFormat.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioFileFormat.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioFileFormat.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioKit.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioKit.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioKit.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNestedType.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioNestedType.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioNestedType.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioNestedType.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionEvolution.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioPartitionEvolution.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioPartitionEvolution.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioPartitionEvolution.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSchemaEvolution.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioSchemaEvolution.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioSchemaEvolution.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioSchemaEvolution.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableProperty.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioTableProperty.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioTableProperty.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioTableProperty.scala diff --git a/integrations/spark/delta-harness/src/test/scala/harness/TableLifecycleTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/framework/TableLifecycleTest.scala similarity index 100% rename from integrations/spark/delta-harness/src/test/scala/harness/TableLifecycleTest.scala rename to integrations/spark/delta-harness/src/test/scala/harness/framework/TableLifecycleTest.scala diff --git a/integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala b/integrations/spark/delta-harness/src/test/scala/harness/scenarios/CaseCatalogTest.scala similarity index 100% rename from integrations/spark/delta-harness/src/test/scala/harness/CaseCatalogTest.scala rename to integrations/spark/delta-harness/src/test/scala/harness/scenarios/CaseCatalogTest.scala From d793d4b353c222efcbfb7445ff163e2e79da515a Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Wed, 2 Sep 2026 16:42:42 -0700 Subject: [PATCH 22/24] refactor(delta-harness): relocate RTAS scenarios Keep replace-table behavior with the shared scenario suite while preserving its package and published names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../scala/harness/openhouse/{ => scenarios}/ScenarioRtas.scala | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioRtas.scala (100%) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRtas.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioRtas.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioRtas.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioRtas.scala From 563118db6d696864833c81b8b2afda1526426c66 Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Wed, 2 Sep 2026 16:44:19 -0700 Subject: [PATCH 23/24] refactor(delta-harness): relocate merge-on-read scenarios Keep merge-on-read behavior with the shared scenario suite while preserving its package and published names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../harness/openhouse/{ => scenarios}/ScenarioMergeOnRead.scala | 0 .../openhouse/{ => scenarios}/ScenarioMergeOnReadKit.scala | 0 .../{ => scenarios}/ScenarioMergeOnReadMaintenance.scala | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioMergeOnRead.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioMergeOnReadKit.scala (100%) rename integrations/spark/delta-harness/src/main/scala/harness/openhouse/{ => scenarios}/ScenarioMergeOnReadMaintenance.scala (100%) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnRead.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioMergeOnRead.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnRead.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioMergeOnRead.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioMergeOnReadKit.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadKit.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioMergeOnReadKit.scala diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadMaintenance.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioMergeOnReadMaintenance.scala similarity index 100% rename from integrations/spark/delta-harness/src/main/scala/harness/openhouse/ScenarioMergeOnReadMaintenance.scala rename to integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioMergeOnReadMaintenance.scala From 0ac34dd754cf899a50ea3a2f30b0668f72c1943b Mon Sep 17 00:00:00 2001 From: mkuchenbecker Date: Wed, 2 Sep 2026 16:46:03 -0700 Subject: [PATCH 24/24] docs(delta-harness): describe scenario ownership Keep source documentation aligned with behavior-based validation and the new framework and scenario directory boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../scala/harness/openhouse/scenarios/ScenarioCatalog.scala | 2 +- .../main/scala/harness/openhouse/scenarios/ScenarioKit.scala | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioCatalog.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioCatalog.scala index c2e8a5d1c..4ba6564ce 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioCatalog.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioCatalog.scala @@ -37,7 +37,7 @@ object Scenarios * placement. * * A layer adds a capability through two append points: one mixin on `Scenarios` and one entry in - * `extensionContributions`. It writes its own scenario source and focused pin test while the foundation tests and + * `extensionContributions`. It keeps its behavior and assertions in its own scenario source while the framework and * shared kit remain stable. * * Composition is all this object does: a scenario body, a preparation and a case ID all belong to the capability that diff --git a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioKit.scala b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioKit.scala index e4b63f86a..8d8878717 100644 --- a/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioKit.scala +++ b/integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioKit.scala @@ -10,7 +10,7 @@ import java.util.concurrent.TimeUnit * Every capability trait extends this kit, so mixing them into `object Scenarios` puts ScenarioKit first in the * linearization and its vals initialize before any capability's. It holds copy-on-write layouts and preparations only; * each feature layer carries its own kit that extends this one. `protected` members are the shared kit; `public` ones - * are also consumed by `object ScenarioCatalog`, `object Plan` and the catalog tests. + * are also consumed by `object ScenarioCatalog`, `object Plan`, and downstream runners. */ trait ScenarioKit {