Add bidirectional Apache Ossie <-> Cube converter - #289
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new converters/cube/ Python converter that round-trips between Apache Ossie semantic models and Cube YAML data models, including a CLI and extensive tests/fixtures, and registers CUBE in the supported-vendors list.
Changes:
- Introduces bidirectional conversion logic (
convert_cube_to_ossie/convert_ossie_to_cube) with stash/parking mechanisms to preserve unmapped constructs and enable lossless round-trips. - Adds a full test suite (fixtures, property-based tests with Hypothesis fallback, CLI tests) plus Cube converter documentation and packaging (
pyproject.toml). - Adds a dedicated GitHub Actions workflow to run Cube converter tests in CI and updates
converters/README.mdto listCUBE.
Reviewed changes
Copilot reviewed 28 out of 29 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| converters/README.md | Registers CUBE as a supported vendor extension. |
| converters/cube/src/ossie_cube/_common.py | Shared conversion utilities (YAML handling, stash protocol, expression translation, mappings). |
| converters/cube/src/ossie_cube/cube_to_osi.py | Cube → Ossie conversion implementation (datasets/fields/relationships/metrics + preservation). |
| converters/cube/src/ossie_cube/osi_to_cube.py | Ossie → Cube export implementation (layout, meta parking, joins/measures/view generation). |
| converters/cube/src/ossie_cube/converter_issues.py | Structured issue types + issue log for lossy/unsafe conversions. |
| converters/cube/src/ossie_cube/cli.py | ossie-cube CLI: import/export behavior, IO, error reporting. |
| converters/cube/src/ossie_cube/init.py | Public API surface for the converter package. |
| converters/cube/README.md | Converter documentation: mapping table, fan-out semantics, usage, limitations. |
| converters/cube/pyproject.toml | Packaging + dev dependencies/test config for the converter. |
| converters/cube/tests/** | Comprehensive unit, fixture round-trip, property-based, edge-case, and CLI tests. |
| converters/cube/tests/fixtures/** | Cube model fixtures used for round-trip and baseline tests (including TPC-DS). |
| .github/workflows/converter-cube-ci.yml | CI workflow for the Cube converter test suite. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Both from the Copilot review on apache#289. Dimension names were sanitized separately in _convert_model (to decide which members a cube has) and again in _build_dimensions (to name them). The first pass used a fresh `taken` set per field, so a collision was silently swallowed by a set comprehension there and only rejected later in the second pass -- meaning the member set that decides `{CUBE.member}` vs `{CUBE}.column`, and where a measure lands, could be short a name while measures were being placed. Demonstrated: "Order Status" and "order status" collapsed to one name with no error. Now resolved once in _resolve_dimension_names and reused. That also fixes a defect the review did not mention: the old set included the two halves of a split geo dimension (location_latitude, location_longitude), which never exist as Cube dimensions since they merge back into `location`, so a metric referencing one would emit an unresolvable `{CUBE.location_…}`. The halves now resolve to the dimension they merge into. _HypothesisRnd.chance() ignored its `p` argument and always drew an unweighted boolean, so the Hypothesis driver explored a different distribution than the seeded one despite the docstring claiming they share a generator. Now weighted, and drawn so the minimal value means False -- shrinking toward the smallest model rather than the largest. 231 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both addressed in d362c9e.
Names are now resolved once in This also fixed something not mentioned: the old set included both halves of a split
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 31 out of 32 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
converters/cube/src/ossie_cube/osi_to_cube.py:721
- Using
queue.pop(0)makes this BFS O(n²) due to repeated list shifting; on larger relationship graphs this can become unnecessarily slow. Use an index cursor (or a deque) to avoid O(n) pops from the front.
queue = [base]
while queue:
current = queue.pop(0)
for neighbor in adjacency.get(current, []):
if neighbor in paths:
continue
paths[neighbor] = f"{paths[current]}.{neighbor}"
entries.append({"join_path": paths[neighbor], "includes": "*"})
queue.append(neighbor)
converters/cube/tests/test_roundtrip.py:42
- Typo: "licence" is misspelled here (the rest of the repo uses "license").
licence headers on the fixtures) are not part of the data model, and key order
Both from the Copilot review on apache#289. Dimension names were sanitized separately in _convert_model (to decide which members a cube has) and again in _build_dimensions (to name them). The first pass used a fresh `taken` set per field, so a collision was silently swallowed by a set comprehension there and only rejected later in the second pass -- meaning the member set that decides `{CUBE.member}` vs `{CUBE}.column`, and where a measure lands, could be short a name while measures were being placed. Demonstrated: "Order Status" and "order status" collapsed to one name with no error. Now resolved once in _resolve_dimension_names and reused. That also fixes a defect the review did not mention: the old set included the two halves of a split geo dimension (location_latitude, location_longitude), which never exist as Cube dimensions since they merge back into `location`, so a metric referencing one would emit an unresolvable `{CUBE.location_…}`. The halves now resolve to the dimension they merge into. _HypothesisRnd.chance() ignored its `p` argument and always drew an unweighted boolean, so the Hypothesis driver explored a different distribution than the seeded one despite the docstring claiming they share a generator. Now weighted, and drawn so the minimal value means False -- shrinking toward the smallest model rather than the largest. 231 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both from the Copilot review on apache#289. The BFS popped from the front of a list, which is O(n) per pop; a deque makes it O(1). Semantic models are small enough that this was never going to matter in practice, but the deque is also the more idiomatic form. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
508c9f5 to
7d981c9
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 31 out of 32 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
converters/cube/src/ossie_cube/osi_to_cube.py:685
- When the model has foreign-vendor
custom_extensionsbut the imported Cube model had multiple views and none was selected (mapped_viewis missing), export currently drops those extensions while loggingPARKED_IN_META. This causes avoidable data loss and the issue type/message is inconsistent ("parked" vs "dropped"). Prefer parking the extensions on a deterministic view (or failing fast) so Ossie -> Cube stays lossless even in the "no mapped view" case.
if foreign and mapped is None:
issues.add(IssueType.PARKED_IN_META, "model",
"no mapped view to park foreign-vendor custom_extensions on; "
"they have no Cube home and are dropped")
converters/cube/README.md:279
- The README hard-codes an exact test count ("234 tests"), but the PR description claims a different number. Since this value will drift over time, it’s better to avoid a specific count (or generate it automatically) to prevent documentation from becoming stale.
234 tests at 96% line coverage: example-based unit tests per direction, CLI
behavior tests, fixture round-trip tests (including the
Both from the Copilot review on apache#289. Model-level foreign-vendor custom_extensions ride on the view that represents the model. When the source Cube model had several views and none was chosen, there is no such view -- and export silently dropped them. Reachable in practice: import a multi-view Cube model, add a SNOWFLAKE extension to the Ossie model, export, and it is gone. Confirmed by reproducing it. Now refused, with the fix in the message (re-import with `--view`). Parking on an arbitrary view was considered and rejected: only the mapped view's parked extensions are read back on import, so it would look lossless while still losing them. The review also noted the issue type contradicted its own message -- PARKED_IN_META for something reported as "dropped". That was true in two places, not one, and it matters: the README defines PARKED_IN_META as preserved-but-invisible-to-Cube, so a pipeline gating on issue types would have concluded the data survived. Adds DROPPED_NO_CUBE_EQUIVALENT for values that genuinely cannot be preserved, and uses it for relationship ai_context -- a Cube join entry takes only name/sql/relationship, with no `meta` field, making it the one construct with nowhere to go. Also drops the hard-coded test count from the README. It had already drifted out of sync with the PR description, which is the reviewer's point: the number carries no information a reader needs, while the description of what the suite covers does. 236 tests, 97% line coverage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both worth addressing, and the first one was a real bug. Fixed in 31d49d2. Dropped foreign-vendor extensions. Reproduced it: import a Cube model with two views (so no view is mapped), add a Now refused, with the fix in the message ( The type/message inconsistency was in two places, not one. Hard-coded test count. Removed from the README. It had already drifted out of sync with the PR description, which is exactly the point — the number carries no information a reader needs, while the description of what the suite covers does. |
From the Copilot review on apache#289, which found that the placeholder holding a geo dimension's position was only reserved when the half encountered first happened to be `latitude`. With `longitude` first, the recorded index pointed at whatever real dimension had already been appended, and `dimensions[index] = dim` overwrote it. Reproduced: a `city` dimension between the two halves disappeared from the output entirely. Rather than reserve the placeholder earlier, the index arithmetic is gone. Dimensions are now built into a dict keyed by target name, with order taken from each name's first appearance -- which is well defined however the two halves are arranged, adjacent or not, in either order. Probing around the fix turned up two more silent-corruption paths in the same code, both order-dependent: - A geo base colliding with an ordinary field of the same name emitted two dimensions called `home` (invalid Cube) when the ordinary field came first, but was correctly rejected when it came second. Now checked during name resolution, so order does not decide. - Two fields both claiming the same half silently discarded one. Now rejected. Also validates the geo `part` and `of` values, and moves the missing-half check into name resolution so every geo problem is caught in one place before anything is built. 241 tests, 97% line coverage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Confirmed and fixed in d92711e. Reproduced it first — with I took the second of your two suggestions rather than the first. Reserving the placeholder earlier would have worked, but the index arithmetic was the fragile part, so it's gone: dimensions are built into a dict keyed by target name, with order taken from each name's first appearance. That's well defined however the halves are arranged — either order, adjacent or not. Probing around it turned up two more order-dependent paths in the same code:
Also validated the geo Six cases pinned by tests: either order, collision in both orders, duplicate half, missing half, unknown part. |
From the Copilot review on apache#289: additional `semantic_model` entries were reported as PARKED_IN_META, but they are neither converted nor preserved anywhere -- a drop. That is the third instance of the same mislabelling, so rather than patch the flagged line I audited all seven export-side uses. Exactly one was a genuine park: unique_keys -> PARKED_IN_META (correct) extra semantic_model entries -> DROPPED (was parked) dimension.is_time role -> DROPPED (was parked) dimension.is_time opt-out -> DROPPED (was parked) synthesized primary-key dimension -> APPROXIMATED (was parked) no datatype -> Cube type 'string' -> APPROXIMATED (was parked) cross-dataset metric placement -> APPROXIMATED (was parked) The import-direction uses were all genuine parks and are unchanged. Adds APPROXIMATED for the middle case, which neither of the existing types described: nothing is lost and nothing is hidden, but Cube requires a value Ossie does not carry, so the converter chose one and the output asserts slightly more than the input did. Calling that "parked" was wrong in the same way as calling a drop "parked" -- nothing was parked. The point of keeping three types apart is that a caller gating on them can distinguish preserved-but-unreadable from actually-lost from emitted-with-a-guess. Two of the three could not be told apart before. 241 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Correct, and it was the third instance of this — so rather than patch the flagged line I audited all seven export-side uses. Exactly one was a genuine park. Fixed in c22e6d2.
The import-direction uses were all genuine parks and are unchanged. Added Three types kept apart on purpose, since that is what a caller gating on them needs: preserved-but-unreadable-by-Cube, actually lost, and emitted-with-a-guess. Two of the three were indistinguishable before. |
Found the same way as the Databricks issues: by checking what a spoke actually made of our output rather than that it exited zero. Cube -> Ossie -> Snowflake produced a Cortex Analyst model with **zero dimensions and 27 facts** across TPC-DS -- every categorical column classified as a numeric measure. The cause is on our side. The Snowflake converter classifies "a field with no `dimension` block as a fact regardless of datatype", which is a fair reading: the block is the role marker. Import emitted it only for time dimensions, so everything else looked like a fact. A Cube `dimensions:` entry is a dimension by definition, so the block is now always emitted -- empty for a non-time one, which leaves the consumer to apply the spec's own default instead of this converter asserting `is_time: false`. Snowflake output for the same model, before -> after: store_sales dim=0 fact=9 -> dim=9 fact=0 customer dim=0 fact=6 -> dim=6 fact=0 date_dim dim=0 time=3 fact=2 -> dim=2 time=3 fact=0 which matches the shape of that converter's own committed example. No other spoke's result changed. Also covers the geo halves, whose fields are built on a separate path; and pins that the dialect fallback is not Databricks-specific -- Snowflake and BigQuery alone convert too. The two Ossie snapshot fixtures are regenerated. 559 tests with both gates, 533 with neither, 96% coverage.
All three blockers are regressions from the previous two commits: each fixed the forward direction by making a choice Cube requires, and each choice was one-way, so `Ossie -> Cube -> Ossie` no longer returned the document it was given. The pattern is the one the rest of the converter already uses -- record the choice in `meta.ossie`, undo it on the way back: - A warehouse dialect used in place of ANSI is recorded, so re-import labels the SQL as that dialect instead of calling vendor SQL `ANSI_SQL`. On measures as well as fields; metrics carry expressions too. - A `unique_keys` entry promoted to satisfy Cube's join requirement is recorded, so re-import does not hand back a declared `primary_key` the model never had. The dimension the promotion synthesized is recorded too, so it does not come back as a field for a column the Ossie model never described. - An Ossie field with no `dimension` block is recorded, so it returns as the fact it was rather than as a dimension. Cube has one kind of dimension, so the block still goes out on every member -- that is what the Snowflake classification needs. `test_a_model_from_another_converter_survives_the_round_trip_exactly` pins all of it on the committed Databricks-authored fixture: dialects, keys, fields and roles compared before and after. It would have failed on each of the three. Worth noting why the property tests missed these: the generator draws ANSI expressions, declares a primary key, and gives every field a dimension role -- so none of the three shapes can occur in a generated model. The fixture from another converter is the only thing in the suite that has them. 560 tests with both gates, 533 with neither, 96% coverage. Snowflake classification and the interop matrix unchanged.
Three review rounds found bugs in one blind spot: the generator drew ANSI expressions, always declared a `primary_key`, and gave every field a `dimension` role -- so none of the 210 generated models per run could contain the shapes that were breaking. The committed Databricks-authored fixture was the only thing in the suite that had them, which is why the same class of defect came back three times. The generator now draws all three, since each is a place where export must make a choice Cube requires and then be able to undo it: - a dialect per field and per metric, often a warehouse one with no ANSI alternative; - either `primary_key` or `unique_keys` (never neither -- Cube rightly refuses a cube with a join and no key); - a `dimension` role or none, the latter being a fact. And the property compares what those choices affect -- dialects, keys, roles and datatypes -- not just expressions, which is how one-way fixes slipped past it before. Checked that it can fail: reverting each of the three provenance records in turn breaks 61, 61 and 34 of the 122 property cases. A green test that cannot fail is not a test. 560 tests with both gates, 534 with neither, 96% coverage.
Both blockers were provenance recorded by halves. - Ossie names a primary key by *column*; Cube marks a *dimension*. The two differ whenever the dimension carrying the key is not named after its column -- a field `order_id` reading column `id`, or a synthesized `id_pk` where a computed field shadows the column -- and import rebuilt the key from dimension names, so it came back naming something the table need not have. The column list is recorded when it cannot be read back off the dimensions, and only then, so a model whose names already agree keeps a clean Cube round trip. `_primary_key_of` returns columns now, which also fixes the rebuilt `COUNT(DISTINCT ...)`: it was naming the synthesized dimension, a member the Ossie side does not have at all. - Recording only the chosen dialect's *name* lost the alternatives. Cube holds one `sql` per member, so nothing short of the whole expression object brings a multi-dialect expression back; it is parked entire, on measures as well as fields. The generator now draws several dialects per expression and the property compares them all rather than `dialects[0]`, which is what let the second one through. That immediately found two more: - an expression offering two warehouse dialects and no ANSI was dropped outright, because the fallback insisted on a sole candidate. It takes the first in document order and reports it -- Cube passes SQL to one data source, and the alternatives are parked. - `COUNT(DISTINCT <pk>)` drifted to the synthesized dimension name, above. Also a flaw in the generator itself: it picked an alternative dialect out of a `set`, whose iteration order varies between processes, so the seeded sweep produced different models each run and could not name a reproducible seed. Sorted now -- the same suite ran green and red in consecutive invocations before this. Checked both fixes can fail: reverting each breaks 60 and 9 of 122 property cases. Metric drift across 400 generated models is zero. 560 tests with both gates, 534 with neither, 97% coverage.
[P1] The recorded key column list is *columns*, but the `computed_primary_key` inference read it as dimension names -- so a key column `id` alongside a computed field also named `id` came back flagged as computed, and the second export marked `LOWER(email)` as the key instead of synthesizing `id_pk`. Cube then deduplicated on a different value, which changes the counts it returns. The inference is skipped when `meta.ossie.primary_key` supplied the key, because those entries are columns by construction. Both Ossie documents were identical in that case; only the *Cube* model changed. So the property now runs a second export and requires it to reproduce the first, which is the only way to see a record that one side writes and the other reads differently. That found two more, neither reachable in a single cycle: - A decomposed metric's public measure was stashed verbatim and restored with references to hidden parts the next export no longer generated. Cube's verdict on the second cycle: "fact.crossing_part_1 cannot be resolved" -- a broken model. The public half is marked, so re-import rebuilds it from its expression and both halves are regenerated together. - `COUNT(DISTINCT DIM_0.ID)` was not recognized as the primary-key count because the comparison was case-sensitive, so cycle 1 emitted `count_distinct` and cycle 2 -- reading a canonically regenerated expression -- emitted the bare `count`. Compared on normalized identifiers now, which also means a metric spelling the key in any case gets Cube's fan-out-safe form. Non-blocking wording fixed too: the fallback may pick the first of several warehouse dialects, not only a sole one. Checked the new checks can fail: reverting the inference fix breaks 9 cases across the property sweep and the targeted two-cycle test. 561 tests with both gates, 534 with neither, 97% coverage.
|
PR #289 — reviewer note Tests: 534, or 561 with the optional Cube gate. Coverage: 97%. Layers:
Cross-matrix (tools/interop_matrix.py): our Ossie output is run through all nine Python converters, with a report showing what each one made of it. This is what the stash reduction was measured against: 7 CUBE entries on TPC-DS instead of 41, and 2 Databricks warnings instead of 32. Databricks — both directions:
Snowflake: that converter is export-only, so there is no reverse path in the repository. One direction was verified: Cube → Ossie → Cortex Analyst. These interop paths are what the tests specifically pin down: which dialect an expression carries, where the primary key comes from, and whether a field has a dimension role. Each is a point where the two formats disagree and the conversion must make an explicit choice. Limitations:
What I need: CI has never run on this PR because GitHub holds workflows from external contributors until a committer approves them. The PR needs a review and a CI run. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 46 out of 48 changed files in this pull request and generated no new comments.
Suppressed comments (5)
converters/cube/tools/cube_compile.js:1
- Using
path.basename(p)forfileNamecan cause collisions when compiling multiple files with the same basename (e.g.model/cubes/orders.ymlandmodel/views/orders.yml). Cube keys schema files byfileName, so this can make the compile check fail or (worse) compile the wrong combined model. Use a stable unique key such as the path relative to a common root (or the provided relative model path) forfileName, and ensurelocalPathmatches that root.
converters/cube/tools/interop_matrix.py:1 - This relies on the implicit
bool→intcoercion (True== 1). It works, but it’s easy to misread/accidentally change. Consider making the increment explicit (e.g.,if result == 'FAIL': failures += 1) for clarity.
converters/cube/src/ossie_cube/expressions.py:310 - As written, any whitespace at depth 0 makes this return
True, including leading/trailing spaces. That can cause unnecessary parentheses when inlining an otherwise single-term expression like'SUM(x) '(or any expression with incidental formatting). A simple fix is to scanstr(expr).strip()and/or treat whitespace as structural only when it participates in a token pattern (e.g.,CASE,WHEN,AND,OR) rather than any space.
def has_top_level_operator(expr):
"""True if `expr` is not a single self-contained term.
Used to decide whether inlining it back into a larger expression needs
parentheses: a lone `SUM(x)` does not, `SUM(x) / 2` does.
"""
depth, quote = 0, None
for ch in str(expr):
if quote:
if ch == quote:
quote = None
elif ch in "'\"":
quote = ch
elif ch == "(":
depth += 1
elif ch == ")":
depth -= 1
elif depth == 0 and (ch in "+-*/%<>=|&" or ch.isspace()):
# Whitespace at depth 0 also implies structure (`CASE WHEN ...`).
return True
return False
converters/cube/tests/_roundtrip_helpers.py:318
- This hard-codes the Ossie version string (
0.2.0.dev0) in the generator. IfOSSIE_VERSIONchanges, these generated models will drift from the rest of the converter/tests and can cause confusing failures. Prefer importing and usingOSSIE_VERSION(as other tests do) to keep versioning consistent.
def build_ossie_model(rnd):
"""Generate a hand-authored Ossie model (no stash) as a YAML string."""
dim_names = [f"dim_{i}" for i in range(rnd.count(1, 2))]
fact = "fact"
lines = ["version: 0.2.0.dev0", "semantic_model:", "- name: shop"]
converters/cube/src/ossie_cube/cli.py:145
- File I/O here (and in export) relies on platform default encodings. For deterministic behavior across OSes/locales, it’s better to open text files with an explicit encoding (typically UTF-8) when reading and writing YAML/model files.
with open(path) as fh:
files[rel] = fh.read()
Cube keeps cubes and views in one global namespace, so a view may not share a name with a cube. The exporter did not check, and an Ossie model named after one of its own datasets produced exactly that -- Cube rejected the whole model with "Cannot read properties of undefined (reading 'toString')". Not an exotic input: it is what every Databricks metric view over a same-named table converts to, and `databricks_ossie.yaml` is one. The generated view becomes `<name>_view` and the model's own name is recorded in `meta.ossie.model_name`, since the mapped view's name is the model's name on the way back and the rename would otherwise stick. Renamed rather than refused because a cube is addressed by joins and by every member reference, a generated view by nothing. That went unnoticed because the gate meant to catch it was dropping half its input. `cube_compile.js` keyed model files by basename; Cube's own FileRepository keys them by path relative to the model root, so `cubes/orders.yml` and `views/orders.yml` collided and one was discarded silently. A valid cube plus an invalid same-named view reported COMPILED OK, while the identical pair under distinct names failed as it should. Keyed by relative path now, matching Cube, and duplicate keys are refused outright rather than resolved by chance -- a gate that quietly compiles less than it was given is worse than no gate. The same flattening was already fixed a layer up in `_cube_gate.py`, where the temp files are written; the JS undid it. With the gate honest, a second defect surfaced one cycle out: a `DATABRICKS` metric came back as `ANSI_SQL` on the second export. The verbatim-restore path hands back the Cube SQL a previous import stashed instead of picking a dialect, so it had no dialect to pass to `_park_expression` and the label was dropped. It falls back to the sole declared dialect, which is the one that SQL came from. `Ossie -> Cube -> Ossie` is byte-stable from the first cycle now; the existing one-cycle comparison could not see this, since cycle one was correct. Also, `validation/validate.py` reports a missing `jsonschema` by calling `sys.exit(1)` at import time, and SystemExit does not derive from Exception -- so it escaped the guard around the validator import and aborted pytest *collection*. The whole suite refused to run on any machine without jsonschema, the exact case `validator_gate` exists to skip. Smaller, from the same review: - The CLI's file I/O used the platform default encoding, so a model carrying any non-ASCII text died under a non-UTF-8 locale: `title: Größe` gave `UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3`. Pinned to UTF-8. - The property generator hard-coded the spec version instead of using `OSSIE_VERSION`. - `has_top_level_operator` treated any depth-0 whitespace as structure, including the trailing newline off a YAML block scalar, and parenthesized a lone `SUM(x)` needlessly. - The interop matrix counted failures by adding a bool; made explicit. Checked the new checks can fail. Reverting the gate keying (and its duplicate guard) leaves the collision test passing a model Cube refuses; reverting the view rename breaks 3 tests including the Databricks compile; reverting the dialect fallback breaks the two-cycle stability test; reverting the encoding breaks the non-UTF-8 CLI test. 566 tests with both gates, 399 with neither, 97% coverage.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 46 out of 48 changed files in this pull request and generated no new comments.
Suppressed comments (4)
converters/cube/tools/interop_matrix.py:1
- For directory outputs,
dest.rglob('*')yields directories as well as files, so a converter that only created empty directories would be reported asOKinstead ofEMPTY. To align the result with the column definition ('nothing written'), consider counting only files (or only non-empty files) whenis_diris true.
converters/cube/tools/interop_matrix.py:1 Path.read_text()defaults to the platform encoding, which can break on non-UTF-8 locales (and this repo already has tests ensuring the CLI is robust under non-UTF-8 locales). Consider reading with an explicitencoding='utf-8'(and similarly, any other reads/writes in this tool that should be Unicode-safe) to keep behavior consistent across environments.
converters/cube/src/ossie_cube/cli.py:104- CLI input paths commonly include
~(home directory).os.path.abspath()does not expand~, so valid user inputs like-i ~/modelwill be reported as missing. Consider expanding user input first (e.g., viaos.path.expanduser) before resolving to an absolute path.
resolved = [os.path.abspath(p) for p in paths]
converters/cube/src/ossie_cube/expressions.py:287
- The containment check re-walks each previously found scope for every node (
O(n^2)traversals). If this runs on many expressions/models, it can become a noticeable hotspot. Consider a single-pass approach that tracks whether the current traversal position is inside an aggregate scope (e.g., using parent/ancestor checks if available, or maintaining a stack during a DFS-style walk) so each node is visited a bounded number of times.
def _outermost_aggregate_scopes(tree):
"""Aggregate scopes that are not inside another one.
Nesting is resolved so an ordered-set aggregate is counted once: `WithinGroup` and
the `PercentileCont` inside it are one aggregate, and treating the inner one as its
own scope would find no columns there and blame the declaring cube.
"""
scopes = []
for node in tree.walk():
if not _is_aggregate_scope(node):
continue
if any(any(inner is node for inner in scope.walk()) for scope in scopes):
continue
scopes.append(node)
return scopes
[P1] The record added for cube/view collisions was scoped to that one cause, and the ordinary causes went unrecorded. `Sales Model` is a legal Ossie name that cannot be a Cube identifier, so it exported as view `sales_model` with nothing parked and came back named `sales_model`. The same for `--name "Sales Model"` over a stashed view already called `sales_model`: the stashed branch compared the mapped name against the *sanitized* model name, the two matched, and the override was silently undone. Keyed on the difference now rather than on the reason for it -- the raw name is preserved whenever it differs from the name of the view that will carry it, whether that difference comes from sanitizing, an override, or a collision. A model whose name is already its view's name still parks nothing, so an ordinary Cube document stays clean. Verified stable over three cycles, since the value has to survive being read back out of the stash and not merely written once. Also non-blocking, from the same review: `test_two_export_cycles_produce_the_same_cube_model` was entirely behind the optional Cube gate, but comparing two exports needs no Cube installation -- so the regression it exists for was unchecked everywhere without a built checkout, CI included. Split, with only `assert_cube_compiles` gated. An audit of every `cube_gate` test found one more of mine with the same mistake (`test_a_renamed_view_still_compiles_and_stays_renamed`); split the same way. The rest are genuinely Cube-only. Checked the new checks can fail: restoring the collision-only rule breaks both name tests, and the two split tests now run (and pass) with no Cube checkout present where they were previously skipped. 571 tests with both gates, 404 with neither, 97% coverage.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 46 out of 48 changed files in this pull request and generated no new comments.
Suppressed comments (4)
converters/cube/src/ossie_cube/expressions.py:104
aggregate_spans()parses the same expression twice on the hot path (parse(text)and thenis_single_aggregate(text)which re-parses). Cache the parse result (or refactoris_single_aggregateto accept a pre-parsed tree) to avoid redundant sqlglot parsing costs, especially when decomposing many metrics.
text = str(expr)
if parse(text) is None or is_single_aggregate(text):
return []
return _scan_aggregates(text)
converters/cube/tests/_util.py:43
- These helpers read fixture files without an explicit encoding. For consistency with the converter’s UTF-8 IO (and to avoid platform/locale-dependent failures), open fixtures with
encoding='utf-8'and usepath.read_text(encoding='utf-8').
def load_fixture(name):
with open(FIXTURES / name) as fh:
return fh.read()
def load_fixture_dir(name):
"""Read a fixture Cube model directory as {relative posix path: text}."""
root = FIXTURES / name
files = {}
for path in sorted(root.rglob("*")):
if path.is_file():
files[path.relative_to(root).as_posix()] = path.read_text()
return files
.github/workflows/converter-cube-ci.yml:53
- Piping a remote install script directly into
shis a supply-chain risk. Prefer using a pinned, maintained GitHub Action for uv (or pin the installer to a specific version + verify its checksum/signature) to make the CI installation step auditable and harder to tamper with.
- name: Install uv
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "${HOME}/.local/bin" >> "${GITHUB_PATH}"
converters/cube/src/ossie_cube/converter_issues.py:114
- The new dataclass fields are untyped (
issues: list,strict_types: frozenset), which makes it harder to use/validate IssueLog from type checkers and IDEs. Consider annotating asissues: list[ConverterIssue]andstrict_types: frozenset[IssueType](and typing method params/returns accordingly).
class IssueLog:
"""Collects issues during a conversion.
`strict_types` names the issue types that should abort the conversion instead of
being recorded. Nothing is in there by default: a converter that refuses a whole
model over one metric leaves the spoke on the other side with nothing. Passing
`--strict-fanout` adds `FANOUT_UNSAFE_METRIC`, mirroring Cube's own refusal to
answer a query whose measures reference cubes that lead to row multiplication.
"""
issues: list = field(default_factory=list)
strict_types: frozenset = frozenset()
[P1] Model-level metadata has no Cube field of its own, so it rides on the view representing the model -- and export emitted no view at all when the stash recorded none mapped. A Cube model need not contain a view, and one with several need not say which is the model, so this is an ordinary input rather than an edge case. Both cases dropped the name, description and AI context in silence, with no issue reported: a cube-only model imported with `--name 'Sales Model'` came back as the synthesized `cube_model`. That contradicts the documented lossless Ossie -> Cube -> Ossie round trip. They ride on a cube now, under `meta.ossie.model`, and import reads them back when no view is mapped. The carrier is the alphabetically first cube -- deterministic, and independent of both dataset ordering and the relationship graph, so every export picks the same one. Import does not depend on the choice; it reads whichever cube carries the record, which cannot accumulate because the record is consumed and stripped from the stash. Only values import could not otherwise recover are parked, so a Cube model that never had model-level metadata still round-trips byte-identical rather than acquiring a `meta.ossie` key it never had. That matters beyond tidiness: every fixture in the feature matrix is cube-only, and their structural round trips would all have started failing. A name equal to the one import synthesizes is recoverable by definition, hence the shared DEFAULT_MODEL_NAME rather than a second copy of the literal. Foreign-vendor `custom_extensions` deliberately keep refusing export in this case, and the README now says why rather than leaving it looking inconsistent: import restores those only from the mapped view, so a cube carrier would not bring them home. That path fails loudly, which was never the complaint here. Checked the new checks can fail: removing the export half breaks 3 tests, removing the import half breaks 2, and both halves are exercised over three cycles because the value has to survive being read back out of a cube's stash rather than merely written once. The carrier's output is put through the Cube compile gate too, since it is new YAML in the emitted model and holds a literal brace -- Cube compiles every string as an f-string. 576 tests with both gates, 408 with neither, 97% coverage. Cross-converter matrix unchanged.
|
Hi @MikeNitsenko @jbonofre , this PR can be merged or additional checks are expected ? |
|
@mfournioux, all checks and updates from my side were implemented - looking for review and approval. @jbonofre @khush-bhatia this adds a bidirectional Cube <-> Ossie converter, following the same shape as #247 and #239. It’s the implementation for issue #248, which asked whether a Cube converter would be welcome and hasn’t had an answer yet. Would one of you be able to review it and approve the CI workflow run? |
Review: everything parseable from an Ossie metric expression should not
ride in custom_extensions. Three reductions, all symmetric:
- `filters` regenerate from the folded CASE: the fold import writes
(Cube's own applyMeasureFilters shape) is deterministic, so export
unfolds it back into structured filters -- verified by refolding, so a
hand-written CASE that merely looks similar stays one expression. A
filtered measure now travels with no stash at all; when the fold is
not invertible (the operand is itself a CASE), both spellings ride.
- Cube-only measure keys (format, drill_members, public, ...) ride flat,
the protocol dimensions already use, instead of forcing a copy of the
whole measure that duplicated the sql and type the expression carries.
- The owning cube is recorded only when the expression does not say it;
export already places a metric on the sole dataset it references.
A declared type the expression would not regenerate (a calculated
measure whose sql is a single aggregate, a count_distinct over the
primary key) is recorded as a flat `type` entry -- the latter was a
latent round-trip flip to bare `count`.
On the fixtures: completed_amount and cities now carry no extension;
total_amount carries {format}; TPC-DS drops from 7 stash entries to 4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review: composite metrics should stay decomposed -- an Ossie document
should not render final values. The expression language lists Metric
references among its supported constructs, and a bare identifier in a
model-level metric expression resolves in the metric namespace. So
`{total_amount} / {count}` now imports as `total_amount /
orders__count` (the referenced metrics' Ossie names) rather than as an
inlined copy of every referenced definition -- which is the metric
drift a shared semantic model exists to prevent. Export renders a bare
metric name back as `{measure}` on the same cube, `{cube.measure}`
across cubes; the fixtureA calculated measure now carries no extension
at all.
Because bare identifiers are references at the model level, raw columns
in measure SQL are dataset-qualified on import (parser-based, string
surgery so spellings survive): `SUM(amount * 2)` reads as
`SUM(orders.amount * 2)`, which also removes the column/metric
ambiguity outright.
The resolver now keeps two forms per measure: the emitted one, and the
fully inlined one Cube itself renders -- which is what the fan-out
analysis reads, since a reference hides the aggregates it stands for.
Inlining remains where a reference cannot: generated decomposition
parts, windowed dependencies (both park, as before), keyword-named
metrics, and multi-aggregate expressions authored as a single measure
(whose spelling rides in the stash so export does not decompose them).
Reference cycles are refused in both directions, as Cube refuses them;
a metric referencing a dialect-dropped metric drops with it,
transitively.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review: generated Cube models should use references. Cube interpolates
a member's sql verbatim into generated queries, so a bare column in a
computed expression is ambiguous the moment the cube is joined against
a table sharing the name. Every bare column in generated member SQL --
computed dimension sql, measure operands, reconstructed filters -- is
now qualified as {CUBE}.column, the reference Cube's own documentation
recommends. Parser-based (sqlglot finds the column tokens, string
surgery applies them), so keywords, function names and EXTRACT units
are never touched, and an unparseable expression is left exactly as it
was. {CUBE}.column rather than a {member} reference on purpose: the two
coincide for a plain member, but the column form keeps meaning the
column even when a computed field shadows its name.
A single-column dimension keeps the bare `sql: column` form Cube
models conventionally use. A pleasant side effect: a hand-written
`CONCAT({CUBE}.tenant_id, {CUBE}.id)` key now round-trips
byte-identically instead of coming back bare. The TPC-DS fixture's
computed dimension is updated to the reference form it should have
been written in.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by running Cube -> Ossie -> Databricks -> Ossie -> Cube end to
end: the Databricks importer emits a metric view's source columns
unqualified (SUM(ss_ext_sales_price) -- unqualified *means* the source
there), and reading that as opaque SQL placed the aggregate on whatever
cube the rest of the expression named. TPC-DS's customer_lifetime_value
put SUM(ss_ext_sales_price) on the customer cube -- a measure over a
column that cube does not have, which compiles (SQL is opaque to Cube's
compiler) and reads the wrong table at query time. Pre-existing, not a
regression: the pre-branch converter placed it identically, just
spelled bare.
A bare identifier in a model-level metric expression that is no metric
but is a declared field of exactly one dataset can only mean that
dataset's column. It now renders through the ordinary reference
machinery ({CUBE}.column, {CUBE.member}, or the cross-cube
{other.member} that carries the implicit join), decomposition places
its aggregate on the owning cube, and the metric's own cube derivation
sees it. A name declared on several datasets is never guessed at: it
stays raw SQL of the fallback cube, as before.
With this, the full interop chain ends in a model Cube compiles, with
one repair the Databricks format forces (a metric view cannot carry the
source table's own key; both converters report it).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-up: customer_lifetime_value still carried a rendered copy
of its own SQL in custom_extensions -- the one measure shape left where
the extension duplicated the expression. Root causes fixed:
- Cross-cube member references ({customer.c_customer_sk}) are
reversible in model-level metric SQL: export re-emits them verbatim,
so the spelling never needed recording. sql_is_reversible now knows
each cube's members and accepts the canonical spelling.
- The owning cube mirrors export's full derivation, base cube included:
a cross-dataset metric on the FK sink needs no cube record.
- Export decomposes a composite only when its aggregates read different
cubes -- that is where per-aggregate fan-out correction lives. A
single-cube composite (MAX(x) - MIN(x)) stays one calculated measure
and round-trips verbatim, stash-free. An inline-authored cross-cube
composite normalizes to the decomposed form on its first round trip,
a documented normalization whose second cycle is a fixed point; the
tpcds fixture now commits that fixed point.
- A geo half's sql rides in the stash only when the field's expression
would not regenerate it -- {CUBE}.lat is what the expression already
says, so the stash keeps only of/part.
Full-fixture audit: fixtureA carries 7 extension entries, tpcds 4
(views curation, segments, geo structure, name mapping, format,
public:false, and two foreign vendors) -- every one something the
expression genuinely cannot say.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-up: a sub_query dimension's sql references a measure
({orders.count}), which Cube resolves through a correlated subquery.
Emitting the flattened reference as an Ossie field expression
(orders.count) claimed a column no dataset has -- text that reads as
valid SQL and computes nothing anywhere -- softened only by an
APPROXIMATED issue. That was inconsistent with the converter's own
precedents: switch dimensions and multi-stage measures, whose Ossie
renderings would equally claim something they are not, are parked
whole. The sub_query dimension now rides the same protocol
(PARKED_IN_META, dataset stash, original position restored verbatim),
and the aggregate itself still reaches the model as the hoisted metric
the reference points at.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Re-verified every fixture and test against the review principles.
Fixtures: both Ossie snapshots regenerate byte-identically; exports
match the committed Cube fixtures structurally (remaining byte diffs
are YAML formatting only); hand-authored fixture carries only its
deliberate foreign-vendor extension.
Two behaviors the audit found tested only implicitly are now pinned:
- A canonical cross-cube member reference ({customer.c_sk}) travels
with no stash and returns verbatim; a case-variant spelling
({CUSTOMER.c_sk}) would come back canonicalized, so that one keeps
the original.
- A bare identifier that is both a metric's name and another dataset's
field resolves in the metric namespace -- a measure reference, not a
column read that would silently bypass the metric's definition.
Stale prose retired: _MEASURE_NATIVE_KEYS no longer claims extra keys
force a whole-measure copy; the README's stashed-on-import list moves
sub_query out of dimension extras and into parked-whole; the TPC-DS
stash count is 2, not 4; the property generator's single-cube composite
comment no longer says export splits it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verifying the three fixture identities showed the third one leaking: import(export(hand_authored)) gained a model-level stash recording the view export had just generated -- a rendered copy of regeneration's own output, exactly what the extensions-minimization principle forbids, and a foreign-vendor warning for every other spoke downstream. The view builder (generated_view_cubes, uncollided_view_name) moves to _common so import can predict the generated view with export's own code: when the model's sole view is byte-equal to the prediction -- same base cube, same member lists, same prefix/exclude decisions, the canonical path, no leftover meta -- the view set is not stashed and the next export generates it again. Anything off that shape (a curated includes list, an edited prefix, a second view, an off-layout path) is stashed verbatim exactly as before, so a user's edits in Cube are never dropped. Falls out naturally: the TPC-DS view is exactly the generated shape, so its model-level CUBE extension disappears altogether -- the model now carries only the two foreign vendors -- and the mixed-file test's view, also generated-shaped, keeps only its file-layout record. hand_authored round trip is now asserted as whole-document identity, with an edited-view negative pinning the exact-match guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@jbonofre @QMalcolm @khush-bhatia could you please review and approve? |
Summary
Adds a bidirectional converter between Apache Ossie semantic models and Cube data models, under
converters/cube/. Pure offline YAML transform — no Cube deployment, API token, or network access required, matching the other Python converters in this repo.ossie-cube import): Cube files → Ossie. Cube-only constructs (segments, pre-aggregations, hierarchies, folders, view curation, formats, access policies, …) are preserved incustom_extensions[CUBE], so Cube → Ossie → Cube is lossless.ossie-cube export): Ossie → Cube files. Cube has ametafield at every level, so Ossie constructs Cube has no slot for (unique_keys, foreign-vendorcustom_extensions, the structured form ofai_context) are parked undermeta.ossierather than dropped — making Ossie → Cube → Ossie lossless too.The Ossie
semantic_modelmaps to a Cube view, not a cube. Cube users are view-first, and Cube's own AI agent readsmeta.ai_contextonly from views and individual members — cube-level AI context is explicitly not consumed — so the view is the natural model boundary.Fan-out semantics
Cube corrects for join row-multiplication at query time: when a cube sits on the multiplied side of a join it builds
SELECT DISTINCT <primary key> FROM <join>, joins that key set back to the measure's own cube, and aggregates there, so each source row is counted once. A static Ossie expression has no way to inherit that. Cube also refuses outright when the measures themselves span cubes that fan out.So the converter emits the fan-out-safe form wherever one exists, and reports the cases where none exists:
countCOUNT(DISTINCT <pk>)count(pk)normally andcount(distinct pk)when multiplied;COUNT(DISTINCT pk)equals bothcount_distinct/count_distinct_approxCOUNT(DISTINCT x)/APPROX_COUNT_DISTINCT(x)min/maxMIN(x)/MAX(x)sum,avg,count+sqlSUM(x),AVG(x),COUNT(x)Only the last row is at risk, and only when its cube is the
to(one) side of a relationship in the model. That is computable from the Ossie graph, and the converter records a structuredFANOUT_UNSAFE_METRICissue naming the metric, the dataset, and the responsible relationship. Refusing the model outright would leave the spoke on the other side with nothing, and these are the metrics most worth converting;--strict-fanoutrestores the refusal, mirroring Cube's own.Going the other way, an Ossie metric combining several aggregates is decomposed into one
public: falsemeasure per aggregate, each declared on the cube its own operand reads, plus atype: numbermeasure referencing them. Cube's correction then applies per aggregate instead of once for the whole expression. The parts carrymeta.ossie.part_of, so import skips them and inlines their SQL back through the references, recovering the original expression exactly.This points at a spec gap. Ossie has no additivity or grain declaration to record non-additivity properly. dbt's
non_additive_dimensionis the nearest precedent, and this repo's dbt converter already loses the same information (osi_to_msi.pyhard-codesnon_additive_dimension=None, with a namedCUMULATIVE_SEMANTICS_LOSSissue type). Raised separately as #290.Other design notes
{CUBE.member}when the dataset declares a field of that name (reuses the member's SQL, compile-time checked),{CUBE}.columnfor a raw physical column, and{other_cube.member}across cubes — which is also what gives a cross-dataset Ossie metric its implicit join. The cube's own name is never emitted, so models surviveextends.{other_measure}references, because that is what Cube itself does; Ossie has no metric-to-metric reference. Cycles are rejected. Locating the aggregates inside a composite expression usessqlglot, already a runtime dependency of the dbt and NVIDIA GSF converters for the same purpose.filtersfold intoCASE WHEN … ENDinside the aggregate, matching Cube's ownapplyMeasureFiltersrendering and the filtered-aggregation idiom the Ossie expression language endorses.type: numberomits Ossiedatatyperather than asserting a precision the model does not carry — Cube collapses Integer/Decimal/Float into one type, and the spec says to omit when unknown. The original type is stashed.type: geodimensions split into<name>_latitude/<name>_longitude, since an Ossie field holds one expression and a geo dimension has two.Unsupported constructs
extends— resolving it means reproducing Cube's definition-merge semantics exactly, so it is refused rather than half-applied..js/.tsmodels — preserved verbatim and never half-converted. Jinja is detected per file, the same rule Cube's ownCubeSchemaConverteruses for the Rollup Designer.Losses the converter can absorb are returned as structured
ConverterIssues (following theosi-dbtconverter) rather than printed to stderr and forgotten, so a pipeline can gate on them.Testing
226 tests, 96% line coverage:
examples/tpcds_semantic_model.yamlas the converter guide asks;core-spec/osi-schema.json;hypothesisis unavailable.Also verified by hand against a real production Cube model: round trip content-identical with original filenames preserved, output passing
validation/validate.py, and the fan-out guard correctly flagging the two measures on the joined cube'soneside.Related Issues
Related to #290 (spec has no way to declare a non-additive metric).
Checklist
Specification
core-spec/changesOntology
ontology/changesConverters
CUBEregistered in the supported-vendors table inconverters/README.mdValidation
validation/changes; emitted models are validated by the existingvalidation/validate.pyDocumentation
converters/cube/README.mddocuments the full mapping, the fan-out behavior, requirements, and limitationsExamples
Tests
Compliance
jsonschemadev dependency already ships inconverters/orionbelt/AI assistance
This contribution was developed with AI assistance (Claude). I have reviewed the code and tests and take responsibility for them, per the ASF Generative Tooling Guidance.