Skip to content

Add bidirectional Apache Ossie <-> Cube converter - #289

Open
MikeNitsenko wants to merge 54 commits into
apache:mainfrom
MikeNitsenko:feature/cube-converter
Open

Add bidirectional Apache Ossie <-> Cube converter#289
MikeNitsenko wants to merge 54 commits into
apache:mainfrom
MikeNitsenko:feature/cube-converter

Conversation

@MikeNitsenko

@MikeNitsenko MikeNitsenko commented Jul 29, 2026

Copy link
Copy Markdown

Summary

Adds a bidirectional converter between Apache Ossie semantic models and Cube data models, under converters/cube/. Pure offline YAML transform — no Cube deployment, API token, or network access required, matching the other Python converters in this repo.

  • Import (ossie-cube import): Cube files → Ossie. Cube-only constructs (segments, pre-aggregations, hierarchies, folders, view curation, formats, access policies, …) are preserved in custom_extensions[CUBE], so Cube → Ossie → Cube is lossless.
  • Export (ossie-cube export): Ossie → Cube files. Cube has a meta field at every level, so Ossie constructs Cube has no slot for (unique_keys, foreign-vendor custom_extensions, the structured form of ai_context) are parked under meta.ossie rather than dropped — making Ossie → Cube → Ossie lossless too.

The Ossie semantic_model maps to a Cube view, not a cube. Cube users are view-first, and Cube's own AI agent reads meta.ai_context only from views and individual members — cube-level AI context is explicitly not consumed — so the view is the natural model boundary.

Fan-out semantics

Cube corrects for join row-multiplication at query time: when a cube sits on the multiplied side of a join it builds SELECT DISTINCT <primary key> FROM <join>, joins that key set back to the measure's own cube, and aggregates there, so each source row is counted once. A static Ossie expression has no way to inherit that. Cube also refuses outright when the measures themselves span cubes that fan out.

So the converter emits the fan-out-safe form wherever one exists, and reports the cases where none exists:

Cube measure Ossie expression Safe under fan-out?
bare count COUNT(DISTINCT <pk>) Yes, exactly — Cube renders count(pk) normally and count(distinct pk) when multiplied; COUNT(DISTINCT pk) equals both
count_distinct / count_distinct_approx COUNT(DISTINCT x) / APPROX_COUNT_DISTINCT(x) Yes, inherently
min / max MIN(x) / MAX(x) Yes — idempotent under duplication
sum, avg, count + sql SUM(x), AVG(x), COUNT(x) No

Only the last row is at risk, and only when its cube is the to (one) side of a relationship in the model. That is computable from the Ossie graph, and the converter records a structured FANOUT_UNSAFE_METRIC issue naming the metric, the dataset, and the responsible relationship. Refusing the model outright would leave the spoke on the other side with nothing, and these are the metrics most worth converting; --strict-fanout restores the refusal, mirroring Cube's own.

Going the other way, an Ossie metric combining several aggregates is decomposed into one public: false measure per aggregate, each declared on the cube its own operand reads, plus a type: number measure referencing them. Cube's correction then applies per aggregate instead of once for the whole expression. The parts carry meta.ossie.part_of, so import skips them and inlines their SQL back through the references, recovering the original expression exactly.

This points at a spec gap. Ossie has no additivity or grain declaration to record non-additivity properly. dbt's non_additive_dimension is the nearest precedent, and this repo's dbt converter already loses the same information (osi_to_msi.py hard-codes non_additive_dimension=None, with a named CUMULATIVE_SEMANTICS_LOSS issue type). Raised separately as #290.

Other design notes

  • Member references follow Cube's semantics rather than being uniform: {CUBE.member} when the dataset declares a field of that name (reuses the member's SQL, compile-time checked), {CUBE}.column for a raw physical column, and {other_cube.member} across cubes — which is also what gives a cross-dataset Ossie metric its implicit join. The cube's own name is never emitted, so models survive extends.
  • Calculated measures inline their {other_measure} references, because that is what Cube itself does; Ossie has no metric-to-metric reference. Cycles are rejected. Locating the aggregates inside a composite expression uses sqlglot, already a runtime dependency of the dbt and NVIDIA GSF converters for the same purpose.
  • Measure filters fold into CASE WHEN … END inside the aggregate, matching Cube's own applyMeasureFilters rendering and the filtered-aggregation idiom the Ossie expression language endorses.
  • Dimension type: number omits Ossie datatype rather than asserting a precision the model does not carry — Cube collapses Integer/Decimal/Float into one type, and the spec says to omit when unknown. The original type is stashed.
  • type: geo dimensions split into <name>_latitude / <name>_longitude, since an Ossie field holds one expression and a geo dimension has two.

Unsupported constructs

  • extends — resolving it means reproducing Cube's definition-merge semantics exactly, so it is refused rather than half-applied.
  • Jinja-templated YAML and .js/.ts models — preserved verbatim and never half-converted. Jinja is detected per file, the same rule Cube's own CubeSchemaConverter uses for the Rollup Designer.

Losses the converter can absorb are returned as structured ConverterIssues (following the osi-dbt converter) rather than printed to stderr and forgotten, so a pipeline can gate on them.

Testing

226 tests, 96% line coverage:

  • example-based unit tests per direction, plus CLI behavior tests;
  • fixture round-trip tests in both directions, including a TPC-DS model derived from examples/tpcds_semantic_model.yaml as the converter guide asks;
  • every emitted Ossie document validated against core-spec/osi-schema.json;
  • Hypothesis property-based round-trip tests over generated Cube models, with a seeded fallback so the properties still run where hypothesis is unavailable.

Also verified by hand against a real production Cube model: round trip content-identical with original filenames preserved, output passing validation/validate.py, and the fan-out guard correctly flagging the two measures on the joined cube's one side.

Related Issues

Related to #290 (spec has no way to declare a non-additive metric).

Checklist

Specification

  • N/A — no core-spec/ changes

Ontology

  • N/A — no ontology/ changes

Converters

  • New converters include tests under the converter's test directory
  • CUBE registered in the supported-vendors table in converters/README.md

Validation

  • N/A — no validation/ changes; emitted models are validated by the existing validation/validate.py

Documentation

  • converters/cube/README.md documents the full mapping, the fan-out behavior, requirements, and limitations

Examples

  • N/A — no new spec constructs; the existing TPC-DS example is used as the test baseline

Tests

  • All existing tests pass (226 tests, 96% coverage; CI workflow added)
  • New functionality is covered by tests

Compliance

  • ASF license headers are present on all new source files
  • No new runtime dependencies (PyYAML only, as with the other Python converters). The jsonschema dev dependency already ships in converters/orionbelt/

AI assistance

This contribution was developed with AI assistance (Claude). I have reviewed the code and tests and take responsibility for them, per the ASF Generative Tooling Guidance.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new converters/cube/ Python converter that round-trips between Apache Ossie semantic models and Cube YAML data models, including a CLI and extensive tests/fixtures, and registers CUBE in the supported-vendors list.

Changes:

  • Introduces bidirectional conversion logic (convert_cube_to_ossie / convert_ossie_to_cube) with stash/parking mechanisms to preserve unmapped constructs and enable lossless round-trips.
  • Adds a full test suite (fixtures, property-based tests with Hypothesis fallback, CLI tests) plus Cube converter documentation and packaging (pyproject.toml).
  • Adds a dedicated GitHub Actions workflow to run Cube converter tests in CI and updates converters/README.md to list CUBE.

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
converters/README.md Registers CUBE as a supported vendor extension.
converters/cube/src/ossie_cube/_common.py Shared conversion utilities (YAML handling, stash protocol, expression translation, mappings).
converters/cube/src/ossie_cube/cube_to_osi.py Cube → Ossie conversion implementation (datasets/fields/relationships/metrics + preservation).
converters/cube/src/ossie_cube/osi_to_cube.py Ossie → Cube export implementation (layout, meta parking, joins/measures/view generation).
converters/cube/src/ossie_cube/converter_issues.py Structured issue types + issue log for lossy/unsafe conversions.
converters/cube/src/ossie_cube/cli.py ossie-cube CLI: import/export behavior, IO, error reporting.
converters/cube/src/ossie_cube/init.py Public API surface for the converter package.
converters/cube/README.md Converter documentation: mapping table, fan-out semantics, usage, limitations.
converters/cube/pyproject.toml Packaging + dev dependencies/test config for the converter.
converters/cube/tests/** Comprehensive unit, fixture round-trip, property-based, edge-case, and CLI tests.
converters/cube/tests/fixtures/** Cube model fixtures used for round-trip and baseline tests (including TPC-DS).
.github/workflows/converter-cube-ci.yml CI workflow for the Cube converter test suite.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread converters/cube/tests/test_roundtrip_properties.py Outdated
Comment thread converters/cube/src/ossie_cube/osi_to_cube.py Outdated
MikeNitsenko added a commit to MikeNitsenko/ossie that referenced this pull request Jul 30, 2026
Both from the Copilot review on apache#289.

Dimension names were sanitized separately in _convert_model (to decide
which members a cube has) and again in _build_dimensions (to name them).
The first pass used a fresh `taken` set per field, so a collision was
silently swallowed by a set comprehension there and only rejected later
in the second pass -- meaning the member set that decides
`{CUBE.member}` vs `{CUBE}.column`, and where a measure lands, could be
short a name while measures were being placed. Demonstrated: "Order
Status" and "order status" collapsed to one name with no error.

Now resolved once in _resolve_dimension_names and reused. That also fixes
a defect the review did not mention: the old set included the two halves
of a split geo dimension (location_latitude, location_longitude), which
never exist as Cube dimensions since they merge back into `location`, so a
metric referencing one would emit an unresolvable `{CUBE.location_…}`.
The halves now resolve to the dimension they merge into.

_HypothesisRnd.chance() ignored its `p` argument and always drew an
unweighted boolean, so the Hypothesis driver explored a different
distribution than the seeded one despite the docstring claiming they share
a generator. Now weighted, and drawn so the minimal value means False --
shrinking toward the smallest model rather than the largest.

231 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@MikeNitsenko

Copy link
Copy Markdown
Author

Both addressed in d362c9e.

members_by_cube collisions. Correct, and worse than described. The two sanitization passes could disagree: {"Order Status", "order status"} collapsed to a single order_status with no error in _convert_model, and was only rejected later in _build_dimensions — after measures had already been placed against the short member set.

Names are now resolved once in _resolve_dimension_names and reused by every stage, so sanitization and collision detection happen in exactly one place. Added a test that a collision is rejected with a metric present, which is the ordering that was fragile.

This also fixed something not mentioned: the old set included both halves of a split geo dimension (location_latitude, location_longitude). Those never exist as Cube dimensions — they merge back into location — so a metric referencing one emitted an unresolvable {CUBE.location_latitude}. The halves now resolve to the dimension they merge into.

_HypothesisRnd.chance() ignoring p. Also correct. Now weighted, and drawn so the minimal value means False, which shrinks toward the smallest model rather than the largest. Verified the realised probability matches p exactly for every value the builders use.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

converters/cube/src/ossie_cube/osi_to_cube.py:721

  • Using queue.pop(0) makes this BFS O(n²) due to repeated list shifting; on larger relationship graphs this can become unnecessarily slow. Use an index cursor (or a deque) to avoid O(n) pops from the front.
    queue = [base]
    while queue:
        current = queue.pop(0)
        for neighbor in adjacency.get(current, []):
            if neighbor in paths:
                continue
            paths[neighbor] = f"{paths[current]}.{neighbor}"
            entries.append({"join_path": paths[neighbor], "includes": "*"})
            queue.append(neighbor)

converters/cube/tests/test_roundtrip.py:42

  • Typo: "licence" is misspelled here (the rest of the repo uses "license").
    licence headers on the fixtures) are not part of the data model, and key order

MikeNitsenko added a commit to MikeNitsenko/ossie that referenced this pull request Jul 30, 2026
Both from the Copilot review on apache#289.

Dimension names were sanitized separately in _convert_model (to decide
which members a cube has) and again in _build_dimensions (to name them).
The first pass used a fresh `taken` set per field, so a collision was
silently swallowed by a set comprehension there and only rejected later
in the second pass -- meaning the member set that decides
`{CUBE.member}` vs `{CUBE}.column`, and where a measure lands, could be
short a name while measures were being placed. Demonstrated: "Order
Status" and "order status" collapsed to one name with no error.

Now resolved once in _resolve_dimension_names and reused. That also fixes
a defect the review did not mention: the old set included the two halves
of a split geo dimension (location_latitude, location_longitude), which
never exist as Cube dimensions since they merge back into `location`, so a
metric referencing one would emit an unresolvable `{CUBE.location_…}`.
The halves now resolve to the dimension they merge into.

_HypothesisRnd.chance() ignored its `p` argument and always drew an
unweighted boolean, so the Hypothesis driver explored a different
distribution than the seeded one despite the docstring claiming they share
a generator. Now weighted, and drawn so the minimal value means False --
shrinking toward the smallest model rather than the largest.

231 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MikeNitsenko added a commit to MikeNitsenko/ossie that referenced this pull request Jul 30, 2026
Both from the Copilot review on apache#289. The BFS popped from the front of a
list, which is O(n) per pop; a deque makes it O(1). Semantic models are
small enough that this was never going to matter in practice, but the
deque is also the more idiomatic form.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@MikeNitsenko
MikeNitsenko force-pushed the feature/cube-converter branch from 508c9f5 to 7d981c9 Compare July 30, 2026 07:47
@MikeNitsenko
MikeNitsenko requested a review from Copilot July 30, 2026 08:55

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

converters/cube/src/ossie_cube/osi_to_cube.py:685

  • When the model has foreign-vendor custom_extensions but the imported Cube model had multiple views and none was selected (mapped_view is missing), export currently drops those extensions while logging PARKED_IN_META. This causes avoidable data loss and the issue type/message is inconsistent ("parked" vs "dropped"). Prefer parking the extensions on a deterministic view (or failing fast) so Ossie -> Cube stays lossless even in the "no mapped view" case.
        if foreign and mapped is None:
            issues.add(IssueType.PARKED_IN_META, "model",
                       "no mapped view to park foreign-vendor custom_extensions on; "
                       "they have no Cube home and are dropped")

converters/cube/README.md:279

  • The README hard-codes an exact test count ("234 tests"), but the PR description claims a different number. Since this value will drift over time, it’s better to avoid a specific count (or generate it automatically) to prevent documentation from becoming stale.
234 tests at 96% line coverage: example-based unit tests per direction, CLI
behavior tests, fixture round-trip tests (including the

MikeNitsenko added a commit to MikeNitsenko/ossie that referenced this pull request Jul 30, 2026
Both from the Copilot review on apache#289.

Model-level foreign-vendor custom_extensions ride on the view that
represents the model. When the source Cube model had several views and
none was chosen, there is no such view -- and export silently dropped
them. Reachable in practice: import a multi-view Cube model, add a
SNOWFLAKE extension to the Ossie model, export, and it is gone.
Confirmed by reproducing it.

Now refused, with the fix in the message (re-import with `--view`).
Parking on an arbitrary view was considered and rejected: only the mapped
view's parked extensions are read back on import, so it would look
lossless while still losing them.

The review also noted the issue type contradicted its own message --
PARKED_IN_META for something reported as "dropped". That was true in two
places, not one, and it matters: the README defines PARKED_IN_META as
preserved-but-invisible-to-Cube, so a pipeline gating on issue types would
have concluded the data survived. Adds DROPPED_NO_CUBE_EQUIVALENT for
values that genuinely cannot be preserved, and uses it for relationship
ai_context -- a Cube join entry takes only name/sql/relationship, with no
`meta` field, making it the one construct with nowhere to go.

Also drops the hard-coded test count from the README. It had already
drifted out of sync with the PR description, which is the reviewer's point:
the number carries no information a reader needs, while the description of
what the suite covers does.

236 tests, 97% line coverage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@MikeNitsenko

Copy link
Copy Markdown
Author

Both worth addressing, and the first one was a real bug. Fixed in 31d49d2.

Dropped foreign-vendor extensions. Reproduced it: import a Cube model with two views (so no view is mapped), add a SNOWFLAKE extension to the Ossie model, export — and the extension is gone, reported under an issue type that says "parked".

Now refused, with the fix in the message (--view <name> on the import so a view is mapped). I considered parking on a deterministic view as suggested, but rejected it: import only reads parked extensions back from the mapped view, so that would have looked lossless while still losing them. Refusing is the honest option and matches the converter's stated contract of never dropping a field silently.

The type/message inconsistency was in two places, not one. PARKED_IN_META is documented as preserved-but-invisible-to-Cube, so reporting a genuine drop under it would let a caller gating on issue types conclude the data survived. Added DROPPED_NO_CUBE_EQUIVALENT and used it for relationship ai_context — a Cube join entry takes only name/sql/relationship, with no meta, which makes it the one construct here with nowhere to go.

Hard-coded test count. Removed from the README. It had already drifted out of sync with the PR description, which is exactly the point — the number carries no information a reader needs, while the description of what the suite covers does.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 1 comment.

Comment thread converters/cube/src/ossie_cube/osi_to_cube.py Outdated
MikeNitsenko added a commit to MikeNitsenko/ossie that referenced this pull request Jul 30, 2026
From the Copilot review on apache#289, which found that the placeholder holding a
geo dimension's position was only reserved when the half encountered first
happened to be `latitude`. With `longitude` first, the recorded index
pointed at whatever real dimension had already been appended, and
`dimensions[index] = dim` overwrote it. Reproduced: a `city` dimension
between the two halves disappeared from the output entirely.

Rather than reserve the placeholder earlier, the index arithmetic is gone.
Dimensions are now built into a dict keyed by target name, with order taken
from each name's first appearance -- which is well defined however the two
halves are arranged, adjacent or not, in either order.

Probing around the fix turned up two more silent-corruption paths in the
same code, both order-dependent:

- A geo base colliding with an ordinary field of the same name emitted two
  dimensions called `home` (invalid Cube) when the ordinary field came
  first, but was correctly rejected when it came second. Now checked during
  name resolution, so order does not decide.
- Two fields both claiming the same half silently discarded one. Now
  rejected.

Also validates the geo `part` and `of` values, and moves the missing-half
check into name resolution so every geo problem is caught in one place
before anything is built.

241 tests, 97% line coverage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@MikeNitsenko

Copy link
Copy Markdown
Author

Confirmed and fixed in d92711e. Reproduced it first — with longitude first and a city dimension between the halves, city disappeared from the output entirely:

emitted dimensions:
   {'name': 'home', 'type': 'geo', 'latitude': {...}, 'longitude': {...}}
'city' present? False

I took the second of your two suggestions rather than the first. Reserving the placeholder earlier would have worked, but the index arithmetic was the fragile part, so it's gone: dimensions are built into a dict keyed by target name, with order taken from each name's first appearance. That's well defined however the halves are arranged — either order, adjacent or not.

Probing around it turned up two more order-dependent paths in the same code:

  • A geo base colliding with an ordinary field of the same name. With the ordinary field first, export emitted two dimensions called home (invalid Cube); with it second, sanitize_name correctly rejected it. Order was deciding whether an invalid model was produced. Now checked during name resolution.
  • Two fields both claiming the same half silently discarded one. Now rejected.

Also validated the geo part and of values, and moved the missing-half check into name resolution so every geo problem is caught in one place before anything is built.

Six cases pinned by tests: either order, collision in both orders, duplicate half, missing half, unknown part.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 1 comment.

Comment thread converters/cube/src/ossie_cube/osi_to_cube.py
MikeNitsenko added a commit to MikeNitsenko/ossie that referenced this pull request Jul 30, 2026
From the Copilot review on apache#289: additional `semantic_model` entries were
reported as PARKED_IN_META, but they are neither converted nor preserved
anywhere -- a drop.

That is the third instance of the same mislabelling, so rather than patch
the flagged line I audited all seven export-side uses. Exactly one was a
genuine park:

  unique_keys                        -> PARKED_IN_META    (correct)
  extra semantic_model entries       -> DROPPED           (was parked)
  dimension.is_time role             -> DROPPED           (was parked)
  dimension.is_time opt-out          -> DROPPED           (was parked)
  synthesized primary-key dimension  -> APPROXIMATED      (was parked)
  no datatype -> Cube type 'string'  -> APPROXIMATED      (was parked)
  cross-dataset metric placement     -> APPROXIMATED      (was parked)

The import-direction uses were all genuine parks and are unchanged.

Adds APPROXIMATED for the middle case, which neither of the existing types
described: nothing is lost and nothing is hidden, but Cube requires a value
Ossie does not carry, so the converter chose one and the output asserts
slightly more than the input did. Calling that "parked" was wrong in the
same way as calling a drop "parked" -- nothing was parked.

The point of keeping three types apart is that a caller gating on them can
distinguish preserved-but-unreadable from actually-lost from
emitted-with-a-guess. Two of the three could not be told apart before.

241 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@MikeNitsenko

Copy link
Copy Markdown
Author

Correct, and it was the third instance of this — so rather than patch the flagged line I audited all seven export-side uses. Exactly one was a genuine park. Fixed in c22e6d2.

site was now
unique_keys parked parked (correct)
extra semantic_model entries parked dropped
dimension.is_time role parked dropped
dimension.is_time opt-out parked dropped
synthesized primary-key dimension parked approximated
no datatype → Cube type: string parked approximated
cross-dataset metric placement parked approximated

The import-direction uses were all genuine parks and are unchanged.

Added APPROXIMATED for the middle group, which neither existing type described: nothing is lost and nothing is hidden, but Cube requires a value Ossie does not carry, so the converter chose one and the output asserts slightly more than the input did. Labelling that "parked" was wrong in the same way as labelling a drop "parked" — nothing was parked in either case.

Three types kept apart on purpose, since that is what a caller gating on them needs: preserved-but-unreadable-by-Cube, actually lost, and emitted-with-a-guess. Two of the three were indistinguishable before.

Found the same way as the Databricks issues: by checking what a spoke actually made of
our output rather than that it exited zero. Cube -> Ossie -> Snowflake produced a Cortex
Analyst model with **zero dimensions and 27 facts** across TPC-DS -- every categorical
column classified as a numeric measure.

The cause is on our side. The Snowflake converter classifies "a field with no `dimension`
block as a fact regardless of datatype", which is a fair reading: the block is the role
marker. Import emitted it only for time dimensions, so everything else looked like a
fact. A Cube `dimensions:` entry is a dimension by definition, so the block is now always
emitted -- empty for a non-time one, which leaves the consumer to apply the spec's own
default instead of this converter asserting `is_time: false`.

Snowflake output for the same model, before -> after:

  store_sales  dim=0 fact=9   ->  dim=9 fact=0
  customer     dim=0 fact=6   ->  dim=6 fact=0
  date_dim     dim=0 time=3 fact=2 -> dim=2 time=3 fact=0

which matches the shape of that converter's own committed example. No other spoke's
result changed.

Also covers the geo halves, whose fields are built on a separate path; and pins that the
dialect fallback is not Databricks-specific -- Snowflake and BigQuery alone convert too.
The two Ossie snapshot fixtures are regenerated.

559 tests with both gates, 533 with neither, 96% coverage.
All three blockers are regressions from the previous two commits: each fixed the forward
direction by making a choice Cube requires, and each choice was one-way, so
`Ossie -> Cube -> Ossie` no longer returned the document it was given.

The pattern is the one the rest of the converter already uses -- record the choice in
`meta.ossie`, undo it on the way back:

- A warehouse dialect used in place of ANSI is recorded, so re-import labels the SQL as
  that dialect instead of calling vendor SQL `ANSI_SQL`. On measures as well as fields;
  metrics carry expressions too.
- A `unique_keys` entry promoted to satisfy Cube's join requirement is recorded, so
  re-import does not hand back a declared `primary_key` the model never had. The
  dimension the promotion synthesized is recorded too, so it does not come back as a
  field for a column the Ossie model never described.
- An Ossie field with no `dimension` block is recorded, so it returns as the fact it was
  rather than as a dimension. Cube has one kind of dimension, so the block still goes out
  on every member -- that is what the Snowflake classification needs.

`test_a_model_from_another_converter_survives_the_round_trip_exactly` pins all of it on
the committed Databricks-authored fixture: dialects, keys, fields and roles compared
before and after. It would have failed on each of the three.

Worth noting why the property tests missed these: the generator draws ANSI expressions,
declares a primary key, and gives every field a dimension role -- so none of the three
shapes can occur in a generated model. The fixture from another converter is the only
thing in the suite that has them.

560 tests with both gates, 533 with neither, 96% coverage. Snowflake classification and
the interop matrix unchanged.
Three review rounds found bugs in one blind spot: the generator drew ANSI expressions,
always declared a `primary_key`, and gave every field a `dimension` role -- so none of the
210 generated models per run could contain the shapes that were breaking. The committed
Databricks-authored fixture was the only thing in the suite that had them, which is why
the same class of defect came back three times.

The generator now draws all three, since each is a place where export must make a choice
Cube requires and then be able to undo it:

- a dialect per field and per metric, often a warehouse one with no ANSI alternative;
- either `primary_key` or `unique_keys` (never neither -- Cube rightly refuses a cube
  with a join and no key);
- a `dimension` role or none, the latter being a fact.

And the property compares what those choices affect -- dialects, keys, roles and
datatypes -- not just expressions, which is how one-way fixes slipped past it before.

Checked that it can fail: reverting each of the three provenance records in turn breaks
61, 61 and 34 of the 122 property cases. A green test that cannot fail is not a test.

560 tests with both gates, 534 with neither, 96% coverage.
Both blockers were provenance recorded by halves.

- Ossie names a primary key by *column*; Cube marks a *dimension*. The two differ
  whenever the dimension carrying the key is not named after its column -- a field
  `order_id` reading column `id`, or a synthesized `id_pk` where a computed field shadows
  the column -- and import rebuilt the key from dimension names, so it came back naming
  something the table need not have. The column list is recorded when it cannot be read
  back off the dimensions, and only then, so a model whose names already agree keeps a
  clean Cube round trip. `_primary_key_of` returns columns now, which also fixes the
  rebuilt `COUNT(DISTINCT ...)`: it was naming the synthesized dimension, a member the
  Ossie side does not have at all.

- Recording only the chosen dialect's *name* lost the alternatives. Cube holds one `sql`
  per member, so nothing short of the whole expression object brings a multi-dialect
  expression back; it is parked entire, on measures as well as fields.

The generator now draws several dialects per expression and the property compares them
all rather than `dialects[0]`, which is what let the second one through. That immediately
found two more:

- an expression offering two warehouse dialects and no ANSI was dropped outright, because
  the fallback insisted on a sole candidate. It takes the first in document order and
  reports it -- Cube passes SQL to one data source, and the alternatives are parked.
- `COUNT(DISTINCT <pk>)` drifted to the synthesized dimension name, above.

Also a flaw in the generator itself: it picked an alternative dialect out of a `set`,
whose iteration order varies between processes, so the seeded sweep produced different
models each run and could not name a reproducible seed. Sorted now -- the same suite ran
green and red in consecutive invocations before this.

Checked both fixes can fail: reverting each breaks 60 and 9 of 122 property cases. Metric
drift across 400 generated models is zero.

560 tests with both gates, 534 with neither, 97% coverage.
[P1] The recorded key column list is *columns*, but the `computed_primary_key` inference
read it as dimension names -- so a key column `id` alongside a computed field also named
`id` came back flagged as computed, and the second export marked `LOWER(email)` as the key
instead of synthesizing `id_pk`. Cube then deduplicated on a different value, which changes
the counts it returns. The inference is skipped when `meta.ossie.primary_key` supplied the
key, because those entries are columns by construction.

Both Ossie documents were identical in that case; only the *Cube* model changed. So the
property now runs a second export and requires it to reproduce the first, which is the only
way to see a record that one side writes and the other reads differently. That found two
more, neither reachable in a single cycle:

- A decomposed metric's public measure was stashed verbatim and restored with references
  to hidden parts the next export no longer generated. Cube's verdict on the second cycle:
  "fact.crossing_part_1 cannot be resolved" -- a broken model. The public half is marked,
  so re-import rebuilds it from its expression and both halves are regenerated together.
- `COUNT(DISTINCT DIM_0.ID)` was not recognized as the primary-key count because the
  comparison was case-sensitive, so cycle 1 emitted `count_distinct` and cycle 2 -- reading
  a canonically regenerated expression -- emitted the bare `count`. Compared on normalized
  identifiers now, which also means a metric spelling the key in any case gets Cube's
  fan-out-safe form.

Non-blocking wording fixed too: the fallback may pick the first of several warehouse
dialects, not only a sole one.

Checked the new checks can fail: reverting the inference fix breaks 9 cases across the
property sweep and the targeted two-cycle test.

561 tests with both gates, 534 with neither, 97% coverage.
@MikeNitsenko

Copy link
Copy Markdown
Author

PR #289 — reviewer note

Tests: 534, or 561 with the optional Cube gate. Coverage: 97%.

Layers:

  • Three whole models: a small one, TPC-DS (as requested by the converter guide), and one produced by the Databricks converter, where nothing was written “for Cube.” They are compared as whole documents, so any unintended change appears as a readable diff.
  • Feature matrix: one fixture per Cube feature, ten in total: case/switch dimensions, granularities, presentation metadata, measure variants, hierarchies, segments, pre-aggregations, access policies, view curation, sub_query, and computed primary key. Each fixture is a valid Cube model and is asked the same four questions: does it convert; is the Ossie output valid according to the spec; does Cube → Ossie → Cube reproduce it; and does Cube still compile the result.
  • Property tests from both ends: 60 seeds plus 150 generated cases per direction. Generating only from Cube says nothing about hand-authored Ossie, and that is the harder direction.
  • Cube itself as a gate: tools/cube_compile.js asks Cube whether what we emit actually loads. It requires a locally built Cube checkout (OSSIE_CUBE_REPO); without one, those tests are skipped, so CI is unaffected.
  • Idempotence: a second export cycle must produce the same Cube model as the first. This is the only way to catch a record that one side writes and the other reads differently.

Cross-matrix (tools/interop_matrix.py): our Ossie output is run through all nine Python converters, with a report showing what each one made of it. This is what the stash reduction was measured against: 7 CUBE entries on TPC-DS instead of 41, and 2 Databricks warnings instead of 32.

Databricks — both directions:

  • Cube → Ossie → metric view.
  • metric view → Ossie → Cube, using all three fixtures from that converter; the result compiles in Cube.
  • Full loop: metric view → Ossie → Cube → Ossie → metric view. Source, joins, dimensions, and measures come back with the same names.

Snowflake: that converter is export-only, so there is no reverse path in the repository. One direction was verified: Cube → Ossie → Cortex Analyst.

These interop paths are what the tests specifically pin down: which dialect an expression carries, where the primary key comes from, and whether a field has a dimension role. Each is a point where the two formats disagree and the conversion must make an explicit choice.

Limitations:

  • Ossie cannot declare a measure as non-additive, so the fan-out risk can only be reported, not recorded in the model — issue Spec has no way to declare a non-additive metric #290.
  • A metric view does not carry a primary key, while Cube requires one on any cube with a join. It must therefore be added manually using primary_key or unique_keys. We report this using Cube’s own error message.
  • The compile gate requires a local Cube checkout, so it is skipped in CI.
  • Polaris and Salesforce are written in Java and require Maven. They are listed in the matrix as unsupported rather than being silently skipped.
  • dbt fails in the matrix because of its own CLI bug — issue ossie-dbt osi-to-msi crashes: PydanticSemanticManifest has no model_dump_json #296, not ours.

What I need: CI has never run on this PR because GitHub holds workflows from external contributors until a committer approves them. The PR needs a review and a CI run.

@MikeNitsenko
MikeNitsenko requested a lite review from Copilot August 5, 2026 08:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 48 changed files in this pull request and generated no new comments.

Suppressed comments (5)

converters/cube/tools/cube_compile.js:1

  • Using path.basename(p) for fileName can cause collisions when compiling multiple files with the same basename (e.g. model/cubes/orders.yml and model/views/orders.yml). Cube keys schema files by fileName, so this can make the compile check fail or (worse) compile the wrong combined model. Use a stable unique key such as the path relative to a common root (or the provided relative model path) for fileName, and ensure localPath matches that root.
    converters/cube/tools/interop_matrix.py:1
  • This relies on the implicit boolint coercion (True == 1). It works, but it’s easy to misread/accidentally change. Consider making the increment explicit (e.g., if result == 'FAIL': failures += 1) for clarity.
    converters/cube/src/ossie_cube/expressions.py:310
  • As written, any whitespace at depth 0 makes this return True, including leading/trailing spaces. That can cause unnecessary parentheses when inlining an otherwise single-term expression like 'SUM(x) ' (or any expression with incidental formatting). A simple fix is to scan str(expr).strip() and/or treat whitespace as structural only when it participates in a token pattern (e.g., CASE, WHEN, AND, OR) rather than any space.
def has_top_level_operator(expr):
    """True if `expr` is not a single self-contained term.

    Used to decide whether inlining it back into a larger expression needs
    parentheses: a lone `SUM(x)` does not, `SUM(x) / 2` does.
    """
    depth, quote = 0, None
    for ch in str(expr):
        if quote:
            if ch == quote:
                quote = None
        elif ch in "'\"":
            quote = ch
        elif ch == "(":
            depth += 1
        elif ch == ")":
            depth -= 1
        elif depth == 0 and (ch in "+-*/%<>=|&" or ch.isspace()):
            # Whitespace at depth 0 also implies structure (`CASE WHEN ...`).
            return True
    return False

converters/cube/tests/_roundtrip_helpers.py:318

  • This hard-codes the Ossie version string (0.2.0.dev0) in the generator. If OSSIE_VERSION changes, these generated models will drift from the rest of the converter/tests and can cause confusing failures. Prefer importing and using OSSIE_VERSION (as other tests do) to keep versioning consistent.
def build_ossie_model(rnd):
    """Generate a hand-authored Ossie model (no stash) as a YAML string."""
    dim_names = [f"dim_{i}" for i in range(rnd.count(1, 2))]
    fact = "fact"

    lines = ["version: 0.2.0.dev0", "semantic_model:", "- name: shop"]

converters/cube/src/ossie_cube/cli.py:145

  • File I/O here (and in export) relies on platform default encodings. For deterministic behavior across OSes/locales, it’s better to open text files with an explicit encoding (typically UTF-8) when reading and writing YAML/model files.
    with open(path) as fh:
        files[rel] = fh.read()

Cube keeps cubes and views in one global namespace, so a view may not share a name with
a cube. The exporter did not check, and an Ossie model named after one of its own
datasets produced exactly that -- Cube rejected the whole model with "Cannot read
properties of undefined (reading 'toString')". Not an exotic input: it is what every
Databricks metric view over a same-named table converts to, and `databricks_ossie.yaml`
is one. The generated view becomes `<name>_view` and the model's own name is recorded in
`meta.ossie.model_name`, since the mapped view's name is the model's name on the way
back and the rename would otherwise stick. Renamed rather than refused because a cube is
addressed by joins and by every member reference, a generated view by nothing.

That went unnoticed because the gate meant to catch it was dropping half its input.
`cube_compile.js` keyed model files by basename; Cube's own FileRepository keys them by
path relative to the model root, so `cubes/orders.yml` and `views/orders.yml` collided
and one was discarded silently. A valid cube plus an invalid same-named view reported
COMPILED OK, while the identical pair under distinct names failed as it should. Keyed by
relative path now, matching Cube, and duplicate keys are refused outright rather than
resolved by chance -- a gate that quietly compiles less than it was given is worse than
no gate. The same flattening was already fixed a layer up in `_cube_gate.py`, where the
temp files are written; the JS undid it.

With the gate honest, a second defect surfaced one cycle out: a `DATABRICKS` metric came
back as `ANSI_SQL` on the second export. The verbatim-restore path hands back the Cube
SQL a previous import stashed instead of picking a dialect, so it had no dialect to pass
to `_park_expression` and the label was dropped. It falls back to the sole declared
dialect, which is the one that SQL came from. `Ossie -> Cube -> Ossie` is byte-stable
from the first cycle now; the existing one-cycle comparison could not see this, since
cycle one was correct.

Also, `validation/validate.py` reports a missing `jsonschema` by calling `sys.exit(1)` at
import time, and SystemExit does not derive from Exception -- so it escaped the guard
around the validator import and aborted pytest *collection*. The whole suite refused to
run on any machine without jsonschema, the exact case `validator_gate` exists to skip.

Smaller, from the same review:

- The CLI's file I/O used the platform default encoding, so a model carrying any
  non-ASCII text died under a non-UTF-8 locale: `title: Größe` gave
  `UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3`. Pinned to UTF-8.
- The property generator hard-coded the spec version instead of using `OSSIE_VERSION`.
- `has_top_level_operator` treated any depth-0 whitespace as structure, including the
  trailing newline off a YAML block scalar, and parenthesized a lone `SUM(x)` needlessly.
- The interop matrix counted failures by adding a bool; made explicit.

Checked the new checks can fail. Reverting the gate keying (and its duplicate guard)
leaves the collision test passing a model Cube refuses; reverting the view rename breaks
3 tests including the Databricks compile; reverting the dialect fallback breaks the
two-cycle stability test; reverting the encoding breaks the non-UTF-8 CLI test.

566 tests with both gates, 399 with neither, 97% coverage.
@MikeNitsenko
MikeNitsenko requested a lite review from Copilot August 5, 2026 09:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 48 changed files in this pull request and generated no new comments.

Suppressed comments (4)

converters/cube/tools/interop_matrix.py:1

  • For directory outputs, dest.rglob('*') yields directories as well as files, so a converter that only created empty directories would be reported as OK instead of EMPTY. To align the result with the column definition ('nothing written'), consider counting only files (or only non-empty files) when is_dir is true.
    converters/cube/tools/interop_matrix.py:1
  • Path.read_text() defaults to the platform encoding, which can break on non-UTF-8 locales (and this repo already has tests ensuring the CLI is robust under non-UTF-8 locales). Consider reading with an explicit encoding='utf-8' (and similarly, any other reads/writes in this tool that should be Unicode-safe) to keep behavior consistent across environments.
    converters/cube/src/ossie_cube/cli.py:104
  • CLI input paths commonly include ~ (home directory). os.path.abspath() does not expand ~, so valid user inputs like -i ~/model will be reported as missing. Consider expanding user input first (e.g., via os.path.expanduser) before resolving to an absolute path.
    resolved = [os.path.abspath(p) for p in paths]

converters/cube/src/ossie_cube/expressions.py:287

  • The containment check re-walks each previously found scope for every node (O(n^2) traversals). If this runs on many expressions/models, it can become a noticeable hotspot. Consider a single-pass approach that tracks whether the current traversal position is inside an aggregate scope (e.g., using parent/ancestor checks if available, or maintaining a stack during a DFS-style walk) so each node is visited a bounded number of times.
def _outermost_aggregate_scopes(tree):
    """Aggregate scopes that are not inside another one.

    Nesting is resolved so an ordered-set aggregate is counted once: `WithinGroup` and
    the `PercentileCont` inside it are one aggregate, and treating the inner one as its
    own scope would find no columns there and blame the declaring cube.
    """
    scopes = []
    for node in tree.walk():
        if not _is_aggregate_scope(node):
            continue
        if any(any(inner is node for inner in scope.walk()) for scope in scopes):
            continue
        scopes.append(node)
    return scopes

[P1] The record added for cube/view collisions was scoped to that one cause, and the
ordinary causes went unrecorded. `Sales Model` is a legal Ossie name that cannot be a Cube
identifier, so it exported as view `sales_model` with nothing parked and came back named
`sales_model`. The same for `--name "Sales Model"` over a stashed view already called
`sales_model`: the stashed branch compared the mapped name against the *sanitized* model
name, the two matched, and the override was silently undone.

Keyed on the difference now rather than on the reason for it -- the raw name is preserved
whenever it differs from the name of the view that will carry it, whether that difference
comes from sanitizing, an override, or a collision. A model whose name is already its
view's name still parks nothing, so an ordinary Cube document stays clean. Verified stable
over three cycles, since the value has to survive being read back out of the stash and not
merely written once.

Also non-blocking, from the same review: `test_two_export_cycles_produce_the_same_cube_model`
was entirely behind the optional Cube gate, but comparing two exports needs no Cube
installation -- so the regression it exists for was unchecked everywhere without a built
checkout, CI included. Split, with only `assert_cube_compiles` gated. An audit of every
`cube_gate` test found one more of mine with the same mistake
(`test_a_renamed_view_still_compiles_and_stays_renamed`); split the same way. The rest are
genuinely Cube-only.

Checked the new checks can fail: restoring the collision-only rule breaks both name tests,
and the two split tests now run (and pass) with no Cube checkout present where they were
previously skipped.

571 tests with both gates, 404 with neither, 97% coverage.
@MikeNitsenko
MikeNitsenko requested a lite review from Copilot August 5, 2026 10:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 48 changed files in this pull request and generated no new comments.

Suppressed comments (4)

converters/cube/src/ossie_cube/expressions.py:104

  • aggregate_spans() parses the same expression twice on the hot path (parse(text) and then is_single_aggregate(text) which re-parses). Cache the parse result (or refactor is_single_aggregate to accept a pre-parsed tree) to avoid redundant sqlglot parsing costs, especially when decomposing many metrics.
    text = str(expr)
    if parse(text) is None or is_single_aggregate(text):
        return []
    return _scan_aggregates(text)

converters/cube/tests/_util.py:43

  • These helpers read fixture files without an explicit encoding. For consistency with the converter’s UTF-8 IO (and to avoid platform/locale-dependent failures), open fixtures with encoding='utf-8' and use path.read_text(encoding='utf-8').
def load_fixture(name):
    with open(FIXTURES / name) as fh:
        return fh.read()


def load_fixture_dir(name):
    """Read a fixture Cube model directory as {relative posix path: text}."""
    root = FIXTURES / name
    files = {}
    for path in sorted(root.rglob("*")):
        if path.is_file():
            files[path.relative_to(root).as_posix()] = path.read_text()
    return files

.github/workflows/converter-cube-ci.yml:53

  • Piping a remote install script directly into sh is a supply-chain risk. Prefer using a pinned, maintained GitHub Action for uv (or pin the installer to a specific version + verify its checksum/signature) to make the CI installation step auditable and harder to tamper with.
      - name: Install uv
        run: |
          curl -LsSf https://astral.sh/uv/install.sh | sh
          echo "${HOME}/.local/bin" >> "${GITHUB_PATH}"

converters/cube/src/ossie_cube/converter_issues.py:114

  • The new dataclass fields are untyped (issues: list, strict_types: frozenset), which makes it harder to use/validate IssueLog from type checkers and IDEs. Consider annotating as issues: list[ConverterIssue] and strict_types: frozenset[IssueType] (and typing method params/returns accordingly).
class IssueLog:
    """Collects issues during a conversion.

    `strict_types` names the issue types that should abort the conversion instead of
    being recorded. Nothing is in there by default: a converter that refuses a whole
    model over one metric leaves the spoke on the other side with nothing. Passing
    `--strict-fanout` adds `FANOUT_UNSAFE_METRIC`, mirroring Cube's own refusal to
    answer a query whose measures reference cubes that lead to row multiplication.
    """

    issues: list = field(default_factory=list)
    strict_types: frozenset = frozenset()

[P1] Model-level metadata has no Cube field of its own, so it rides on the view
representing the model -- and export emitted no view at all when the stash recorded none
mapped. A Cube model need not contain a view, and one with several need not say which is
the model, so this is an ordinary input rather than an edge case. Both cases dropped the
name, description and AI context in silence, with no issue reported: a cube-only model
imported with `--name 'Sales Model'` came back as the synthesized `cube_model`. That
contradicts the documented lossless Ossie -> Cube -> Ossie round trip.

They ride on a cube now, under `meta.ossie.model`, and import reads them back when no view
is mapped. The carrier is the alphabetically first cube -- deterministic, and independent
of both dataset ordering and the relationship graph, so every export picks the same one.
Import does not depend on the choice; it reads whichever cube carries the record, which
cannot accumulate because the record is consumed and stripped from the stash.

Only values import could not otherwise recover are parked, so a Cube model that never had
model-level metadata still round-trips byte-identical rather than acquiring a `meta.ossie`
key it never had. That matters beyond tidiness: every fixture in the feature matrix is
cube-only, and their structural round trips would all have started failing. A name equal to
the one import synthesizes is recoverable by definition, hence the shared
DEFAULT_MODEL_NAME rather than a second copy of the literal.

Foreign-vendor `custom_extensions` deliberately keep refusing export in this case, and the
README now says why rather than leaving it looking inconsistent: import restores those only
from the mapped view, so a cube carrier would not bring them home. That path fails loudly,
which was never the complaint here.

Checked the new checks can fail: removing the export half breaks 3 tests, removing the
import half breaks 2, and both halves are exercised over three cycles because the value has
to survive being read back out of a cube's stash rather than merely written once. The
carrier's output is put through the Cube compile gate too, since it is new YAML in the
emitted model and holds a literal brace -- Cube compiles every string as an f-string.

576 tests with both gates, 408 with neither, 97% coverage. Cross-converter matrix unchanged.
@mfournioux

Copy link
Copy Markdown

Hi @MikeNitsenko @jbonofre , this PR can be merged or additional checks are expected ?

@MikeNitsenko

Copy link
Copy Markdown
Author

@mfournioux, all checks and updates from my side were implemented - looking for review and approval.

@jbonofre @khush-bhatia this adds a bidirectional Cube <-> Ossie converter, following the same shape as #247 and #239. It’s the implementation for issue #248, which asked whether a Cube converter would be welcome and hasn’t had an answer yet.

Would one of you be able to review it and approve the CI workflow run?

MikeNitsenko and others added 4 commits August 6, 2026 13:01
Review: everything parseable from an Ossie metric expression should not
ride in custom_extensions. Three reductions, all symmetric:

- `filters` regenerate from the folded CASE: the fold import writes
  (Cube's own applyMeasureFilters shape) is deterministic, so export
  unfolds it back into structured filters -- verified by refolding, so a
  hand-written CASE that merely looks similar stays one expression. A
  filtered measure now travels with no stash at all; when the fold is
  not invertible (the operand is itself a CASE), both spellings ride.
- Cube-only measure keys (format, drill_members, public, ...) ride flat,
  the protocol dimensions already use, instead of forcing a copy of the
  whole measure that duplicated the sql and type the expression carries.
- The owning cube is recorded only when the expression does not say it;
  export already places a metric on the sole dataset it references.

A declared type the expression would not regenerate (a calculated
measure whose sql is a single aggregate, a count_distinct over the
primary key) is recorded as a flat `type` entry -- the latter was a
latent round-trip flip to bare `count`.

On the fixtures: completed_amount and cities now carry no extension;
total_amount carries {format}; TPC-DS drops from 7 stash entries to 4.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review: composite metrics should stay decomposed -- an Ossie document
should not render final values. The expression language lists Metric
references among its supported constructs, and a bare identifier in a
model-level metric expression resolves in the metric namespace. So
`{total_amount} / {count}` now imports as `total_amount /
orders__count` (the referenced metrics' Ossie names) rather than as an
inlined copy of every referenced definition -- which is the metric
drift a shared semantic model exists to prevent. Export renders a bare
metric name back as `{measure}` on the same cube, `{cube.measure}`
across cubes; the fixtureA calculated measure now carries no extension
at all.

Because bare identifiers are references at the model level, raw columns
in measure SQL are dataset-qualified on import (parser-based, string
surgery so spellings survive): `SUM(amount * 2)` reads as
`SUM(orders.amount * 2)`, which also removes the column/metric
ambiguity outright.

The resolver now keeps two forms per measure: the emitted one, and the
fully inlined one Cube itself renders -- which is what the fan-out
analysis reads, since a reference hides the aggregates it stands for.
Inlining remains where a reference cannot: generated decomposition
parts, windowed dependencies (both park, as before), keyword-named
metrics, and multi-aggregate expressions authored as a single measure
(whose spelling rides in the stash so export does not decompose them).
Reference cycles are refused in both directions, as Cube refuses them;
a metric referencing a dialect-dropped metric drops with it,
transitively.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review: generated Cube models should use references. Cube interpolates
a member's sql verbatim into generated queries, so a bare column in a
computed expression is ambiguous the moment the cube is joined against
a table sharing the name. Every bare column in generated member SQL --
computed dimension sql, measure operands, reconstructed filters -- is
now qualified as {CUBE}.column, the reference Cube's own documentation
recommends. Parser-based (sqlglot finds the column tokens, string
surgery applies them), so keywords, function names and EXTRACT units
are never touched, and an unparseable expression is left exactly as it
was. {CUBE}.column rather than a {member} reference on purpose: the two
coincide for a plain member, but the column form keeps meaning the
column even when a computed field shadows its name.

A single-column dimension keeps the bare `sql: column` form Cube
models conventionally use. A pleasant side effect: a hand-written
`CONCAT({CUBE}.tenant_id, {CUBE}.id)` key now round-trips
byte-identically instead of coming back bare. The TPC-DS fixture's
computed dimension is updated to the reference form it should have
been written in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by running Cube -> Ossie -> Databricks -> Ossie -> Cube end to
end: the Databricks importer emits a metric view's source columns
unqualified (SUM(ss_ext_sales_price) -- unqualified *means* the source
there), and reading that as opaque SQL placed the aggregate on whatever
cube the rest of the expression named. TPC-DS's customer_lifetime_value
put SUM(ss_ext_sales_price) on the customer cube -- a measure over a
column that cube does not have, which compiles (SQL is opaque to Cube's
compiler) and reads the wrong table at query time. Pre-existing, not a
regression: the pre-branch converter placed it identically, just
spelled bare.

A bare identifier in a model-level metric expression that is no metric
but is a declared field of exactly one dataset can only mean that
dataset's column. It now renders through the ordinary reference
machinery ({CUBE}.column, {CUBE.member}, or the cross-cube
{other.member} that carries the implicit join), decomposition places
its aggregate on the owning cube, and the metric's own cube derivation
sees it. A name declared on several datasets is never guessed at: it
stays raw SQL of the fallback cube, as before.

With this, the full interop chain ends in a model Cube compiles, with
one repair the Databricks format forces (a metric view cannot carry the
source table's own key; both converters report it).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MikeNitsenko and others added 4 commits August 6, 2026 21:40
Review follow-up: customer_lifetime_value still carried a rendered copy
of its own SQL in custom_extensions -- the one measure shape left where
the extension duplicated the expression. Root causes fixed:

- Cross-cube member references ({customer.c_customer_sk}) are
  reversible in model-level metric SQL: export re-emits them verbatim,
  so the spelling never needed recording. sql_is_reversible now knows
  each cube's members and accepts the canonical spelling.
- The owning cube mirrors export's full derivation, base cube included:
  a cross-dataset metric on the FK sink needs no cube record.
- Export decomposes a composite only when its aggregates read different
  cubes -- that is where per-aggregate fan-out correction lives. A
  single-cube composite (MAX(x) - MIN(x)) stays one calculated measure
  and round-trips verbatim, stash-free. An inline-authored cross-cube
  composite normalizes to the decomposed form on its first round trip,
  a documented normalization whose second cycle is a fixed point; the
  tpcds fixture now commits that fixed point.
- A geo half's sql rides in the stash only when the field's expression
  would not regenerate it -- {CUBE}.lat is what the expression already
  says, so the stash keeps only of/part.

Full-fixture audit: fixtureA carries 7 extension entries, tpcds 4
(views curation, segments, geo structure, name mapping, format,
public:false, and two foreign vendors) -- every one something the
expression genuinely cannot say.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-up: a sub_query dimension's sql references a measure
({orders.count}), which Cube resolves through a correlated subquery.
Emitting the flattened reference as an Ossie field expression
(orders.count) claimed a column no dataset has -- text that reads as
valid SQL and computes nothing anywhere -- softened only by an
APPROXIMATED issue. That was inconsistent with the converter's own
precedents: switch dimensions and multi-stage measures, whose Ossie
renderings would equally claim something they are not, are parked
whole. The sub_query dimension now rides the same protocol
(PARKED_IN_META, dataset stash, original position restored verbatim),
and the aggregate itself still reaches the model as the hoisted metric
the reference points at.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Re-verified every fixture and test against the review principles.
Fixtures: both Ossie snapshots regenerate byte-identically; exports
match the committed Cube fixtures structurally (remaining byte diffs
are YAML formatting only); hand-authored fixture carries only its
deliberate foreign-vendor extension.

Two behaviors the audit found tested only implicitly are now pinned:

- A canonical cross-cube member reference ({customer.c_sk}) travels
  with no stash and returns verbatim; a case-variant spelling
  ({CUSTOMER.c_sk}) would come back canonicalized, so that one keeps
  the original.
- A bare identifier that is both a metric's name and another dataset's
  field resolves in the metric namespace -- a measure reference, not a
  column read that would silently bypass the metric's definition.

Stale prose retired: _MEASURE_NATIVE_KEYS no longer claims extra keys
force a whole-measure copy; the README's stashed-on-import list moves
sub_query out of dimension extras and into parked-whole; the TPC-DS
stash count is 2, not 4; the property generator's single-cube composite
comment no longer says export splits it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verifying the three fixture identities showed the third one leaking:
import(export(hand_authored)) gained a model-level stash recording the
view export had just generated -- a rendered copy of regeneration's own
output, exactly what the extensions-minimization principle forbids, and
a foreign-vendor warning for every other spoke downstream.

The view builder (generated_view_cubes, uncollided_view_name) moves to
_common so import can predict the generated view with export's own
code: when the model's sole view is byte-equal to the prediction --
same base cube, same member lists, same prefix/exclude decisions, the
canonical path, no leftover meta -- the view set is not stashed and the
next export generates it again. Anything off that shape (a curated
includes list, an edited prefix, a second view, an off-layout path) is
stashed verbatim exactly as before, so a user's edits in Cube are never
dropped.

Falls out naturally: the TPC-DS view is exactly the generated shape, so
its model-level CUBE extension disappears altogether -- the model now
carries only the two foreign vendors -- and the mixed-file test's view,
also generated-shaped, keeps only its file-layout record.

hand_authored round trip is now asserted as whole-document identity,
with an edited-view negative pinning the exact-match guard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@MikeNitsenko

Copy link
Copy Markdown
Author

@jbonofre @QMalcolm @khush-bhatia could you please review and approve?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants