msi_to_osi silently discards config.meta instead of mapping it to custom_extensions

[georgea-tk]

MSIToOSIConverter silently discards config.meta from metrics, dimensions and semantic models. MSI carries the metadata, Ossie has a place to put it (custom_extensions), but the converter never reads one or writes the other — and unlike other lossy paths in this converter, no ConverterIssue is emitted.

Root cause

converters/dbt/src/ossie_dbt/msi_to_osi.py. Each OSI object is built from a fixed handful of attributes:

OSIMetric(
    name=metric.name,
    expression=self._make_expression(expr),
    description=metric.description,
)

return OSIField(
    name=dim.name,
    expression=self._make_expression(expr),
    dimension=OSIDimension(is_time=is_time),
    label=dim.label,
    description=dim.description,
)

grep -n '\.config\|meta' converters/dbt/src/ossie_dbt/msi_to_osi.py returns nothing — .config is never accessed anywhere in the converter.

Both sides of the mapping already exist:

• Source: PydanticMetric, PydanticSemanticModel and PydanticDimension all expose a config field of type PydanticSemanticLayerElementConfig, whose meta is an arbitrary dict. dbt writes user-supplied config.meta into semantic_manifest.json, so it is present in the object the converter is handed.
• Destination: custom_extensions is defined on Field, Metric, Dataset, Relationship and SemanticModel in core-spec/osi-schema.json, and Vendor is documented as "Any string value is accepted", with DBT given as an example.

Repro (HEAD 88e0011148283302c9a04cd0287e00e0b9d87354), using the repo's own test helpers:

import sys
sys.path.insert(0, "src"); sys.path.insert(0, "."); sys.path.insert(0, "../../python/src")

from ossie_dbt.msi_to_osi import MSIToOSIConverter
from tests.helpers import _entity, _manifest, _simple_metric
from metricflow_semantic_interfaces.implementations.element_config import PydanticSemanticLayerElementConfig
from metricflow_semantic_interfaces.implementations.elements.dimension import PydanticDimension
from metricflow_semantic_interfaces.implementations.elements.measure import PydanticMeasure
from metricflow_semantic_interfaces.test_utils import semantic_model_with_guaranteed_meta
from metricflow_semantic_interfaces.type_enums import AggregationType, DimensionType, EntityType

meta = {"value_format": '[>=1000000]$#,##0.00,,"M";$#,##0.00', "group_label": "Booking Costs"}

sm = semantic_model_with_guaranteed_meta(
    name="bookings",
    entities=[_entity("booking", EntityType.PRIMARY, "booking_id")],
    measures=[PydanticMeasure(name="amount", agg=AggregationType.SUM, expr="amount_eur")],
    dimensions=[PydanticDimension(name="status", type=DimensionType.CATEGORICAL,
                                  config=PydanticSemanticLayerElementConfig(meta=meta))],
)
metric = _simple_metric("total_amount", "amount")
metric.config = PydanticSemanticLayerElementConfig(meta=meta)

res = MSIToOSIConverter().convert(_manifest(semantic_models=[sm], metrics=[metric]))
osi_metric = res.output.semantic_model[0].metrics[0]
osi_field = [f for f in res.output.semantic_model[0].datasets[0].fields if f.name == "status"][0]

print("metric.custom_extensions:", osi_metric.custom_extensions)
print("field.custom_extensions :", osi_field.custom_extensions)
print("ISSUES:", res.issues)

Run from converters/dbt/.

Observed:

metric.custom_extensions: None
field.custom_extensions : None
ISSUES: []

Expected: the meta dict carried through as a custom_extensions entry — or, if that is deliberately out of scope, a ConverterIssue so the loss is visible.

Why this matters

config.meta is the only place dbt users can attach metadata that the MSI spec does not model — units, display formatting, grouping, ownership, links back to a source system. For anyone migrating an existing BI semantic layer into dbt it tends to be where all the presentation metadata ends up, precisely because the spec has nowhere else for it. Discarding it at the Ossie boundary means that content cannot reach any downstream consumer, even ones that would understand it.

It is also inconsistent with how this converter handles its other lossy paths: CUMULATIVE metrics emit CUMULATIVE_SEMANTICS_LOSS, conversion metrics emit CONVERSION_METRIC_DROPPED, natural entities emit NATURAL_ENTITY_DROPPED. This loss is silent.

Suggested fix

Map config.meta onto a single custom_extensions entry when non-empty, e.g.

def _meta_extensions(config) -> Optional[List[OSICustomExtension]]:
    meta = getattr(config, "meta", None) if config else None
    if not meta:
        return None
    return [OSICustomExtension(vendor_name="DBT", data=json.dumps(meta))]

applied in _convert_metrics, _convert_dimension, _convert_entity, _convert_measure and _convert_dataset. CustomExtension.data is a JSON string, so the dict serialises directly.

Happy to send a PR if the approach sounds right. Two things worth deciding first:

1. Vendor name. DBT matches the schema's own example, but COMMON may be more appropriate if the intent is that any consumer may read it rather than dbt-specific tooling.
2. Round-tripping. osi_to_msi would need the inverse to make this lossless in both directions. Happy to include that in the same PR.

Note: the same msi_to_osi implementation is vendored in dbt-labs/metricflow as metricflow/converters/msi_to_osi.py (byte-identical apart from one word in a docstring) and ships inside dbt-core 1.12, where it generates target/osi_document.json. The same gap exists there.

Related: #111 covers custom_extensions being dropped again on the Snowflake side, so both would need addressing for metadata to survive a full dbt → Ossie → Snowflake conversion.