Repository navigation
DESCRIBE TABLE truncates complex STRUCT/ARRAY type strings in Databricks #5868
Description
Summary
DatabricksEngineAdapter.columns() (inherited from BaseEngineAdapter) uses DESCRIBE TABLE to fetch column types. Databricks silently truncates the type string for deeply-nested STRUCT/ARRAY columns, replacing trailing fields with ... N more fields>>. This causes create_external_models to write incomplete type definitions into external_models.yaml, silently dropping fields with no warning.
Root cause
This is intentional Databricks behaviour. From the Databricks docs for DESCRIBE TABLE:
The non-JSON output of DESCRIBE TABLE can change, and new columns might be appended over time. Avoid relying on the output meeting a specific size or schema. For programmatic consumption, use DESCRIBE TABLE ... AS JSON.
Reproducible example
import itertools
from sqlmesh import Context
ctx = Context(paths=["<path-to-your-sqlmesh-project>"])
adapter = ctx.engine_adapter
adapter.execute("""
CREATE TEMPORARY TABLE sqlmesh_describe_repro (
id INT,
nested ARRAY<STRUCT<
field_one: STRING, field_two: STRING, field_three: STRING,
field_four: STRING, field_five: STRING, field_six: STRING,
field_seven: STRING, field_eight: STRING, field_nine: STRING,
field_ten: STRING,
inner_items: ARRAY<STRUCT<
item_id: STRING,
item_location: STRUCT<location_id: STRING, location_name: STRING>,
item_warehouse: STRUCT<warehouse_id: STRING, warehouse_name: STRING>,
item_quantity: DOUBLE,
item_uom: STRING,
item_expiration_date: TIMESTAMP,
item_description: STRING
>>,
field_eleven: STRING, field_twelve: STRING, field_thirteen: STRING,
field_fourteen: STRING, field_fifteen: STRING, field_sixteen: STRING,
field_seventeen: STRING, field_eighteen: STRING,
field_nineteen: STRING, field_twenty: STRING,
field_twenty_one: STRING, field_twenty_two: STRING,
field_twenty_three: STRING, field_twenty_four: STRING,
field_twenty_five: DOUBLE, field_twenty_six: DOUBLE,
field_twenty_seven: INT, field_twenty_eight: INT,
field_twenty_nine: TIMESTAMP
>>
)
""")
cols = adapter.columns("sqlmesh_describe_repro")
print(cols["nested"].sql(dialect="databricks"))
# ARRAY<STRUCT<..., field_twenty_four: STRING, ... 5 more fields>>
Activity
EhabEasee commentedon Jun 30, 2026
This has already been fixed in #5783. I was running an older version