SubstraitTypeSystem is the type system isthmus configures, and ConverterProvider.Builder.typeFactory(RelDataTypeFactory) lets a caller supply their own. Two legs of isthmus consult neither, so a limit the configured type system states is not the limit those legs apply.
Substrait → SQL still caps at Calcite's default. SubstraitSqlDialect.DEFAULT_CONTEXT is SqlDialect.EMPTY_CONTEXT, and Calcite's emptyContext() pins RelDataTypeSystemImpl.DEFAULT; SqlDialect.getCastSpec then clips a CHAR/VARCHAR precision to getTypeSystem().getMaxPrecision(...), i.e. 65536. So a varchar<100000> renders as CAST(… AS VARCHAR(65536)) — a truncating cast that declares a width the plan does not. getCastSpec does not switch on BINARY, so fixed_binary<100000> renders faithfully as BINARY(100000) and the three length-carrying types disagree with each other. DECIMAL has the same shape and predates any of this: decimal<38,10> renders as DECIMAL(19,10), since getCastSpec reads getMaxPrecision/getMaxScale from the default system there too.
The fix is a context carrying the type system isthmus already has:
public static final SqlDialect.Context DEFAULT_CONTEXT =
SqlDialect.EMPTY_CONTEXT.withDataTypeSystem(SubstraitTypeSystem.TYPE_SYSTEM);The SQL → Substrait cluster ignores the provider's factory. SubstraitSqlToCalcite.createDefaultRelOptCluster() is static, takes no argument, and hardcodes new RexBuilder(new JavaTypeFactoryImpl(SubstraitTypeSystem.TYPE_SYSTEM)). The provider-aware overloads call it without the provider, while the validator does get the provider's factory, through the catalog reader. The two therefore derive types under different type systems for the same query. SqlConverterBase does build a cluster from the provider's factory, but that field is consumed only by the extended-expression path, so SqlToSubstrait.convert gets the hardcoded one.
While the two type systems agree on a bound this is invisible. Once they do not, SqlToRelConverter.checkConvertedType compares the validated and converted row types and throws:
java.lang.AssertionError: Conversion to relational algebra failed to preserve datatypes:
validated type: RecordType(VARCHAR EXPR$0) NOT NULL
converted type: RecordType(VARCHAR(80000) EXPR$0) NOT NULL
for SELECT a || b FROM t over t(a VARCHAR(40000), b VARCHAR(40000)) — no wide type anywhere — with a provider built on new SqlTypeFactoryImpl(RelDataTypeSystem.DEFAULT). That is an explicit throw new AssertionError, not a bare assert, so it is not -ea-dependent. This half is live only once getMaxPrecision diverges between the two, which #1169 is what does; it is raised on that PR and may be fixed there, in which case only the dialect half of this issue remains.
Same shape as #1219, which is the type mapper rather than the type system: a decision taken on one leg with no hook on the other. Not folded in — the mapper has its own interface question.
ConverterProviderBuilderTest never calls .typeFactory(...), so nothing currently covers a provider whose type system is not Substrait's; that is the gap that leaves both halves invisible.
Checked on #1169's head 880090ae and its base.
SubstraitTypeSystemis the type system isthmus configures, andConverterProvider.Builder.typeFactory(RelDataTypeFactory)lets a caller supply their own. Two legs of isthmus consult neither, so a limit the configured type system states is not the limit those legs apply.Substrait → SQL still caps at Calcite's default.
SubstraitSqlDialect.DEFAULT_CONTEXTisSqlDialect.EMPTY_CONTEXT, and Calcite'semptyContext()pinsRelDataTypeSystemImpl.DEFAULT;SqlDialect.getCastSpecthen clips aCHAR/VARCHARprecision togetTypeSystem().getMaxPrecision(...), i.e. 65536. So avarchar<100000>renders asCAST(… AS VARCHAR(65536))— a truncating cast that declares a width the plan does not.getCastSpecdoes not switch onBINARY, sofixed_binary<100000>renders faithfully asBINARY(100000)and the three length-carrying types disagree with each other.DECIMALhas the same shape and predates any of this:decimal<38,10>renders asDECIMAL(19,10), sincegetCastSpecreadsgetMaxPrecision/getMaxScalefrom the default system there too.The fix is a context carrying the type system isthmus already has:
The SQL → Substrait cluster ignores the provider's factory.
SubstraitSqlToCalcite.createDefaultRelOptCluster()is static, takes no argument, and hardcodesnew RexBuilder(new JavaTypeFactoryImpl(SubstraitTypeSystem.TYPE_SYSTEM)). The provider-aware overloads call it without the provider, while the validator does get the provider's factory, through the catalog reader. The two therefore derive types under different type systems for the same query.SqlConverterBasedoes build a cluster from the provider's factory, but that field is consumed only by the extended-expression path, soSqlToSubstrait.convertgets the hardcoded one.While the two type systems agree on a bound this is invisible. Once they do not,
SqlToRelConverter.checkConvertedTypecompares the validated and converted row types and throws:for
SELECT a || b FROM tovert(a VARCHAR(40000), b VARCHAR(40000))— no wide type anywhere — with a provider built onnew SqlTypeFactoryImpl(RelDataTypeSystem.DEFAULT). That is an explicitthrow new AssertionError, not a bareassert, so it is not-ea-dependent. This half is live only oncegetMaxPrecisiondiverges between the two, which #1169 is what does; it is raised on that PR and may be fixed there, in which case only the dialect half of this issue remains.Same shape as #1219, which is the type mapper rather than the type system: a decision taken on one leg with no hook on the other. Not folded in — the mapper has its own interface question.
ConverterProviderBuilderTestnever calls.typeFactory(...), so nothing currently covers a provider whose type system is not Substrait's; that is the gap that leaves both halves invisible.Checked on #1169's head
880090aeand its base.