Part of #5572.
HashUtils.unsupportedReasonFor declines a DecimalType with precision > 18, walking into structs, arrays and map key/value types to find one:
case d: DecimalType if d.precision > 18 => Some(unsupportedDecimalReason)
spark/src/main/scala/org/apache/comet/serde/hash.scala:137
Four serdes route through it and none mixes in CodegenDispatchFallback: CometMurmur3Hash (hash.scala:49), CometXxHash64 (:27), CometSha1 (:101) and CometSha2 (:75). So hash(high_precision_decimal_col) or xxhash64(...) fails the whole projection back to Spark, which matters because these turn up in bucketing, partitioning and dedup paths where the surrounding operator is worth keeping native.
The reason given — Spark hashes via Java BigDecimal — is again a case where the dispatcher is the right answer rather than a native fix, since running Spark's HashExpression.doGenCode reproduces the BigDecimal hashing exactly. DecimalType is in CometBatchKernelCodegen.isSupportedDataType at any precision.
CometSha2 has a second decline worth covering in the same change:
if (!expr.right.foldable) {
Unsupported(Some(nonFoldableNumBitsReason))
}hash.scala:84-85 — a numBits argument that is a column rather than a literal. Sha2.doGenCode handles that fine.
Not in scope: the TimeType arm of the same helper. isTimeType values are admitted by the kernel via case dt if isTimeType(dt) => true, so it may be dispatchable too, but Comet's TimeType support is new enough that it deserves its own look rather than being folded in here.
Part of #5572.
HashUtils.unsupportedReasonFordeclines aDecimalTypewith precision > 18, walking into structs, arrays and map key/value types to find one:spark/src/main/scala/org/apache/comet/serde/hash.scala:137Four serdes route through it and none mixes in
CodegenDispatchFallback:CometMurmur3Hash(hash.scala:49),CometXxHash64(:27),CometSha1(:101) andCometSha2(:75). Sohash(high_precision_decimal_col)orxxhash64(...)fails the whole projection back to Spark, which matters because these turn up in bucketing, partitioning and dedup paths where the surrounding operator is worth keeping native.The reason given — Spark hashes via Java
BigDecimal— is again a case where the dispatcher is the right answer rather than a native fix, since running Spark'sHashExpression.doGenCodereproduces theBigDecimalhashing exactly.DecimalTypeis inCometBatchKernelCodegen.isSupportedDataTypeat any precision.CometSha2has a second decline worth covering in the same change:hash.scala:84-85— anumBitsargument that is a column rather than a literal.Sha2.doGenCodehandles that fine.Not in scope: the
TimeTypearm of the same helper.isTimeTypevalues are admitted by the kernel viacase dt if isTimeType(dt) => true, so it may be dispatchable too, but Comet'sTimeTypesupport is new enough that it deserves its own look rather than being folded in here.