Explore options for accelerating InMemoryTableScanExec

[andygrove]

What is the problem the feature request solves?

In queries with InMemoryTableScanExec, we have to perform a CometColumnarToRowExec before this operator and then we perform a CometSparkRowToColumnar after the operator.

Here is an example plan from the test SparkToColumnar over InMemoryTableScanExec.

*(1) CometColumnarToRow
+- !CometHashAggregate [key#304L, count#371L], Final, [key#304L], [count(1)]
   +- !CometHashAggregate [key#304L], Partial, [key#304L], [partial_count(1)]
      +- CometSparkRowToColumnar
         +- Scan In-memory table abc [key#304L]
               +- InMemoryRelation [key#304L, value#305L, (key + 1)#308L], StorageLevel(disk, memory, deserialized, 1 replicas)
                     +- *(2) CometColumnarToRow
                        +- CometProject [key#6L, value#7L, (key + 1)#10L], [id#0L AS key#6L, (id#0L % 8) AS value#7L, (id#0L + 1) AS (key + 1)#10L]
                           +- CometSparkRowToColumnar
                              +- *(1) Range (0, 1000, step=1, splits=5)

Describe the potential solution

No response

Additional context

No response

[andygrove]

Spark's InMemoryTableScanExec can support columnar data (in Spark format, not in Arrow format). I wonder if we can leverage this rather than replacing the operator.

[andygrove]

It seems possible to either use InMemoryTableScanExec directly, or copy the code into a new CometInMemoryTableScanExec if needed, and then provide a custom CachedBatchSerializer serializer in CachedRDDBuilder within the InMemoryRelation.

case class CachedRDDBuilder(
    serializer: CachedBatchSerializer,
    storageLevel: StorageLevel,
    @transient cachedPlan: SparkPlan,
    tableName: Option[String]) {

[pchintar]

Hi @andygrove I've just opened a PR titled Add Native Support for In-Memory Cache for this issue.

My implementation follows the CachedBatchSerializer approach discussed above. It adds a custom cache serializer together with CometInMemoryTableScanExec, allowing cached data to be written and read in a Comet-compatible format without any extra conversion steps. I've also added diverse tests to check my new configuration flag behavior as thoroughly as possible.

So, could you kindly pls check it out?