Avro deserialization

[jaj42]

Hi!
I'm looking for an efficient way to stream data from Apache Kafka to Ducklake and millpond seems perfect.
The only limit I see is the reliance on JSON as single serialization format. I think Avro makes sense for high throughput data.

Your documentation says: "If you need Avro/Protobuf deserialization with schema registry, you handle it yourself.".
I was wondering if that means that you do not wish to add Avro support or whether you would be open to contributions.

Kind regards

[jghoman]

Hey @jaj42! Apologies for the delay; didn't see this. Avro has such a different memory footprint/behavior footprint that I'm not sure it would be a good mix here directly (I added Avro support to Hive, so I'm spent quite a bit of time in its guts). I'd probably fork the project to add support, since it's small code base, see how it works out and then see if later they can be reconciled.

[jghoman]

Also, take a look the direct Arrow exploration that's going on.