[Umbrella] Kafka Protocols Compatibility - Basic Produce

[gyang94]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Apache Fluss needs a first, reviewable Kafka protocol compatibility milestone that lets standard Kafka producers write data through Fluss.

This umbrella issue tracks the basic Produce path as a dependency-ordered series. The work starts with the protocol framework and metadata/admin foundations, then adds Produce semantics, authentication, and schema-aware record conversion. Keeping these stages separate gives reviewers an independently motivated diff at each step while preserving a working progression.

Solution

Implement the milestone as four sequential pull requests:

• [kafka] Establish compatibility framework #4186 — establish the Kafka compatibility framework, API registry and dispatch, topic mapping, and metadata/admin foundations.
• [kafka] Add basic Produce compatibility #4187 — add basic Produce compatibility, acknowledgement handling, and the server-side delayed follower-fetch wake-up required by acks=all.
• [kafka] Add SASL/PLAIN authentication #4188 — add SASL/PLAIN authentication for Kafka client connections.
• [kafka] Add schema-aware Produce transcoding #4189 — add schema-aware Produce transcoding and Kafka-to-Fluss record mapping.

The PRs are published as a dependency-ordered draft stack. PR2 through PR4 include their predecessors until those PRs are merged and the remaining branches are rebased onto the updated main.

Acceptance criteria

• The four PRs can be reviewed and merged in dependency order.
• A standard Kafka producer can write through the basic compatibility path.
• Authentication and schema-aware conversion are covered by isolated follow-up PRs.
• Existing Fluss RPC and server behavior remains compatible.

Anything else?

Out of scope for this milestone:

• Idempotent and transactional Produce
• Authoritative ISR metadata follow-ups
• Bounded Produce admission and flow-control work
• Kafka consumer and Fetch compatibility

Willingness to contribute

• I'm willing to submit a PR!