[protocol][compat] Stage StoreMetaValue v48 (version-level maxNearlineRecordSizeBytes) - #2991
Merged
KaiSernLim merged 1 commit intoAug 26, 2026
Conversation
KaiSernLim
force-pushed
the
kailim/nearline-record-size-protocol-v48
branch
from
August 21, 2026 19:51
3dcad42 to
06709b5
Compare
KaiSernLim
changed the title
KaiSernLim
added a commit
to KaiSernLim/venice
that referenced
this pull request
Aug 21, 2026
Read and snapshot the nearline oversized-record limit per Version instead of live from the Store. The limit is captured onto the Version at version-creation time in AbstractStore.addVersion (mirroring chunkingEnabled/storageMode); nearline ingestion is continuous, so reading it live from the store would retroactively change enforcement for already in-flight versions. The server reads the per-version snapshot in LeaderFollowerStoreIngestionTask.getMaxNearlineRecordSizeBytes() via a new StoreIngestionTask.getVersion() accessor. A null snapshot (a version created before the field existed) falls back to the live store-level value; a non-null value (including -1) is an immutable per-version snapshot that later store-level changes do not affect. Stacked on linkedin#2991, which introduces StoreMetaValue v48 and the version-level maxNearlineRecordSizeBytes field this logic reads/writes. This PR is schema-free; it compiles once linkedin#2991 is merged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
maxNearlineRecordSizeBytes) Introduce StoreMetaValue schema v48, which adds a version-level maxNearlineRecordSizeBytes field on the nested StoreVersion record (nullable union ["null","int"], default null) and refines the doc on the existing store-level StoreProperties.maxNearlineRecordSizeBytes to describe it as the snapshot source (and to note that only null-snapshot legacy versions fall back to reading it live). Following the staged-protocol convention, generation is pinned to v47 via the compileAvro versionOverrides so the generated classes stay on v47 and no code path can serialize v48 yet. The override is removed and METADATA_SYSTEM_SCHEMA_STORE is bumped to v48 in the follow-up PR (linkedin#2986) that reads and snapshots the per-version limit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
KaiSernLim
force-pushed
the
kailim/nearline-record-size-protocol-v48
branch
from
August 25, 2026 08:58
06709b5 to
0a9bc12
Compare
KaiSernLim
changed the title
KaiSernLim
added a commit
to KaiSernLim/venice
that referenced
this pull request
Aug 25, 2026
Read and snapshot the nearline oversized-record limit per Version instead of live from the Store. The limit is captured onto the Version at version-creation time in AbstractStore.addVersion (mirroring chunkingEnabled/storageMode); nearline ingestion is continuous, so reading it live from the store would retroactively change enforcement for already in-flight versions. The server reads the per-version snapshot in LeaderFollowerStoreIngestionTask.getMaxNearlineRecordSizeBytes() via a new StoreIngestionTask.getVersion() accessor. A null snapshot (a version created before the v48 field existed) falls back to the live store-level value; a non-null value (including -1) is an immutable per-version snapshot that later store-level changes do not affect. Activates StoreMetaValue v48 (staged in linkedin#2991) by removing the compileAvro versionOverrides pin and bumping METADATA_SYSTEM_SCHEMA_STORE 47->48, so the generated StoreVersion carries the new maxNearlineRecordSizeBytes field this logic reads/writes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
sixpluszero
approved these changes
Aug 26, 2026
KaiSernLim
changed the title
KaiSernLim
added a commit
to KaiSernLim/venice
that referenced
this pull request
Aug 31, 2026
Read and snapshot the nearline oversized-record limit per Version instead of live from the Store. The limit is captured onto the Version at version-creation time in AbstractStore.addVersion (mirroring chunkingEnabled/storageMode); nearline ingestion is continuous, so reading it live from the store would retroactively change enforcement for already in-flight versions. The server reads the per-version snapshot in LeaderFollowerStoreIngestionTask.getMaxNearlineRecordSizeBytes() via a new StoreIngestionTask.getVersion() accessor. A null snapshot (a version created before the v48 field existed) falls back to the live store-level value; a non-null value (including -1) is an immutable per-version snapshot that later store-level changes do not affect. Activates StoreMetaValue v48 (staged in linkedin#2991) by removing the compileAvro versionOverrides pin and bumping METADATA_SYSTEM_SCHEMA_STORE 47->48, so the generated StoreVersion carries the new maxNearlineRecordSizeBytes field this logic reads/writes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stages the
StoreMetaValuev48 protocol, split out of #2986 (the per-version logic PR) so it lands as an isolated, low-risk schema change.maxNearlineRecordSizeByteson the nestedStoreVersionrecord, as a nullable union["null","int"](defaultnull):null= the version predates v48 (no snapshot) → runtime falls back to the live store-level value.-1) = an immutable per-version snapshot that later store-level changes do not affect.StoreProperties.maxNearlineRecordSizeBytesto describe it as the snapshot source (type/default unchanged), and notes that only null-snapshot legacy versions fall back to reading it live.Staged, not yet activated
Following the repo's staged-protocol convention (see #2971), code generation is pinned to v47 via the
compileAvroversionOverridesinbuild.gradle, so the generated classes stay on v47 and no code path can serialize v48 yet.AvroProtocolDefinition.METADATA_SYSTEM_SCHEMA_STOREstays at 47 here.The follow-up #2986 removes this pin, bumps
METADATA_SYSTEM_SCHEMA_STORE47→48, and adds the read/snapshot logic that consumes the new field.Validation
:internal:venice-common:compileAvrocopiesStoreMetaValue/v47with(OVERRIDE); the generatedStoreVersionhas nomaxNearlineRecordSizeBytesfield, so the build and all protocol/schema tests stay on v47.SchemaCompatibilitycheck): v47 data deserializes the new field asnull.Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com