diff --git a/docs/en/antalya/cas/architecture/blob-protocol.md b/docs/en/antalya/cas/architecture/blob-protocol.md index 3454baa393ff..bb4d654b3550 100644 --- a/docs/en/antalya/cas/architecture/blob-protocol.md +++ b/docs/en/antalya/cas/architecture/blob-protocol.md @@ -216,7 +216,7 @@ bytes, not inferred from occupancy alone as blob uploads do. Divergent bytes are correct operation and fail closed as `CORRUPTED_DATA`. This is the format-evolution **adoption pin**, documented in the persisted-format registry -(`Formats/README.md`): on a `putDeterministicArtifact` conflict, the writer re-encodes at the `v` +(`Formats/README.md`): on a `putDeterministicArtifact` conflict, the writer re-encodes at the `version` of the *existing* object rather than at its own current build's version, so two writers on different builds replaying the same deterministic round still land on byte-identical output. diff --git a/docs/en/antalya/cas/architecture/garbage-collection.md b/docs/en/antalya/cas/architecture/garbage-collection.md index c918a0a35f14..da64e04f562b 100644 --- a/docs/en/antalya/cas/architecture/garbage-collection.md +++ b/docs/en/antalya/cas/architecture/garbage-collection.md @@ -22,7 +22,7 @@ race over one blob is covered on the ## Leadership {#leadership} There is **no separate `GC` lease object**. The lease lives inside `gc/state` itself as -`{owner, seq}`. +`{lease_owner, lease_sequence}`. ```mermaid stateDiagram-v2 @@ -30,22 +30,22 @@ stateDiagram-v2 Reading --> Creating: object absent, never observed before Creating --> Leader: casPut create-if-absent, cas_gc_shards fixed here, once Reading --> Renewing: lease owner is me - Renewing --> Leader: casPut seq+1, guarded by the observed token + Renewing --> Leader: casPut lease_sequence+1, guarded by the observed token Reading --> Evaluating: foreign owner Evaluating --> NotLeader: incumbent lease moved, or heartbeat moved, or steal not allowed Evaluating --> Stealing: both frozen across a full observation window - Stealing --> Leader: casPut owner=me seq+1, on the observed token + Stealing --> Leader: casPut lease_owner=me lease_sequence+1, on the observed token Stealing --> NotLeader: lost the CAS, re-read and re-arm Leader --> [*]: run the round ``` -Two independent liveness signals are consulted before a steal: whether `(owner, seq)` moved since +Two independent liveness signals are consulted before a steal: whether `(lease_owner, lease_sequence)` moved since the last tick, and whether the separate `gc/hb` heartbeat moved. The heartbeat is compared only under the same remembered heartbeat owner, deliberately not against `lease.owner` — a deposed leader's heartbeat thread keeps pulsing, and that must not cause a live new leader's lease to be stolen. The paced background loop may steal; a manual `SYSTEM CAS GC RUN` may not, because the safety argument needs two observations separated by real wall time. Because every renew or steal -bumps `seq`, `seq` doubles as the round's attempt id. +bumps `lease_sequence`, `lease_sequence` doubles as the round's attempt id. **A deposed leader that keeps running cannot corrupt anything**, and the argument does not rely on exclusivity at all: diff --git a/docs/en/antalya/cas/architecture/mounts-and-leases.md b/docs/en/antalya/cas/architecture/mounts-and-leases.md index d778756ce323..70f9ff771c13 100644 --- a/docs/en/antalya/cas/architecture/mounts-and-leases.md +++ b/docs/en/antalya/cas/architecture/mounts-and-leases.md @@ -67,7 +67,7 @@ Two failure modes this closes: One object, `gc/server-roots//mount`, carries **both** the liveness lease and the build watermark — there is no separate watermark object. `MountLease` fields: `server_uuid`, -`writer_epoch`, `write_attempt_id`, `hostname`, `pid`, `started_at_ms`, renewal `seq`, +`writer_epoch`, `write_attempt_id`, `hostname`, `process_id`, `started_at_ms`, renewal `sequence`, `expires_at_ms`, `min_active` (the build-watermark floor), and `gc_fenced`. - **Logical renewal identity.** Each holder-originated body has a fresh nonzero @@ -165,8 +165,8 @@ the claim outcomes above and is shown here as behavior, not as a type in the cod ```mermaid stateDiagram-v2 [*] --> Absent - Absent --> Live: claimMount putIfAbsent, seq=1 - Live --> Live: keeper beat, putOverwrite seq+1 + Absent --> Live: claimMount putIfAbsent, sequence=1 + Live --> Live: keeper beat, putOverwrite sequence+1 Live --> Fenced: GC observes a stable token past threshold, gc_fenced=1, body preserved Live --> Terminated: certified drain, terminal farewell (expires_at=now, min_active=MAX) Fenced --> Live: same-uuid claim with a fresh writer_epoch, instant reclaim diff --git a/docs/en/antalya/cas/architecture/namespaces.md b/docs/en/antalya/cas/architecture/namespaces.md index c38b65b0ce17..4e7079f53d9d 100644 --- a/docs/en/antalya/cas/architecture/namespaces.md +++ b/docs/en/antalya/cas/architecture/namespaces.md @@ -62,7 +62,7 @@ Each row (`CatalogEntry`) carries: | Field | Meaning | |---|---| -| `ns` | The namespace name | +| `namespace` | The namespace name | | `state` | `Creating`, `Live`, or `Removing` — see below | | `incarnation` | The `life_id` for this row, nonzero, never reused | | `creator` | The mounted writer's fence identity (server root, writer epoch, admission fence generation) that is creating this row — **required** iff `state == Creating`, **forbidden** otherwise | diff --git a/docs/en/antalya/cas/architecture/storage-layout.md b/docs/en/antalya/cas/architecture/storage-layout.md index e4d20725836b..45b88f886b99 100644 --- a/docs/en/antalya/cas/architecture/storage-layout.md +++ b/docs/en/antalya/cas/architecture/storage-layout.md @@ -63,13 +63,13 @@ touches — it is reclaimed only by its own server's next mount. Every persisted CAS metadata object is text: a header line, a body, and an optional trailer. ``` -{"type":"cas_","v":N} <- header line, always present +{"type":"cas_","version":N} <- header line, always present <- one JSON object, sorted NDJSON records, or a descriptor + raw payload zone -{"n":…} <- optional trailer (record/entry count) +{"record_count":…} <- optional trailer (record/entry count) ``` -`v` is the only version field; a reader rejects `v` above what the build supports with +`version` is the only version field; a reader rejects `version` above what the build supports with `UNKNOWN_FORMAT_VERSION`, checked before the body. A `.zst` key suffix means, exactly, that the object kind's compression policy is `Always`: the object is stored as one zstd frame with the checksum flag on, and its declared content size is checked against a per-kind cap before @@ -77,10 +77,10 @@ allocation. Always-small and deterministic kinds (`cas_ref_ckpt`, `cas_blob_meta `cas_run`, …) are stored raw, with no `.zst` suffix. The blob envelope is a special case of the header/body shape: a JSON descriptor padded with ASCII -spaces to a pool-constant `blob_header_len` (256 bytes, a `cas_pool_meta` field), terminated by +spaces to the pool-constant `blob_header_length` (384 bytes by default, a `cas_pool_meta` field), terminated by `\n`, so the raw payload always starts at that fixed offset with no header parse needed to locate it. The part manifest is the other `PayloadHybrid` kind: text header, descriptor, sorted NDJSON -entry records, `{"n":…}` trailer, then a banner-framed raw payload zone for small inline file +entry records, `{"record_count":…}` trailer, then a banner-framed raw payload zone for small inline file bytes. ## Codec table {#codec-table} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp index 65176572e896..76bcf29c7c7a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp @@ -42,7 +42,7 @@ ProvenanceOp opFromWord(std::string_view w) if (w == "mutation") return ProvenanceOp::Mutation; if (w == "attach") return ProvenanceOp::Attach; if (w == "repack") return ProvenanceOp::Repack; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: unknown op '{}'", w); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: unknown operation '{}'", w); } /// The escaped byte-length of one raw ref char under the frozen envelope alphabet (see writeEnvelopeRefField). @@ -109,30 +109,30 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) CasJsonWriter buf(256); bool first = true; writeKey(buf, "type", first); writeStringValue(buf, kBlobType); - writeKey(buf, "v", first); writeIntText(currentCompatibilityVersion(), buf); - writeKey(buf, "tag", first); writeHex128Value(buf, header.incarnation_tag); - writeKey(buf, "bld", first); writeHex128Value(buf, header.build_id); + writeKey(buf, "version", first); writeIntText(currentCompatibilityVersion(), buf); + writeKey(buf, "incarnation_tag", first); writeHex128Value(buf, header.incarnation_tag); + writeKey(buf, "build_id", first); writeHex128Value(buf, header.build_id); if (header.provenance) { - writeKey(buf, "ts", first); writeIntText(header.provenance->created_at_ms, buf); - writeKey(buf, "by", first); writeHex128Value(buf, header.provenance->creator_server_id); - writeKey(buf, "op", first); writeStringValue(buf, opToWord(header.provenance->op)); - writeKey(buf, "ch", first); writeIntText(header.provenance->ch_version, buf); + writeKey(buf, "created_at_ms", first); writeIntText(header.provenance->created_at_ms, buf); + writeKey(buf, "creator_server_id", first); writeHex128Value(buf, header.provenance->creator_server_id); + writeKey(buf, "operation", first); writeStringValue(buf, opToWord(header.provenance->op)); + writeKey(buf, "clickhouse_version", first); writeIntText(header.provenance->ch_version, buf); } /// Test-only critical extension: an unknown `!`-key BEFORE `ref`. if (header.emit_unknown_critical_key) { writeKey(buf, "!x", first); writeStringValue(buf, "1"); } - json = std::move(buf).take(); /// e.g. {"type":"cas_blob","v":3,...,"ch":26006001 (no ref, no closing brace) + json = std::move(buf).take(); /// e.g. {"type":"cas_blob","version":3,...,"clickhouse_version":26006001 } /// Optional `ref`, truncated to the exact remaining budget. Layout after this block: - /// json + `,"ref":` + `"` + + `"` + `}` must be <= blob_header_len-1 + /// json + `,"intended_ref":` + `"` + + `"` + `}` must fit /// (byte blob_header_len-1 is reserved for '\n'; the pad zone fills the gap with spaces). if (header.intended_ref) { - static constexpr std::string_view ref_key = ",\"ref\":"; + static constexpr std::string_view ref_key = ",\"intended_ref\":"; /// +3 = opening quote + closing quote + closing brace. const size_t fixed = json.size() + ref_key.size() + 3; if (blob_header_len < 1 || fixed > static_cast(blob_header_len) - 1) @@ -181,37 +181,37 @@ EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*obje "CAS blob envelope: object is a '{}', not a '{}'", t, kBlobType); saw_type = true; } - else if (key == "v") + else if (key == "version") { h.compatibility_version = r.readU32Number(); checkCompatibility(h.compatibility_version, "blob envelope"); saw_v = true; } - else if (key == "tag") + else if (key == "incarnation_tag") h.incarnation_tag = r.readHex128(); - else if (key == "bld") + else if (key == "build_id") h.build_id = r.readHex128(); - else if (key == "ts") + else if (key == "created_at_ms") { prov.created_at_ms = r.readU64Number(); have_prov = true; } - else if (key == "by") + else if (key == "creator_server_id") { prov.creator_server_id = r.readHex128(); have_prov = true; } - else if (key == "op") + else if (key == "operation") { prov.op = opFromWord(r.readString()); have_prov = true; } - else if (key == "ch") + else if (key == "clickhouse_version") { prov.ch_version = static_cast(r.readU64Number()); have_prov = true; } - else if (key == "ref") + else if (key == "intended_ref") h.intended_ref = r.readString(); else r.skipUnknown(key); /// `!`-key -> UNKNOWN_FORMAT_VERSION; unknown plain key -> skipped (tolerant) @@ -219,7 +219,7 @@ EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*obje if (!saw_type) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: missing type"); if (!saw_v) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: missing v"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: missing version"); if (h.kind != expected_kind) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: kind {} does not match expected {}", diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h index 19250fe69ddd..d36c6d8369a9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h @@ -46,7 +46,7 @@ struct Provenance /// The blob envelope is a fixed-size JSON descriptor followed by the raw payload. The JSON object /// occupies bytes [0, json_len), ASCII spaces occupy [json_len, blob_header_len-1), and '\n' is at byte /// blob_header_len-1. The payload therefore begins at the pool-wide constant offset -/// `blob_header_len` (256 for blob pools, a `PoolMeta` parameter), allowing the locate path to use a +/// `blob_header_len` (384 by default for blob pools, a `PoolMeta` parameter), allowing the locate path to use a /// constant shift without reading an object-specific header first. The header is also the incarnation /// zone: it may differ between incarnations of one logical object, and each upload attempt gets a fresh /// random u128 `tag`, which is used as the exact-token delete identity. @@ -73,7 +73,7 @@ struct EnvelopeHeader bool emit_unknown_critical_key = false; }; -/// Builds the fixed-length header for a pool whose `blob_header_len` is `blob_header_len` (256 for blob +/// Builds the fixed-length header for a pool whose `blob_header_len` is `blob_header_len` (384 by default for blob /// pools). Sets `header.header_len = blob_header_len` and returns exactly that many bytes. The /// diagnostic `ref` is the only truncatable field and is shortened, never dropped, when necessary to /// preserve the fixed layout. The header is built without payload bytes, so an upload can stage the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp index b62fd3b82424..1b005adabdd8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp @@ -46,11 +46,11 @@ String encodeBlobMeta(const BlobMeta & meta) // `version` is represented by the header line. The JSON body contains only fields that describe // the current marker and its accounting data. bool first = true; - writeKey(out, "st", first); + writeKey(out, "state", first); writeStringValue(out, metaStateToWord(meta.state)); - writeKey(out, "cr", first); + writeKey(out, "condemn_round", first); writeU64StringValue(out, meta.condemn_round); - writeKey(out, "sz", first); + writeKey(out, "size", first); writeU64StringValue(out, meta.size); closeObject(out, first); writeChar('\n', out); @@ -72,20 +72,20 @@ BlobMeta decodeBlobMeta(std::string_view bytes) String key; while (r.nextKey(key)) { - if (key == "st") + if (key == "state") { m.state = metaStateFromWord(r.readString()); saw_state = true; } - else if (key == "cr") + else if (key == "condemn_round") m.condemn_round = r.readU64String(); - else if (key == "sz") + else if (key == "size") m.size = r.readU64String(); else r.skipUnknown(key); } if (!saw_state) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: missing st"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: missing state"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: trailing bytes"); return m; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp index b4bba2bff08f..b5fbe035631d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp @@ -88,11 +88,11 @@ void insertRecordOnce(Map & map, const Key & key, Value && value, std::string_vi void writeRun(CasJsonWriter & out, std::string_view kind, const RunRef & r) { bool first = true; - writeKey(out, "k", first); writeStringValue(out, kind); + writeKey(out, "kind", first); writeStringValue(out, kind); writeKey(out, "key", first); writeStringValue(out, r.key); - writeKey(out, "ck", first); writeHex128Value(out, r.checksum); + writeKey(out, "checksum", first); writeHex128Value(out, r.checksum); writeKey(out, "shard", first); writeIntText(r.shard, out); - writeKey(out, "gen", first); writeU64StringValue(out, r.generation); + writeKey(out, "generation", first); writeU64StringValue(out, r.generation); closeObject(out, first); } @@ -205,8 +205,8 @@ String encodeFoldSeal(const CasFoldSeal & seal) /// meta line { bool first = true; - writeKey(out, "g", first); writeU64StringValue(out, seal.generation); - writeKey(out, "pg", first); writeU64StringValue(out, seal.parent_generation); + writeKey(out, "generation", first); writeU64StringValue(out, seal.generation); + writeKey(out, "parent_generation", first); writeU64StringValue(out, seal.parent_generation); closeObject(out, first); closeLine("meta"); } @@ -263,24 +263,24 @@ String encodeFoldSeal(const CasFoldSeal & seal) life_state.cleanup_evidence->remove_txn_id.ref_sequence); bool first = true; - writeKey(out, "k", first); writeStringValue(out, "rfl"); - writeKey(out, "life", first); writeHex128Value(out, life_id); - writeKey(out, "cls", first); writeIntText(static_cast(cov.classification), out); - writeKey(out, "lfe", first); writeU64StringValue(out, cov.last_folded_ref_id.writer_epoch); - writeKey(out, "lfs", first); writeU64StringValue(out, cov.last_folded_ref_id.ref_sequence); + writeKey(out, "kind", first); writeStringValue(out, "rfl"); + writeKey(out, "life_id", first); writeHex128Value(out, life_id); + writeKey(out, "classification", first); writeIntText(static_cast(cov.classification), out); + writeKey(out, "last_folded_writer_epoch", first); writeU64StringValue(out, cov.last_folded_ref_id.writer_epoch); + writeKey(out, "last_folded_ref_sequence", first); writeU64StringValue(out, cov.last_folded_ref_id.ref_sequence); if (cov.hold) { - writeKey(out, "hr", first); writeStringValue(out, holdReasonToWord(cov.hold->reason)); - writeKey(out, "hpe", first); writeU64StringValue(out, cov.hold->offending_position.writer_epoch); - writeKey(out, "hps", first); writeU64StringValue(out, cov.hold->offending_position.ref_sequence); - writeKey(out, "hrc", first); writeIntText(cov.hold->retry_count, out); - writeKey(out, "hnr", first); writeU64StringValue(out, cov.hold->next_retry_round); + writeKey(out, "hold_reason", first); writeStringValue(out, holdReasonToWord(cov.hold->reason)); + writeKey(out, "hold_position_writer_epoch", first); writeU64StringValue(out, cov.hold->offending_position.writer_epoch); + writeKey(out, "hold_position_ref_sequence", first); writeU64StringValue(out, cov.hold->offending_position.ref_sequence); + writeKey(out, "hold_retry_count", first); writeIntText(cov.hold->retry_count, out); + writeKey(out, "hold_next_retry_round", first); writeU64StringValue(out, cov.hold->next_retry_round); } if (life_state.cleanup_evidence) { - writeKey(out, "rte", first); + writeKey(out, "remove_txn_writer_epoch", first); writeU64StringValue(out, life_state.cleanup_evidence->remove_txn_id.writer_epoch); - writeKey(out, "rts", first); + writeKey(out, "remove_txn_ref_sequence", first); writeU64StringValue(out, life_state.cleanup_evidence->remove_txn_id.ref_sequence); } closeObject(out, first); @@ -303,11 +303,11 @@ String encodeFoldSeal(const CasFoldSeal & seal) for (const auto & [shard, s] : seal.condemned_summary) { bool first = true; - writeKey(out, "k", first); writeStringValue(out, "cnd"); + writeKey(out, "kind", first); writeStringValue(out, "cnd"); writeKey(out, "shard", first); writeIntText(shard, out); - writeKey(out, "ct", first); writeIntText(s.condemned_total, out); - writeKey(out, "pt", first); writeIntText(s.pending_total, out); - writeKey(out, "ocr", first); writeU64StringValue(out, s.oldest_nonpending_condemn_round); + writeKey(out, "condemned_total", first); writeIntText(s.condemned_total, out); + writeKey(out, "pending_total", first); writeIntText(s.pending_total, out); + writeKey(out, "oldest_nonpending_condemn_round", first); writeU64StringValue(out, s.oldest_nonpending_condemn_round); closeObject(out, first); closeLine("cnd"); ++n; @@ -338,8 +338,8 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect String key; while (r.nextKey(key)) { - if (key == "g") seal.generation = r.readU64String(); - else if (key == "pg") seal.parent_generation = r.readU64String(); + if (key == "generation") seal.generation = r.readU64String(); + else if (key == "parent_generation") seal.parent_generation = r.readU64String(); else r.skipUnknown(key); /// Strict => any unknown key is CORRUPTED_DATA } } @@ -354,7 +354,7 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: empty line"); - if (key == "n") + if (key == "record_count") { const uint64_t n = r.readU64Number(); if (r.nextKey(key)) @@ -370,8 +370,8 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect seal.generation, *expected_generation); return seal; } - if (key != "k") - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: record must start with \"k\""); + if (key != "kind") + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: record must start with \"kind\""); const String kind = r.readString(); if (kind == "rfl") @@ -395,17 +395,17 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect std::optional remove_txn_sequence; while (r.nextKey(key)) { - if (key == "life") life_id = r.readHex128(); - else if (key == "cls") classification = r.readU64Number(); - else if (key == "lfe") cov.last_folded_ref_id.writer_epoch = r.readU64String(); - else if (key == "lfs") cov.last_folded_ref_id.ref_sequence = r.readU64String(); - else if (key == "hr") hold_reason = holdReasonFromWord(r.readString()); - else if (key == "hpe") hold_epoch = r.readU64String(); - else if (key == "hps") hold_sequence = r.readU64String(); - else if (key == "hrc") hold_retry_count = r.readU32Number(); - else if (key == "hnr") hold_next_retry_round = r.readU64String(); - else if (key == "rte") remove_txn_epoch = r.readU64String(); - else if (key == "rts") remove_txn_sequence = r.readU64String(); + if (key == "life_id") life_id = r.readHex128(); + else if (key == "classification") classification = r.readU64Number(); + else if (key == "last_folded_writer_epoch") cov.last_folded_ref_id.writer_epoch = r.readU64String(); + else if (key == "last_folded_ref_sequence") cov.last_folded_ref_id.ref_sequence = r.readU64String(); + else if (key == "hold_reason") hold_reason = holdReasonFromWord(r.readString()); + else if (key == "hold_position_writer_epoch") hold_epoch = r.readU64String(); + else if (key == "hold_position_ref_sequence") hold_sequence = r.readU64String(); + else if (key == "hold_retry_count") hold_retry_count = r.readU32Number(); + else if (key == "hold_next_retry_round") hold_next_retry_round = r.readU64String(); + else if (key == "remove_txn_writer_epoch") remove_txn_epoch = r.readU64String(); + else if (key == "remove_txn_ref_sequence") remove_txn_sequence = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown rfl key '{}'", key); } @@ -417,7 +417,7 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect /// `cls` is required, not defaulted: an absent one would read as 0 ("no round folded this /// namespace"), which is a claim about a fold, not the absence of one. if (!classification) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: rfl '{}' missing cls", life_hex); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: rfl '{}' missing classification", life_hex); if (!isKnownClassification(*classification)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: coverage '{}' has classification {}, which is not one of the four " @@ -487,9 +487,9 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect while (r.nextKey(key)) { if (key == "key") run_key = r.readString(); - else if (key == "ck") checksum = r.readHex128(); + else if (key == "checksum") checksum = r.readHex128(); else if (key == "shard") shard = r.readU64Number(); - else if (key == "gen") generation = r.readU64String(); + else if (key == "generation") generation = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown run key '{}'", key); } if (!run_key || !checksum || !shard || !generation) @@ -507,9 +507,9 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect while (r.nextKey(key)) { if (key == "shard") shard = r.readU64Number(); - else if (key == "ct") condemned_total = r.readU64Number(); - else if (key == "pt") pending_total = r.readU64Number(); - else if (key == "ocr") oldest_nonpending_condemn_round = r.readU64String(); + else if (key == "condemned_total") condemned_total = r.readU64Number(); + else if (key == "pending_total") pending_total = r.readU64Number(); + else if (key == "oldest_nonpending_condemn_round") oldest_nonpending_condemn_round = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown cnd key '{}'", key); } if (!shard || !condemned_total || !pending_total || !oldest_nonpending_condemn_round) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp index 362306691473..ac67d8b6eb63 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp @@ -145,7 +145,7 @@ constexpr uint64_t kMiB = 1024 * 1024; /// small raw singletons. constexpr FormatTraits TRAITS[] = { - {FormatId::Blob, "cas_blob", TextFamily::PayloadHybrid, KeyStrictness::Tolerant, CompressionPolicy::Never, 256, 256}, + {FormatId::Blob, "cas_blob", TextFamily::PayloadHybrid, KeyStrictness::Tolerant, CompressionPolicy::Never, 512, 512}, {FormatId::BlobMeta, "cas_blob_meta", TextFamily::Control, KeyStrictness::Tolerant, CompressionPolicy::Never, 1 * kMiB, 64 * kKiB}, {FormatId::PoolMeta, "cas_pool_meta", TextFamily::Control, KeyStrictness::Tolerant, CompressionPolicy::Never, 1 * kMiB, 64 * kKiB}, {FormatId::RefLog, "cas_ref_log", TextFamily::Control, KeyStrictness::Tolerant, CompressionPolicy::Always, 64 * kMiB, 64 * kMiB}, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp index c5dda3286ad6..e85f236563b7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp @@ -22,7 +22,7 @@ String encodeGcMaintenanceState(const GcMaintenanceState & state) CasJsonWriter out; writeHeaderLine(out, FormatId::GcMaintenanceState); bool first = true; - writeKey(out, "cur", first); + writeKey(out, "janitor_cursor", first); writeStringValue(out, state.janitor_cursor); closeObject(out, first); writeChar('\n', out); @@ -46,7 +46,7 @@ GcMaintenanceState decodeGcMaintenanceState(std::string_view data) String key; while (reader.nextKey(key)) { - if (key == "cur") + if (key == "janitor_cursor") { result.janitor_cursor = reader.readString(); has_cursor = true; @@ -55,7 +55,7 @@ GcMaintenanceState decodeGcMaintenanceState(std::string_view data) reader.skipUnknown(key); } if (!has_cursor) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc maintenance state: missing cur"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc maintenance state: missing janitor_cursor"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc maintenance state: trailing bytes"); if (result.janitor_cursor.size() > kMaxGcMaintenanceCursorBytes) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp index e69ea98a799e..8422ebd46912 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp @@ -48,11 +48,11 @@ String encodeOutcomeLog(const OutcomeLog & log) for (const OutcomeEntry & e : log.entries) { bool first = true; - writeKey(out, "k", first); + writeKey(out, "kind", first); writeStringValue(out, objectKindToWord(e.kind)); writeBlobRefFields(out, first, e.ref); /// ha + h writeTokenFields(out, first, e.token); /// tt + tv - writeKey(out, "oc", first); + writeKey(out, "outcome", first); writeStringValue(out, outcomeKindToWord(e.outcome)); closeObject(out, first); writeChar('\n', out); @@ -78,7 +78,7 @@ OutcomeLog decodeOutcomeLog(std::string_view data) /// The first key distinguishes a trailer ("n") from a record ("k"). if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: empty line"); - if (key == "n") + if (key == "record_count") { const uint64_t n = r.readU64Number(); while (r.nextKey(key)) @@ -101,17 +101,18 @@ OutcomeLog decodeOutcomeLog(std::string_view data) TokenType tt{}; do { - if (key == "k") e.kind = objectKindFromWord(r.readString(), "outcome log"); - else if (key == "ha") { ha = r.readString(); have_ha = true; } - else if (key == "h") { hhex = r.readString(); have_h = true; } - else if (key == "tt") { tt = tokenTypeFromWord(r.readString(), "outcome log"); have_tt = true; } - else if (key == "tv") tv = r.readString(); - else if (key == "oc") e.outcome = outcomeKindFromWord(r.readString()); + if (key == "kind") e.kind = objectKindFromWord(r.readString(), "outcome log"); + else if (key == "hash_algorithm") { ha = r.readString(); have_ha = true; } + else if (key == "hash") { hhex = r.readString(); have_h = true; } + else if (key == "token_type") { tt = tokenTypeFromWord(r.readString(), "outcome log"); have_tt = true; } + else if (key == "token_value") tv = r.readString(); + else if (key == "outcome") e.outcome = outcomeKindFromWord(r.readString()); else r.skipUnknown(key); } while (r.nextKey(key)); if (!have_ha || !have_h || !have_tt) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: record missing ha/h/tt"); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS outcome log: record missing hash_algorithm/hash/token_type"); const BlobHashAlgo algo = blobHashAlgoFromWord(ha, "outcome log"); /// Validate the digest width before `fromHex`: a width mismatch must surface as the /// CORRUPTED_DATA required for malformed serialized input, not fromHex's BAD_ARGUMENTS. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h index 09a850ee66ff..647cf489aa88 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h @@ -17,7 +17,7 @@ namespace DB::Cas /// the one-pass merge found a live in-degree. The log is written before the round's single state CAS; /// `putIfAbsent` adopts an existing durable log on replay rather than treating a byte difference as /// an error. The uncompressed payload is a header line, one flat JSON record per entry in insertion -/// order, and an `{"n":count}` trailer. `FormatId::GcOutcomes` stores the sealed payload in one zstd +/// order, and a `{"record_count":count}` trailer. `FormatId::GcOutcomes` stores the sealed payload in one zstd /// frame, so its object-storage key has the `.zst` suffix. enum class OutcomeKind : uint8_t { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp index 7012c6787f70..058417e718f6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp @@ -23,14 +23,14 @@ String encodeGcState(const GcState & state) CasJsonWriter out(256); writeHeaderLine(out, FormatId::GcState); bool first = true; - writeKey(out, "rnd", first); writeU64StringValue(out, state.round); - writeKey(out, "gcs", first); writeIntText(state.gc_shards, out); - writeKey(out, "sg", first); writeU64StringValue(out, state.snap_generation); - writeKey(out, "spt", first); writeU64StringValue(out, state.snap_pruned_through); - writeKey(out, "sa", first); writeU64StringValue(out, state.snap_attempt); - writeKey(out, "msc", first); writeStringValue(out, state.manifest_sweep_cursor); - writeKey(out, "lo", first); writeHex128Value(out, state.lease.owner); - writeKey(out, "ls", first); writeU64StringValue(out, state.lease.seq); + writeKey(out, "round", first); writeU64StringValue(out, state.round); + writeKey(out, "gc_shards", first); writeIntText(state.gc_shards, out); + writeKey(out, "snapshot_generation", first); writeU64StringValue(out, state.snap_generation); + writeKey(out, "snapshot_pruned_through", first); writeU64StringValue(out, state.snap_pruned_through); + writeKey(out, "snapshot_attempt", first); writeU64StringValue(out, state.snap_attempt); + writeKey(out, "manifest_sweep_cursor", first); writeStringValue(out, state.manifest_sweep_cursor); + writeKey(out, "lease_owner", first); writeHex128Value(out, state.lease.owner); + writeKey(out, "lease_sequence", first); writeU64StringValue(out, state.lease.seq); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -49,20 +49,20 @@ GcState decodeGcState(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "rnd") state.round = r.readU64String(); - else if (key == "gcs") { state.gc_shards = r.readU64Number(); saw_gcs = true; } - else if (key == "sg") state.snap_generation = r.readU64String(); - else if (key == "spt") state.snap_pruned_through = r.readU64String(); - else if (key == "sa") state.snap_attempt = r.readU64String(); - else if (key == "msc") state.manifest_sweep_cursor = r.readString(); - else if (key == "lo") state.lease.owner = r.readHex128(); - else if (key == "ls") state.lease.seq = r.readU64String(); + if (key == "round") state.round = r.readU64String(); + else if (key == "gc_shards") { state.gc_shards = r.readU64Number(); saw_gcs = true; } + else if (key == "snapshot_generation") state.snap_generation = r.readU64String(); + else if (key == "snapshot_pruned_through") state.snap_pruned_through = r.readU64String(); + else if (key == "snapshot_attempt") state.snap_attempt = r.readU64String(); + else if (key == "manifest_sweep_cursor") state.manifest_sweep_cursor = r.readString(); + else if (key == "lease_owner") state.lease.owner = r.readHex128(); + else if (key == "lease_sequence") state.lease.seq = r.readU64String(); else r.skipUnknown(key); } - /// Fail closed on an absent gcs: the writer always emits it, so a missing key means a corrupt object. + /// Fail closed on an absent `gc_shards`: the writer always emits it, so a missing key means a corrupt object. /// Do NOT silently keep the struct default (1) — that would hide corruption (no-fallback principle). if (!saw_gcs) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: missing gcs"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: missing gc_shards"); if (state.gc_shards == 0) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: gc_shards must be >= 1"); if (!body_in.eof() || !in.eof()) @@ -75,8 +75,8 @@ String encodeGcHeartbeat(const GcHeartbeat & hb) CasJsonWriter out(256); writeHeaderLine(out, FormatId::GcHeartbeat); bool first = true; - writeKey(out, "by", first); writeHex128Value(out, hb.owner); - writeKey(out, "seq", first); writeU64StringValue(out, hb.hb_seq); + writeKey(out, "owner", first); writeHex128Value(out, hb.owner); + writeKey(out, "heartbeat_sequence", first); writeU64StringValue(out, hb.hb_seq); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -96,12 +96,12 @@ GcHeartbeat decodeGcHeartbeat(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "by") + if (key == "owner") { hb.owner = r.readHex128(); saw_by = true; } - else if (key == "seq") + else if (key == "heartbeat_sequence") { hb.hb_seq = r.readU64String(); saw_seq = true; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp index 5e7f9ff5ffbf..4c6b55f2e705 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp @@ -39,23 +39,23 @@ EntryPlacement placementFromWord(std::string_view w) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: unknown placement '{}'", w); } -/// One entry-record line: {"p","pm", then either the Blob's "ha"/"h"/"sz" or the Inline's "il"}. +/// One entry-record line contains `path`, `placement`, and the placement-specific fields. void writeEntryRecord(CasJsonWriter & out, const ManifestEntry & e) { bool first = true; - writeKey(out, "p", first); + writeKey(out, "path", first); writeStringValue(out, e.path); - writeKey(out, "pm", first); + writeKey(out, "placement", first); writeStringValue(out, placementToWord(e.placement)); if (e.placement == EntryPlacement::Blob) { writeBlobRefFields(out, first, e.ref); /// ha + h - writeKey(out, "sz", first); + writeKey(out, "size", first); writeIntText(e.blob_size, out); } else { - writeKey(out, "il", first); + writeKey(out, "inline_length", first); writeIntText(e.inline_bytes.size(), out); } closeObject(out, first); @@ -102,9 +102,9 @@ String encodePartManifest(const PartManifest & m) { bool first = true; writeManifestRefFields(out, first, "", m.ref); - writeKey(out, "ns", first); + writeKey(out, "namespace", first); writeStringValue(out, m.root_namespace_id.string()); - writeKey(out, "pd", first); + writeKey(out, "payload_digest", first); writeHex128Value(out, m.payload_digest); closeObject(out, first); writeChar('\n', out); @@ -152,19 +152,20 @@ PartManifest decodePartManifest(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "me") me = r.readU64String(); - else if (key == "mb") mb = r.readU64String(); - else if (key == "mo") mo = r.readU64Number(); - else if (key == "ns") ns = r.readString(); - else if (key == "pd") pd = r.readHex128(); + if (key == "writer_epoch") me = r.readU64String(); + else if (key == "build_sequence") mb = r.readU64String(); + else if (key == "manifest_ordinal") mo = r.readU64Number(); + else if (key == "namespace") ns = r.readString(); + else if (key == "payload_digest") pd = r.readHex128(); else r.skipUnknown(key); } if (!me || !mb || !mo) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing me/mb/mo"); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "PartManifest: descriptor missing writer_epoch/build_sequence/manifest_ordinal"); if (!ns) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing ns"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing namespace"); if (!pd) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing pd"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing payload_digest"); m.ref = manifestRefFromFields(*me, *mb, *mo, "PartManifest", "descriptor"); m.root_namespace_id = RootNamespace(*ns); m.payload_digest = *pd; @@ -185,7 +186,7 @@ PartManifest decodePartManifest(std::string_view data) if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: empty line"); - if (key == "n") + if (key == "record_count") { const uint64_t declared_n = r.readU64Number(); while (r.nextKey(key)) @@ -198,8 +199,8 @@ PartManifest decodePartManifest(std::string_view data) break; } - if (key != "p") - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: record must start with \"p\""); + if (key != "path") + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: record must start with \"path\""); ManifestEntry e; e.path = r.readString(); @@ -224,23 +225,24 @@ PartManifest decodePartManifest(std::string_view data) std::optional il; while (r.nextKey(key)) { - if (key == "pm") pm = r.readString(); - else if (key == "ha") ha = r.readString(); - else if (key == "h") h = r.readString(); - else if (key == "sz") sz = r.readU64Number(); - else if (key == "il") il = r.readU64Number(); + if (key == "placement") pm = r.readString(); + else if (key == "hash_algorithm") ha = r.readString(); + else if (key == "hash") h = r.readString(); + else if (key == "size") sz = r.readU64Number(); + else if (key == "inline_length") il = r.readU64Number(); else r.skipUnknown(key); } if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: junk after record"); if (!pm) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: entry '{}' missing pm", e.path); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: entry '{}' missing placement", e.path); e.placement = placementFromWord(*pm); if (e.placement == EntryPlacement::Blob) { if (!ha || !h || !sz) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: blob entry '{}' missing ha/h/sz", e.path); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "PartManifest: blob entry '{}' missing hash_algorithm/hash/size", e.path); const BlobHashAlgo algo = blobHashAlgoFromWord(*ha, "PartManifest entry"); /// Validate the digest width before calling `fromHex`. A width mismatch otherwise /// produces `BAD_ARGUMENTS` instead of the `CORRUPTED_DATA` required for malformed @@ -258,7 +260,7 @@ PartManifest decodePartManifest(std::string_view data) else { if (!il) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: inline entry '{}' missing il", e.path); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: inline entry '{}' missing inline_length", e.path); inline_lens.push_back(*il); /// bytes filled from the payload zone below } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h index f2a416d15743..9ba7ce1c80df 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h @@ -14,13 +14,11 @@ namespace DB::Cas /// made of JSON lines followed by a raw payload zone. The public types and helper signatures remain /// stable for the surrounding CAS protocol. /// -/// header line {"type":"cas_part_manifest","v":N} -/// descriptor meta line {"me","mb","mo"} (the ManifestRef, shared rendering with -/// refsnaplog, `CasWireVocab.h`) + "ns" (root namespace) + "pd" +/// header line {"type":"cas_part_manifest","version":N} +/// descriptor meta line the `ManifestRef` fields plus `namespace` and `payload_digest` /// (payload digest, 32 lowercase hex) -/// one entry-record line each {"p":path,"pm":placement-word, then either the Blob's -/// {"ha","h","sz"} or the Inline's {"il"}}, in canonical path order -/// trailer line {"n":entry-count} +/// one entry-record line each `path`, `placement`, and placement-specific fields, in path order +/// trailer line {"record_count":entry-count} /// PAYLOAD ZONE (raw, follows the trailer): for each Inline entry, in path order, a /// `head -v`-style banner line `==> "" il= <==\n`, then /// exactly `n` raw bytes, then `\n`. The path uses the same writer as diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp index 50e9843e9254..464d5408a87b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp @@ -19,21 +19,23 @@ namespace DB::Cas /// Minimum `blob_header_len` that provably fits the v3 `cas_blob` JSON envelope's mandatory (always- /// written) non-ref fields, computed at type maxima from `encodeEnvelopeHeader` (CasBlobEnvelopeFormat.cpp): /// {"type":"cas_blob" 18 -/// ,"v": 5 + 10 (currentCompatibilityVersion) 15 -/// ,"tag":"<32 hex>" 7 + 34 41 -/// ,"bld":"<32 hex>" 7 + 34 41 -/// ,"ts": 6 + 20 (created_at_ms) 26 -/// ,"by":"<32 hex>" 7 + 34 41 -/// ,"op":"" 6 + 10 (longest op word "mutation") 16 -/// ,"ch": 6 + 10 (VERSION_INTEGER) 16 -/// non-ref JSON = 214 bytes -/// The encoder then always frames the ref: `,"ref":` (7) + `""` (2) + `}` (1), and reserves byte -/// blob_header_len-1 for '\n' (1) = 11 bytes. So the mandatory content needs 214 + 11 = 225 bytes; -/// below that, encodeEnvelopeHeader throws LOGICAL_ERROR on the FIRST blob write (the old drop-and-retry -/// that used to mask this is gone). We floor at 240 (a multiple of 8 comfortably above 225, leaving -/// >= 15 bytes for the diagnostic ref even at type maxima, and well under the 256 default) so a -/// misconfigured pool fails at CREATION with BAD_ARGUMENTS, not at first write with LOGICAL_ERROR. -static constexpr uint64_t kMinBlobHeaderLen = 240; +/// ,"version": 11 + 10 21 +/// ,"incarnation_tag":"<32 hex>" 19 + 34 53 +/// ,"build_id":"<32 hex>" 12 + 34 46 +/// ,"created_at_ms": 17 + 20 37 +/// ,"creator_server_id":"<32 hex>" 21 + 34 55 +/// ,"operation":"" 13 + 10 23 +/// (`mutation`: 8 + 2 quotes) +/// ,"clickhouse_version": 22 + 10 32 +/// non-ref JSON = 285 bytes +/// The encoder then always frames the ref: `,"intended_ref":` (16) + `""` (2) + `}` (1), and +/// reserves byte blob_header_len-1 for '\n' (1) = 20 bytes. The mandatory content therefore needs +/// 285 + 20 = 305 bytes. Below that, `encodeEnvelopeHeader` throws `LOGICAL_ERROR` on the first blob +/// write (the old drop-and-retry that used to mask this is gone). We floor at 320, a multiple of 8 +/// comfortably above 305. This leaves exactly 15 bytes for the diagnostic ref at the floor (and more +/// above it) and is well under the 384 default, so a misconfigured pool fails at creation with +/// `BAD_ARGUMENTS`, not at the first write with `LOGICAL_ERROR`. +static constexpr uint64_t kMinBlobHeaderLen = 320; void validatePoolBlobHeaderLen(uint64_t blob_header_len, int error_code, std::string_view what) { @@ -73,15 +75,15 @@ String encodePoolMeta(const PoolMeta & pm) writeHeaderLine(out, FormatId::PoolMeta); bool first = true; - writeKey(out, "pid", first); + writeKey(out, "pool_id", first); writeHex128Value(out, pm.pool_id); - writeKey(out, "hln", first); + writeKey(out, "blob_header_len", first); writeIntText(pm.blob_header_len, out); - writeKey(out, "gcs", first); + writeKey(out, "gc_shards", first); writeIntText(pm.gc_shards, out); - writeKey(out, "mrg", first); + writeKey(out, "min_reader_generation", first); writeIntText(pm.min_reader_generation, out); - writeKey(out, "alg", first); + writeKey(out, "algorithms_used", first); { /// Comma-joined algo words (tiny list, <=3): "ch128" or "ch128,sha256". String joined; @@ -127,21 +129,21 @@ PoolMeta decodePoolMeta(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "pid") + if (key == "pool_id") { pm.pool_id = r.readHex128(); saw_pid = true; } - else if (key == "hln") + else if (key == "blob_header_len") pm.blob_header_len = r.readU64Number(); - else if (key == "gcs") + else if (key == "gc_shards") { pm.gc_shards = r.readU64Number(); saw_gc_shards = true; } - else if (key == "mrg") + else if (key == "min_reader_generation") pm.min_reader_generation = r.readU64Number(); - else if (key == "alg") + else if (key == "algorithms_used") { const String joined = r.readString(); size_t start = 0; @@ -161,9 +163,9 @@ PoolMeta decodePoolMeta(std::string_view data) r.skipUnknown(key); } if (!saw_pid) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing pid"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing pool_id"); if (!saw_gc_shards || pm.gc_shards == 0) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing or zero gcs"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing or zero gc_shards"); if (!body_in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: junk after body object"); if (!in.eof()) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h index 2ca0894d2f01..0a142f112fe6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h @@ -15,7 +15,8 @@ class Layout; /// `_pool_meta` — the pool identity and the pool-wide constants that every reader and writer must /// agree on. The v3 text representation is a header line followed by one JSON body object: -/// {"pid":"<32hex>","hln":,"mrg":,"alg":""}. +/// {"pool_id":"<32hex>","blob_header_len":,"min_reader_generation":, +/// "algorithms_used":""}. /// /// The persisted object is authoritative after creation. On reopen, `createOrValidate` uses its /// `blob_header_len` and reader-generation floor rather than replacing them with local configuration; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp index b21458aaf6e2..ee88396e1e75 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp @@ -107,7 +107,7 @@ void writeRunHeaderLine(WriteBuffer & out, std::string_view kind) bool first = true; writeKey(line, "type", first); writeStringValue(line, t.type); - writeKey(line, "v", first); + writeKey(line, "version", first); writeIntText(currentCompatibilityVersion(), line); writeKey(line, "kind", first); writeStringValue(line, kind); @@ -131,8 +131,8 @@ void expectRunHeaderLine(ReadBuffer & in, std::string_view expected_kind) if (type != t.type) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: object is a '{}', not a '{}'", type, t.type); - if (!r.nextKey(key) || key != "v") - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: header line must carry \"v\" second"); + if (!r.nextKey(key) || key != "version") + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: header line must carry \"version\" second"); const uint32_t v = r.readU32Number(); checkCompatibility(v, t.type); @@ -174,22 +174,22 @@ void SourceEdgeRunWriter::append(const SourceEdgeRecord & rec) scratch.clear(); bool first = true; - writeKey(scratch, "b", first); + writeKey(scratch, "blob_ref", first); writeStringValue(scratch, renderB(rec.ref)); - writeKey(scratch, "s", first); + writeKey(scratch, "source_id", first); writeHex128Value(scratch, rec.source_id); - writeKey(scratch, "m", first); + writeKey(scratch, "marker", first); writeStringValue(scratch, markerToWord(rec.marker)); if (rec.marker == kCondemned) { - writeKey(scratch, "pend", first); + writeKey(scratch, "delete_pending", first); writeBoolValue(scratch, rec.delete_pending); - writeTokenFields(scratch, first, rec.token); /// tt + tv - writeKey(scratch, "sz", first); + writeTokenFields(scratch, first, rec.token); /// token_type + token_value + writeKey(scratch, "size", first); writeIntText(rec.size, scratch); - writeKey(scratch, "cr", first); + writeKey(scratch, "condemn_round", first); writeU64StringValue(scratch, rec.condemn_round); - writeKey(scratch, "mc", first); + writeKey(scratch, "marker_confirmed", first); writeBoolValue(scratch, rec.marker_confirmed); } closeObject(scratch, first); @@ -243,7 +243,7 @@ bool SourceEdgeRunReader::next(SourceEdgeRecord & rec) if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: empty line"); - if (key == "n") + if (key == "record_count") { const uint64_t n = r.readU64Number(); if (r.nextKey(key)) @@ -276,25 +276,26 @@ bool SourceEdgeRunReader::next(SourceEdgeRecord & rec) TokenType tt{}; do { - if (key == "b") { b = r.readString(); have_b = true; } - else if (key == "s") { out.source_id = r.readHex128(); have_s = true; } - else if (key == "m") { out.marker = markerFromWord(r.readString()); have_m = true; } - else if (key == "pend") { out.delete_pending = r.readBool(); have_pend = true; } - else if (key == "tt") { tt = tokenTypeFromWord(r.readString(), "cas_run"); have_tt = true; } - else if (key == "tv") { tv = r.readString(); have_tv = true; } - else if (key == "sz") { out.size = r.readU64Number(); have_sz = true; } - else if (key == "cr") { out.condemn_round = r.readU64String(); have_cr = true; } - else if (key == "mc") { out.marker_confirmed = r.readBool(); have_mc = true; } + if (key == "blob_ref") { b = r.readString(); have_b = true; } + else if (key == "source_id") { out.source_id = r.readHex128(); have_s = true; } + else if (key == "marker") { out.marker = markerFromWord(r.readString()); have_m = true; } + else if (key == "delete_pending") { out.delete_pending = r.readBool(); have_pend = true; } + else if (key == "token_type") { tt = tokenTypeFromWord(r.readString(), "cas_run"); have_tt = true; } + else if (key == "token_value") { tv = r.readString(); have_tv = true; } + else if (key == "size") { out.size = r.readU64Number(); have_sz = true; } + else if (key == "condemn_round") { out.condemn_round = r.readU64String(); have_cr = true; } + else if (key == "marker_confirmed") { out.marker_confirmed = r.readBool(); have_mc = true; } else r.skipUnknown(key); /// Strict => any unknown key is CORRUPTED_DATA } while (r.nextKey(key)); if (!have_b || !have_s || !have_m) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: record missing b/s/m"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: record missing blob_ref/source_id/marker"); out.ref = parseB(b); if (out.marker == kCondemned) { if (!have_pend || !have_tt || !have_tv || !have_sz || !have_cr || !have_mc) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: condemned record missing pend/tt/tv/sz/cr/mc"); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS cas_run: condemned record missing delete_pending/token_type/token_value/size/condemn_round/marker_confirmed"); out.token = Token{tv, tt}; } else if (have_pend || have_tt || have_tv || have_sz || have_cr || have_mc) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h index d5f9a4801caf..e3742477d97c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h @@ -40,17 +40,18 @@ constexpr char kCondemned = 0x02; /// leaking into the format implementation. /// /// File shape: -/// {"type":"cas_run","v":3,"kind":"source_edge"} header line (type + v + kind gate) -/// {"b":"01","s":"<32hex>","m":"edge"} an active-edge / zero-marker row -/// {"b":"01","s":"00000000000000000000000000000000","m":"condemned","pend":false,"tt":"etag","tv":"...","sz":123,"cr":"7","mc":false} -/// {"n":184267} trailer: record count +/// {"type":"cas_run","version":3,"kind":"source_edge"} header line (type + version + kind gate) +/// {"blob_ref":"01","source_id":"<32hex>","marker":"edge"} an active-edge / zero-marker row +/// {"record_count":184267} trailer: record count /// -/// The record key `b` is the algo BYTE as two lowercase hex chars followed by the digest hex at the -/// algo's width; `s` is the 32-hex source id. String-sorting records by (b, s) reproduces the current +/// The record key `blob_ref` is the algorithm byte as two lowercase hex chars followed by the digest +/// hex at the algorithm's width; `source_id` is the 32-hex source id. String-sorting records by +/// (`blob_ref`, `source_id`) reproduces the current /// `(algorithm, digest, source_id)` byte order (lowercase hex preserves unsigned byte order and the /// algorithm byte is emitted first) — the invariant the fold's two-cursor merge depends on. The row-tag word -/// `m` maps to the `kEdgeActive`/`kZeroMarker`/`kCondemned` bytes; a `condemned` row additionally -/// carries the retired incarnation (`pend`/`tt`/`tv`/`sz`/`cr`) and the durable condemn-marker +/// `marker` maps to the `kEdgeActive`/`kZeroMarker`/`kCondemned` bytes; a `condemned` row additionally +/// carries the retired incarnation (`delete_pending`/`token_type`/`token_value`/`size`/`condemn_round`) +/// and the durable condemn-marker /// confirmation bit (`mc`). /// One decoded source-edge row. All fields are identifier-layer types so the codec stays backend-free. @@ -71,7 +72,7 @@ struct SourceEdgeRecord /// The header-line `kind` word for the only live `cas_run` kind. inline constexpr std::string_view kSourceEdgeKindWord = "source_edge"; -/// Write the typed header line `{"type":"cas_run","v":G_BUILD,"kind":""}\n` with a fixed key +/// Write the typed header line `{"type":"cas_run","version":G_BUILD,"kind":""}\n` with a fixed key /// order for byte-determinism. The `kind` field distinguishes the record schema within the run /// family, so a reader can reject a valid run of the wrong kind before interpreting any records. void writeRunHeaderLine(WriteBuffer & out, std::string_view kind); @@ -86,7 +87,7 @@ void expectRunHeaderLine(ReadBuffer & in, std::string_view expected_kind); /// object checksum is `sourceEdgeRunChecksum` over the finished bytes, which keeps this writer free of a /// HashingWriteBuffer finalize-ordering hazard). `append` asserts records arrive in non-decreasing /// (ref, source_id) order and throws on a regression (this replaces the old `prev_key` monotonicity -/// check). `finish` writes the `{"n":count}` trailer. +/// check). `finish` writes the `{"record_count":count}` trailer. class SourceEdgeRunWriter { public: @@ -126,7 +127,7 @@ UInt128 sourceEdgeRunChecksum(std::string_view stored_bytes); /// Sequential streaming reader over a caller-owned `ReadBuffer` (backend-free, O(one 4 KiB line) /// resident). The ctor reads + gates the typed header line. `next` yields records in stored order and -/// returns false once the `{"n"}` trailer is consumed (the count is verified there — the line-truncation +/// returns false once the `{"record_count"}` trailer is consumed (the count is verified there — the line-truncation /// guard). Every byte read is fed through a chained CityHash128; after the trailer, `verifyAgainst` /// compares the accumulated whole-object hash to the seal's `RunRef.checksum` and throws `CORRUPTED_DATA` /// on a mismatch — the caller calls it after draining and BEFORE acting on the records (the deletion diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp index c5b119ba44ca..ca9268c27a02 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp @@ -136,19 +136,19 @@ String encodeRefCatalog(const RefCatalog & catalog) e.ns.string(), nsStateToWord(e.state), e.removal_started_round ? "carries" : "lacks"); bool first = true; - writeKey(out, "k", first); writeStringValue(out, "ent"); - writeKey(out, "ns", first); writeStringValue(out, e.ns.string()); - writeKey(out, "st", first); writeStringValue(out, nsStateToWord(e.state)); - writeKey(out, "inc", first); writeHex128Value(out, e.incarnation); + writeKey(out, "kind", first); writeStringValue(out, "ent"); + writeKey(out, "namespace", first); writeStringValue(out, e.ns.string()); + writeKey(out, "state", first); writeStringValue(out, nsStateToWord(e.state)); + writeKey(out, "incarnation", first); writeHex128Value(out, e.incarnation); if (e.removal_started_round) { - writeKey(out, "rsr", first); writeU64StringValue(out, *e.removal_started_round); + writeKey(out, "removal_started_round", first); writeU64StringValue(out, *e.removal_started_round); } if (e.creator) { - writeKey(out, "csr", first); writeStringValue(out, e.creator->server_root_id); - writeKey(out, "cwe", first); writeU64StringValue(out, e.creator->writer_epoch); - writeKey(out, "cfg", first); writeU64StringValue(out, e.creator->fence_generation); + writeKey(out, "creator_server_root_id", first); writeStringValue(out, e.creator->server_root_id); + writeKey(out, "creator_writer_epoch", first); writeU64StringValue(out, e.creator->writer_epoch); + writeKey(out, "creator_fence_generation", first); writeU64StringValue(out, e.creator->fence_generation); } closeObject(out, first); closeLine("ent"); @@ -178,7 +178,7 @@ RefCatalog decodeRefCatalog(std::string_view data) if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: empty line"); - if (key == "n") + if (key == "record_count") { const uint64_t n = r.readU64Number(); if (r.nextKey(key)) @@ -190,8 +190,8 @@ RefCatalog decodeRefCatalog(std::string_view data) "CAS ref catalog: trailer count {} != {} records", n, seen); return catalog; } - if (key != "k") - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: record must start with \"k\""); + if (key != "kind") + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: record must start with \"kind\""); const String kind = r.readString(); if (kind != "ent") throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown record kind '{}'", kind); @@ -205,23 +205,23 @@ RefCatalog decodeRefCatalog(std::string_view data) std::optional removal_started_round; while (r.nextKey(key)) { - if (key == "ns") ns_str = r.readString(); - else if (key == "st") st_word = r.readString(); - else if (key == "inc") inc = r.readHex128(); - else if (key == "csr") csr = r.readString(); - else if (key == "cwe") cwe = r.readU64String(); - else if (key == "cfg") cfg = r.readU64String(); - else if (key == "rsr") removal_started_round = r.readU64String(); - else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown ent key '{}'", key); + if (key == "namespace") ns_str = r.readString(); + else if (key == "state") st_word = r.readString(); + else if (key == "incarnation") inc = r.readHex128(); + else if (key == "creator_server_root_id") csr = r.readString(); + else if (key == "creator_writer_epoch") cwe = r.readU64String(); + else if (key == "creator_fence_generation") cfg = r.readU64String(); + else if (key == "removal_started_round") removal_started_round = r.readU64String(); + else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown entry key '{}'", key); } if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: junk after record"); if (!st_word) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing st", ns_str); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing state", ns_str); const NsState state = nsStateFromWord(*st_word); /// throws CORRUPTED_DATA on an unknown word - /// A missing "ns" key reads as the same empty string a present-but-empty one would, and both + /// A missing `namespace` key reads as the same empty string a present-but-empty one would, and both /// are refused identically here -- an empty namespace would sort first (every non-empty /// namespace compares strictly greater than ""), passing the canonical-order check below, and /// then wedge every later catalog-driven pass that tries to build a ref/namespace-file key @@ -234,7 +234,7 @@ RefCatalog decodeRefCatalog(std::string_view data) ns_str, ns_str.size(), kMaxNamespaceBytes); if (!inc) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing inc", ns_str); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing incarnation", ns_str); if (*inc == 0) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: namespace '{}' has a zero incarnation -- 0 never names a life", ns_str); @@ -295,7 +295,7 @@ uint64_t foldSealFixedBytes() seal.generation = std::numeric_limits::max(); seal.parent_generation = std::numeric_limits::max(); const uint64_t empty_bytes = encodeFoldSeal(seal).size(); - /// The empty trailer is `{"n":0}`. A real seal may carry a 20-digit record count. + /// The empty trailer is `{"record_count":0}`. A real seal may carry a 20-digit record count. return addByteBudget(empty_bytes, std::numeric_limits::digits10); }(); return bytes; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp index 6ff7fa5dda43..21f9c6004eeb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp @@ -90,15 +90,15 @@ String encodeRefCkpt(const RefCkpt & ckpt) /// three ref formats cannot disagree on the encoding. if (ckpt.life_epoch) { - writeKey(out, "le", first); + writeKey(out, "life_epoch", first); writeU64StringValue(out, *ckpt.life_epoch); } if (ckpt.committed_through) - writeRefTxnIdFields(out, first, "cte", "cts", *ckpt.committed_through); + writeRefTxnIdFields(out, first, "committed_through_writer_epoch", "committed_through_ref_sequence", *ckpt.committed_through); if (ckpt.checkpoint_snapshot_id) - writeRefTxnIdFields(out, first, "cse", "css", *ckpt.checkpoint_snapshot_id); + writeRefTxnIdFields(out, first, "checkpoint_snapshot_writer_epoch", "checkpoint_snapshot_ref_sequence", *ckpt.checkpoint_snapshot_id); if (ckpt.last_epoch_seal) - writeRefTxnIdFields(out, first, "lse", "lss", *ckpt.last_epoch_seal); + writeRefTxnIdFields(out, first, "last_epoch_seal_writer_epoch", "last_epoch_seal_ref_sequence", *ckpt.last_epoch_seal); closeObject(out, first); writeChar('\n', out); @@ -136,13 +136,13 @@ RefCkpt decodeRefCkpt(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "le") ckpt.life_epoch = r.readU64String(); - else if (key == "cte") cte = r.readU64String(); - else if (key == "cts") cts = r.readU64String(); - else if (key == "cse") cse = r.readU64String(); - else if (key == "css") css = r.readU64String(); - else if (key == "lse") lse = r.readU64String(); - else if (key == "lss") lss = r.readU64String(); + if (key == "life_epoch") ckpt.life_epoch = r.readU64String(); + else if (key == "committed_through_writer_epoch") cte = r.readU64String(); + else if (key == "committed_through_ref_sequence") cts = r.readU64String(); + else if (key == "checkpoint_snapshot_writer_epoch") cse = r.readU64String(); + else if (key == "checkpoint_snapshot_ref_sequence") css = r.readU64String(); + else if (key == "last_epoch_seal_writer_epoch") lse = r.readU64String(); + else if (key == "last_epoch_seal_ref_sequence") lss = r.readU64String(); else r.skipUnknown(key); } @@ -154,19 +154,22 @@ RefCkpt decodeRefCkpt(std::string_view data) if (cse || css) { if (!cse || !css) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: checkpoint_snapshot_id needs both cse and css"); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS cas_ref_ckpt: checkpoint snapshot needs both checkpoint_snapshot_writer_epoch and checkpoint_snapshot_ref_sequence"); ckpt.checkpoint_snapshot_id = RefTxnId{*cse, *css}; } if (cte || cts) { if (!cte || !cts) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: committed_through needs both cte and cts"); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS cas_ref_ckpt: committed frontier needs both committed_through_writer_epoch and committed_through_ref_sequence"); ckpt.committed_through = RefTxnId{*cte, *cts}; } if (lse || lss) { if (!lse || !lss) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: last_epoch_seal needs both lse and lss"); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS cas_ref_ckpt: last epoch seal needs both last_epoch_seal_writer_epoch and last_epoch_seal_ref_sequence"); ckpt.last_epoch_seal = RefTxnId{*lse, *lss}; } if (!body_in.eof() || !in.eof()) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp index be7ee5567575..c457f143ec3c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp @@ -30,7 +30,7 @@ std::string_view opKindToWord(RefOpKind k) case RefOpKind::RemoveNamespace: return "remove_namespace"; case RefOpKind::EpochSeal: return "epoch_seal"; } - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: unknown op kind {}", static_cast(k)); + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: unknown operation kind {}", static_cast(k)); } RefOpKind opKindFromWord(std::string_view w) @@ -40,7 +40,7 @@ RefOpKind opKindFromWord(std::string_view w) if (w == "set_published_at") return RefOpKind::SetPublishedAt; if (w == "remove_namespace") return RefOpKind::RemoveNamespace; if (w == "epoch_seal") return RefOpKind::EpochSeal; - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: unknown op kind '{}'", w); + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: unknown operation kind '{}'", w); } /// Byte budget over the encoded text. A removal-class transaction uses the larger complete-table @@ -73,9 +73,9 @@ void writeBindingFields(CasJsonWriter & out, bool & first, std::string_view pref { checkCanonicalRefName(b.ref_name, "RefLogTxn", "owner binding ref_name"); checkManifestRef(b.manifest_ref, "RefLogTxn", "owner binding manifest_ref"); - out.key(prefix, "bk", first); + out.key(prefix, "binding_kind", first); writeStringValue(out, refOwnerKindToWord(b.kind)); - out.key(prefix, "rn", first); + out.key(prefix, "ref_name", first); writeStringValue(out, b.ref_name); writeManifestRefFields(out, first, prefix, b.manifest_ref); } @@ -83,7 +83,7 @@ void writeBindingFields(CasJsonWriter & out, bool & first, std::string_view pref void writeOp(CasJsonWriter & out, const RefOp & op) { bool first = true; - writeKey(out, "op", first); + writeKey(out, "operation", first); writeStringValue(out, opKindToWord(op.kind)); switch (op.kind) { @@ -93,17 +93,17 @@ void writeOp(CasJsonWriter & out, const RefOp & op) break; case RefOpKind::OwnerTransition: if (op.old_binding) - writeBindingFields(out, first, "o", *op.old_binding); + writeBindingFields(out, first, "old_", *op.old_binding); if (op.new_binding) - writeBindingFields(out, first, "n", *op.new_binding); + writeBindingFields(out, first, "new_", *op.new_binding); break; case RefOpKind::SetPublishedAt: checkCanonicalRefName(op.ref_name, "RefLogTxn", "set_published_at ref_name"); checkManifestRef(op.expected_manifest_ref, "RefLogTxn", "set_published_at manifest_ref"); - writeKey(out, "rn", first); + writeKey(out, "ref_name", first); writeStringValue(out, op.ref_name); writeManifestRefFields(out, first, "", op.expected_manifest_ref); - writeKey(out, "ts", first); + writeKey(out, "published_at_ms", first); writeIntText(op.published_at_ms, out); break; } @@ -122,7 +122,8 @@ struct ManifestFields ManifestRef build(std::string_view what) const { if (!me || !mb || !mo) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} manifest_ref missing me/mb/mo", what); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "RefLogTxn: {} manifest_ref missing writer_epoch/build_sequence/manifest_ordinal", what); return manifestRefFromFields(*me, *mb, *mo, "RefLogTxn", what); } }; @@ -138,7 +139,7 @@ struct BindingFields RefOwnerBinding build(std::string_view what) const { if (!bk || !rn) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} binding missing bk/rn", what); + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} binding missing binding_kind/ref_name", what); RefOwnerBinding b; b.kind = refOwnerKindFromWord(*bk, "RefLogTxn owner binding"); b.ref_name = *rn; @@ -151,7 +152,8 @@ struct BindingFields /// The log transaction's header-object meta line (ns + txn_id + the optional `prev_epoch_seal` /// chain). Shared by `encodeRefLogTxn` and `removalFramingSize` so the two never disagree by a byte; /// `removalFramingSize` always passes `std::nullopt` -- a removal transaction is never a sequence-1 -/// epoch-transition record. Additive: the `"!pse"`/`"!pss"` pair is emitted only when +/// epoch-transition record. Additive: the `"!previous_seal_writer_epoch"`/ +/// `"!previous_seal_ref_sequence"` pair is emitted only when /// `prev_epoch_seal` is set, so a body without it is byte-identical to the pre-EpochSeal wire shape. /// `!`-prefixed: `prev_epoch_seal` is INV-2 chain evidence, not cosmetic metadata -- a decoder that /// doesn't understand it must refuse the object rather than silently drop the chain link while @@ -160,11 +162,11 @@ struct BindingFields void writeLogMeta(CasJsonWriter & out, const String & ns, const RefTxnId & txn_id, const std::optional & prev_epoch_seal) { bool first = true; - writeKey(out, "ns", first); + writeKey(out, "namespace", first); writeStringValue(out, ns); - writeRefTxnIdFields(out, first, "we", "rs", txn_id); + writeRefTxnIdFields(out, first, "writer_epoch", "ref_sequence", txn_id); if (prev_epoch_seal) - writeRefTxnIdFields(out, first, "!pse", "!pss", *prev_epoch_seal); + writeRefTxnIdFields(out, first, "!previous_seal_writer_epoch", "!previous_seal_ref_sequence", *prev_epoch_seal); closeObject(out, first); writeChar('\n', out); } @@ -185,29 +187,29 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) String key; while (r.nextKey(key)) { - if (key == "rn") sp_rn = r.readString(); - else if (key == "me") sp_mf.me = r.readU64String(); - else if (key == "mb") sp_mf.mb = r.readU64String(); - else if (key == "mo") sp_mf.mo = r.readU64Number(); - else if (key == "ts") sp_ts = r.readU64Number(); - else if (key == "obk") ob.bk = r.readString(); - else if (key == "orn") ob.rn = r.readString(); - else if (key == "ome") ob.mf.me = r.readU64String(); - else if (key == "omb") ob.mf.mb = r.readU64String(); - else if (key == "omo") ob.mf.mo = r.readU64Number(); - else if (key == "nbk") nb.bk = r.readString(); - else if (key == "nrn") nb.rn = r.readString(); - else if (key == "nme") nb.mf.me = r.readU64String(); - else if (key == "nmb") nb.mf.mb = r.readU64String(); - else if (key == "nmo") nb.mf.mo = r.readU64Number(); - else if (key == "pl") - /// `"pl"` (payload) was removed from the op wire in stage-1 T12 (the `set_payload` op became - /// `set_published_at`). The retired op WORD is already rejected by `opKindFromWord`, but this - /// generic reader reads field keys before switching on kind, so a `"pl"` field paired with a - /// still-recognized op word would otherwise be `skipUnknown`'d. It is a KNOWN-removed field, + if (key == "ref_name") sp_rn = r.readString(); + else if (key == "writer_epoch") sp_mf.me = r.readU64String(); + else if (key == "build_sequence") sp_mf.mb = r.readU64String(); + else if (key == "manifest_ordinal") sp_mf.mo = r.readU64Number(); + else if (key == "published_at_ms") sp_ts = r.readU64Number(); + else if (key == "old_binding_kind") ob.bk = r.readString(); + else if (key == "old_ref_name") ob.rn = r.readString(); + else if (key == "old_writer_epoch") ob.mf.me = r.readU64String(); + else if (key == "old_build_sequence") ob.mf.mb = r.readU64String(); + else if (key == "old_manifest_ordinal") ob.mf.mo = r.readU64Number(); + else if (key == "new_binding_kind") nb.bk = r.readString(); + else if (key == "new_ref_name") nb.rn = r.readString(); + else if (key == "new_writer_epoch") nb.mf.me = r.readU64String(); + else if (key == "new_build_sequence") nb.mf.mb = r.readU64String(); + else if (key == "new_manifest_ordinal") nb.mf.mo = r.readU64Number(); + else if (key == "payload") + /// `payload` was removed from the operation wire in stage-1 T12 (the `set_payload` operation became + /// `set_published_at`). The retired operation word is already rejected by `opKindFromWord`, but this + /// generic reader reads field keys before switching on kind, so a `"payload"` field paired with a + /// still-recognized operation word would otherwise be `skipUnknown`'d. It is a known removed field, /// not a genuinely-unknown one -- reject it explicitly rather than silently discard it. throw Exception(ErrorCodes::CORRUPTED_DATA, - "RefLogTxn: op record carries the removed \"pl\" (payload) field"); + "RefLogTxn: operation record carries the removed \"payload\" field"); else r.skipUnknown(key); } @@ -225,7 +227,7 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) break; case RefOpKind::SetPublishedAt: if (!sp_rn || !sp_ts) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: set_published_at missing rn/ts"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: set_published_at missing ref_name/published_at_ms"); op.ref_name = *sp_rn; checkCanonicalRefName(op.ref_name, "RefLogTxn", "set_published_at ref_name"); op.expected_manifest_ref = sp_mf.build("set_published_at manifest_ref"); @@ -332,21 +334,24 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con String key; while (r.nextKey(key)) { - if (key == "ns") { txn.ns = r.readString(); saw_ns = true; } - else if (key == "we") { txn.txn_id.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "rs") { txn.txn_id.ref_sequence = r.readU64String(); saw_rs = true; } - else if (key == "!pse") pse = r.readU64String(); - else if (key == "!pss") pss = r.readU64String(); + if (key == "namespace") { txn.ns = r.readString(); saw_ns = true; } + else if (key == "writer_epoch") { txn.txn_id.writer_epoch = r.readU64String(); saw_we = true; } + else if (key == "ref_sequence") { txn.txn_id.ref_sequence = r.readU64String(); saw_rs = true; } + else if (key == "!previous_seal_writer_epoch") pse = r.readU64String(); + else if (key == "!previous_seal_ref_sequence") pss = r.readU64String(); else r.skipUnknown(key); } if (!saw_ns || !saw_we || !saw_rs) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: meta line missing ns/we/rs"); - /// Both-or-neither: `nextKey` already rejects a repeated "!pse"/"!pss" (duplicate-key check), so - /// this only guards against a body carrying exactly one of the pair. + throw Exception(ErrorCodes::CORRUPTED_DATA, + "RefLogTxn: meta line missing namespace/writer_epoch/ref_sequence"); + /// Both-or-neither: `nextKey` already rejects a repeated `"!previous_seal_writer_epoch"` or + /// `"!previous_seal_ref_sequence"` (duplicate-key check), so this only guards against a body + /// carrying exactly one of the pair. if (pse || pss) { if (!pse || !pss) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: prev_epoch_seal needs both !pse and !pss"); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "RefLogTxn: previous epoch seal needs both !previous_seal_writer_epoch and !previous_seal_ref_sequence"); txn.prev_epoch_seal = RefTxnId{*pse, *pss}; } if (!m.eof()) @@ -373,7 +378,7 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: empty line"); - if (key == "n") + if (key == "record_count") { const uint64_t n = r.readU64Number(); while (r.nextKey(key)) @@ -385,12 +390,12 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con "RefLogTxn: trailer count {} != {} ops", n, txn.ops.size()); break; } - if (key != "op") - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: record must start with \"op\""); + if (key != "operation") + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: record must start with \"operation\""); const RefOpKind kind = opKindFromWord(r.readString()); txn.ops.push_back(readOpRecord(r, kind)); if (!l.eof()) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: junk after op record"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: junk after operation record"); } /// Finalization: `txn.ops` is now complete, so the context-free seal grammar (which needs the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h index 34347fb97aaf..74245fded4f6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h @@ -13,8 +13,8 @@ namespace DB::Cas /// Text codec for `cas_ref_log`, the immutable object stored at `_log/`. Each object contains /// exactly one committed transaction: its namespace, transaction id, and the batch of `RefOp`s applied -/// by that commit. The body has a header, a meta line `{"ns","we","rs",["!pse","!pss"]}`, one JSON -/// record per op, and a `{"n":count}` trailer. Records are emitted in the transaction's stored order +/// by that commit. The body has a header, a metadata line with namespace and transaction identity, one JSON +/// record per operation, and a `{"record_count":count}` trailer. Records are emitted in stored order /// and contain no codec-generated timestamps, so encoding the same value is byte-identical. This /// determinism is a property of the representation, not an adoption gate: ref commits use /// `putIfAbsentControlled`, and the caller applies the `Always`/`.zst` storage policy by sealing the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp index f31e9fed4ca2..ffc48def7c5a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp @@ -59,12 +59,12 @@ void writeCommittedRow(CasJsonWriter & out, const RefCommittedRow & row) checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "committed ref_name"); checkManifestRef(row.manifest_ref, "RefTableSnapshot", "committed"); bool first = true; - writeKey(out, "k", first); + writeKey(out, "kind", first); writeStringValue(out, "c"); - writeKey(out, "rn", first); + writeKey(out, "ref_name", first); writeStringValue(out, row.ref_name); writeManifestRefFields(out, first, "", row.manifest_ref); - writeKey(out, "ts", first); + writeKey(out, "published_at_ms", first); writeIntText(row.published_at_ms, out); closeObject(out, first); writeChar('\n', out); @@ -79,9 +79,9 @@ void writePrecommitRow(CasJsonWriter & out, const RefOwnerBinding & row) checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "precommit ref_name"); checkManifestRef(row.manifest_ref, "RefTableSnapshot", "precommit"); bool first = true; - writeKey(out, "k", first); + writeKey(out, "kind", first); writeStringValue(out, "p"); - writeKey(out, "rn", first); + writeKey(out, "ref_name", first); writeStringValue(out, row.ref_name); writeManifestRefFields(out, first, "", row.manifest_ref); closeObject(out, first); @@ -95,16 +95,16 @@ void writePrecommitRow(CasJsonWriter & out, const RefOwnerBinding & row) void writeSnapshotMeta(CasJsonWriter & out, const RefTableSnapshot & snapshot) { bool first = true; - writeKey(out, "ns", first); + writeKey(out, "namespace", first); writeStringValue(out, snapshot.ns); - writeRefTxnIdFields(out, first, "we", "rs", snapshot.snapshot_id); - writeKey(out, "lc", first); + writeRefTxnIdFields(out, first, "writer_epoch", "ref_sequence", snapshot.snapshot_id); + writeKey(out, "lifecycle", first); writeStringValue(out, "live"); closeObject(out, first); writeChar('\n', out); } -/// Collector for a ManifestRef's three flat fields (bare "me"/"mb"/"mo"). +/// Collector for a `ManifestRef`'s three flat fields. struct ManifestFields { std::optional me; @@ -117,7 +117,8 @@ struct ManifestFields ManifestRef build(std::string_view what) const { if (!me || !mb || !mo) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: {} manifest_ref missing me/mb/mo", what); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "RefTableSnapshot: {} manifest_ref missing writer_epoch/build_sequence/manifest_ordinal", what); return manifestRefFromFields(*me, *mb, *mo, "RefTableSnapshot", what); } }; @@ -166,10 +167,10 @@ RefTableSnapshot decodeRefTableSnapshot( String key; while (r.nextKey(key)) { - if (key == "ns") { snapshot.ns = r.readString(); saw_ns = true; } - else if (key == "we") { snapshot.snapshot_id.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "rs") { snapshot.snapshot_id.ref_sequence = r.readU64String(); saw_rs = true; } - else if (key == "lc") + if (key == "namespace") { snapshot.ns = r.readString(); saw_ns = true; } + else if (key == "writer_epoch") { snapshot.snapshot_id.writer_epoch = r.readU64String(); saw_we = true; } + else if (key == "ref_sequence") { snapshot.snapshot_id.ref_sequence = r.readU64String(); saw_rs = true; } + else if (key == "lifecycle") { const String lifecycle = r.readString(); if (lifecycle != "live") @@ -177,13 +178,14 @@ RefTableSnapshot decodeRefTableSnapshot( "RefTableSnapshot: lifecycle must be exactly 'live', got '{}'", lifecycle); saw_lc = true; } - else if (key == "rte" || key == "rts") + else if (key == "remove_txn_writer_epoch" || key == "remove_txn_ref_sequence") throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: meta carries retired terminal field '{}'", key); else r.skipUnknown(key); } if (!saw_ns || !saw_we || !saw_rs || !saw_lc) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: meta line missing ns/we/rs/lc"); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "RefTableSnapshot: meta line missing namespace/writer_epoch/ref_sequence/lifecycle"); if (!meta_buf.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: junk after meta line"); } @@ -198,7 +200,7 @@ RefTableSnapshot decodeRefTableSnapshot( if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: empty line"); - if (key == "n") + if (key == "record_count") { const uint64_t n = r.readU64Number(); while (r.nextKey(key)) @@ -210,8 +212,8 @@ RefTableSnapshot decodeRefTableSnapshot( "RefTableSnapshot: trailer count {} != {} rows", n, snapshot.committed.size() + snapshot.precommits.size()); break; } - if (key != "k") - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: record must start with \"k\""); + if (key != "kind") + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: record must start with \"kind\""); const String k = r.readString(); std::optional rn; @@ -219,17 +221,17 @@ RefTableSnapshot decodeRefTableSnapshot( std::optional ts; while (r.nextKey(key)) { - if (key == "rn") rn = r.readString(); - else if (key == "me") mf.me = r.readU64String(); - else if (key == "mb") mf.mb = r.readU64String(); - else if (key == "mo") mf.mo = r.readU64Number(); - else if (key == "ts") ts = r.readU64Number(); - else if (key == "pl") - /// `"pl"` (payload) was removed from the row wire in stage-1 T12. It is a KNOWN-removed + if (key == "ref_name") rn = r.readString(); + else if (key == "writer_epoch") mf.me = r.readU64String(); + else if (key == "build_sequence") mf.mb = r.readU64String(); + else if (key == "manifest_ordinal") mf.mo = r.readU64Number(); + else if (key == "published_at_ms") ts = r.readU64Number(); + else if (key == "payload") + /// `payload` was removed from the row wire in stage-1 T12. It is a known removed /// field, not a genuinely-unknown future one the tolerant reader may skip -- silently /// discarding a persisted payload would lose data -- so reject it explicitly. throw Exception(ErrorCodes::CORRUPTED_DATA, - "RefTableSnapshot: record carries the removed \"pl\" (payload) field"); + "RefTableSnapshot: record carries the removed \"payload\" field"); else r.skipUnknown(key); } if (!l.eof()) @@ -238,7 +240,8 @@ RefTableSnapshot decodeRefTableSnapshot( if (k == "c") { if (!rn || !ts) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: committed row missing rn/ts"); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "RefTableSnapshot: committed row missing ref_name/published_at_ms"); RefCommittedRow row; row.ref_name = *rn; checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "committed ref_name"); @@ -249,7 +252,7 @@ RefTableSnapshot decodeRefTableSnapshot( else if (k == "p") { if (!rn) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: precommit row missing rn"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: precommit row missing ref_name"); RefOwnerBinding row; row.kind = RefOwnerKind::Precommit; row.ref_name = *rn; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h index 07972a0d47ab..9a1ea12b31e9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h @@ -14,7 +14,7 @@ namespace DB::Cas /// Text codec for `cas_ref_snap`, the complete per-namespace ref table snapshot at /// `_snap/`. The object is read whole rather than streamed and belongs to the Control /// family: callers store the encoded text as an Always/`.zst` object. Its canonical text consists of -/// a header, a metadata line, committed and precommit row lines, and a `{"n":count}` trailer. +/// a header, a metadata line, committed and precommit row lines, and a `{"record_count":count}` trailer. /// /// There is no such thing as a "seal snapshot". An epoch is closed IN-BAND, by an `EpochSeal` /// transaction the recovery CAS-walk places at `{E, T+1}` in the `_log` stream (INV-2) -- the exact key diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp index f523553271b4..f93c3a46f3d2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp @@ -32,11 +32,11 @@ String encodeOwner(const OwnerObject & o) CasJsonWriter out(256); writeHeaderLine(out, FormatId::Owner); bool first = true; - writeKey(out, "su", first); + writeKey(out, "server_uuid", first); writeHex128Value(out, o.server_uuid); if (o.retired_at_ms) { - writeKey(out, "rt", first); + writeKey(out, "retired_at_ms", first); writeIntText(*o.retired_at_ms, out); } closeObject(out, first); @@ -58,19 +58,19 @@ OwnerObject decodeOwner(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "su") + if (key == "server_uuid") { o.server_uuid = r.readHex128(); saw = true; } - else if (key == "rt") + else if (key == "retired_at_ms") rt = r.readU64Number(); else r.skipUnknown(key); } o.retired_at_ms = rt; if (!saw) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS owner: missing su"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS owner: missing server_uuid"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS owner: trailing bytes"); return o; @@ -81,7 +81,7 @@ String encodeServerEpoch(const ServerEpoch & e) CasJsonWriter out(256); writeHeaderLine(out, FormatId::ServerEpoch); bool first = true; - writeKey(out, "nwe", first); + writeKey(out, "next_writer_epoch", first); writeU64StringValue(out, e.next_writer_epoch); closeObject(out, first); writeChar('\n', out); @@ -101,7 +101,7 @@ ServerEpoch decodeServerEpoch(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "nwe") + if (key == "next_writer_epoch") { e.next_writer_epoch = r.readU64String(); saw = true; @@ -110,7 +110,7 @@ ServerEpoch decodeServerEpoch(std::string_view data) r.skipUnknown(key); } if (!saw) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-epoch: missing nwe"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-epoch: missing next_writer_epoch"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-epoch: trailing bytes"); return e; @@ -121,15 +121,15 @@ String encodeMountLease(const MountLease & m) CasJsonWriter out(256); writeHeaderLine(out, FormatId::MountLease); bool first = true; - writeKey(out, "su", first); writeHex128Value(out, m.server_uuid); - writeKey(out, "we", first); writeU64StringValue(out, m.writer_epoch); - writeKey(out, "hn", first); writeStringValue(out, m.hostname); - writeKey(out, "pid", first); writeIntText(m.pid, out); - writeKey(out, "sat", first); writeIntText(m.started_at_ms, out); - writeKey(out, "seq", first); writeU64StringValue(out, m.seq); - writeKey(out, "eat", first); writeIntText(m.expires_at_ms, out); - writeKey(out, "ma", first); writeU64StringValue(out, m.min_active); - writeKey(out, "fen", first); writeBoolValue(out, m.gc_fenced); + writeKey(out, "server_uuid", first); writeHex128Value(out, m.server_uuid); + writeKey(out, "writer_epoch", first); writeU64StringValue(out, m.writer_epoch); + writeKey(out, "hostname", first); writeStringValue(out, m.hostname); + writeKey(out, "process_id", first); writeIntText(m.pid, out); + writeKey(out, "started_at_ms", first); writeIntText(m.started_at_ms, out); + writeKey(out, "sequence", first); writeU64StringValue(out, m.seq); + writeKey(out, "expires_at_ms", first); writeIntText(m.expires_at_ms, out); + writeKey(out, "min_active", first); writeU64StringValue(out, m.min_active); + writeKey(out, "gc_fenced", first); writeBoolValue(out, m.gc_fenced); writeKey(out, "write_attempt_id", first); writeHex128Value(out, m.write_attempt_id); closeObject(out, first); writeChar('\n', out); @@ -151,23 +151,23 @@ MountLease decodeMountLease(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "su") + if (key == "server_uuid") { m.server_uuid = r.readHex128(); saw_su = true; } - else if (key == "we") + else if (key == "writer_epoch") { m.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "hn") m.hostname = r.readString(); - else if (key == "pid") m.pid = r.readU64Number(); - else if (key == "sat") m.started_at_ms = r.readU64Number(); - else if (key == "seq") m.seq = r.readU64String(); - else if (key == "eat") m.expires_at_ms = r.readU64Number(); - else if (key == "ma") m.min_active = r.readU64String(); - else if (key == "fen") m.gc_fenced = r.readBool(); + else if (key == "hostname") m.hostname = r.readString(); + else if (key == "process_id") m.pid = r.readU64Number(); + else if (key == "started_at_ms") m.started_at_ms = r.readU64Number(); + else if (key == "sequence") m.seq = r.readU64String(); + else if (key == "expires_at_ms") m.expires_at_ms = r.readU64Number(); + else if (key == "min_active") m.min_active = r.readU64String(); + else if (key == "gc_fenced") m.gc_fenced = r.readBool(); else if (key == "write_attempt_id") { m.write_attempt_id = r.readHex128(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp index 9814d5d14811..ed9b1deb918a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp @@ -263,7 +263,7 @@ void writeHeaderLine(CasJsonWriter & out, FormatId id) bool first = true; writeKey(out, "type", first); writeStringValue(out, t.type); - writeKey(out, "v", first); + writeKey(out, "version", first); writeIntText(currentCompatibilityVersion(), out); closeObject(out, first); writeChar('\n', out); @@ -272,7 +272,7 @@ void writeHeaderLine(CasJsonWriter & out, FormatId id) void writeTrailerLine(CasJsonWriter & out, uint64_t n) { bool first = true; - writeKey(out, "n", first); + writeKey(out, "record_count", first); writeIntText(n, out); closeObject(out, first); writeChar('\n', out); @@ -306,8 +306,8 @@ TextHeader parseHeaderObject(std::string_view line, std::string_view what) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: header line must start with \"type\"", what); TextHeader h; h.type = r.readString(); - if (!r.nextKey(key) || key != "v") - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: header line must carry \"v\" second", what); + if (!r.nextKey(key) || key != "version") + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: header line must carry \"version\" second", what); h.v = r.readU32Number(); while (r.nextKey(key)) r.skipUnknown(key); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h index 50edd33bd4ba..3745e3995bdb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h @@ -13,7 +13,7 @@ namespace DB::Cas { /// Shared container mechanics for versioned content-addressed text objects: a header line -/// {"type":"cas_","v":N}, a body, an optional trailer line, and, for formats whose registry +/// {"type":"cas_","version":N}, a body, an optional trailer line, and, for formats whose registry /// policy requires it, one zstd frame around the whole object. This is the only code that knows /// that container shape; per-object codecs add only key mappings and object-specific invariants. /// @@ -59,7 +59,7 @@ class CasJsonWriter append("\":"); } - /// Same, for the prefixed key vocabulary ("o"/"n" + "me"/"mb"/"mo"/"bk"/"rn") — the + /// Same, for prefixed keys such as `old_writer_epoch` and `new_binding_kind` — the /// prefix and name are appended back to back, no composed temporary. void key(std::string_view prefix, std::string_view name, bool & first) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp index 31d44f260121..6de796c81a9a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp @@ -58,17 +58,17 @@ ObjectKind objectKindFromWord(std::string_view w, std::string_view what) void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t) { - writeKey(out, "tt", first); + writeKey(out, "token_type", first); writeStringValue(out, tokenTypeToWord(t.type)); - writeKey(out, "tv", first); + writeKey(out, "token_value", first); writeStringValue(out, t.value); } void writeBlobRefFields(CasJsonWriter & out, bool & first, const BlobRef & r) { - writeKey(out, "ha", first); + writeKey(out, "hash_algorithm", first); writeStringValue(out, blobHashAlgoName(r.algo)); - writeKey(out, "h", first); + writeKey(out, "hash", first); writeStringValue(out, codecFor(r.algo).toHex(r.digest)); } @@ -76,11 +76,11 @@ void writeManifestRefFields(CasJsonWriter & out, bool & first, std::string_view { /// Unlike the WriteBuffer overload, the two-part key() form appends the prefix and name back /// to back with no composed String(prefix) + "..." temporary. - out.key(prefix, "me", first); + out.key(prefix, "writer_epoch", first); out.u64StringValue(r.writer_epoch); - out.key(prefix, "mb", first); + out.key(prefix, "build_sequence", first); out.u64StringValue(r.build_sequence); - out.key(prefix, "mo", first); + out.key(prefix, "manifest_ordinal", first); out.u64Number(r.manifest_ordinal); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md index 7c4c99aa36fd..3d46aff1fda1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md @@ -1,7 +1,7 @@ # CAS persisted formats — the living registry -Every persisted CAS object is a text file: header line `{"type":"cas_","v":N}`, body -(one JSON object / sorted NDJSON records / raw payload zone), optional `{"n":…}` trailer. +Every persisted CAS object is a text file: header line `{"type":"cas_","version":N}`, body +(one JSON object / sorted NDJSON records / raw payload zone), optional `{"record_count":…}` trailer. Can-grow-large types are stored under a **`.zst` key suffix** and are ALWAYS one zstd frame (checksum on; declared content size checked against the cap before allocation); always-small and deterministic types are raw. `CasTextFormat.{h,cpp}` is the only code that knows this shape. @@ -9,7 +9,7 @@ deterministic types are raw. `CasTextFormat.{h,cpp}` is the only code that knows The object inventory is text end to end — there are no binary CAS formats and no protobuf dependency. The GC source-edge data plane (`cas_run`) is sorted NDJSON written and read as a stream (no seek); its integrity check is the whole-file seal checksum. The part manifest is the -one `PayloadHybrid` object: text header + descriptor meta + sorted entry records + `{"n":…}` +one `PayloadHybrid` object: text header + descriptor meta + sorted entry records + `{"record_count":…}` trailer, followed by a banner-framed raw payload zone for inline file bytes. **Rule:** any change to a persisted format lands in the SAME commit as its row here. @@ -37,22 +37,22 @@ trailer, followed by a banner-framed raw payload zone for inline file bytes. ## Codec table Authoritative per-format traits (type string, family, strictness, compression policy, caps) live -in `CasFormat.cpp` (`TRAITS`), asserted complete by `gtest_cas_text_format.cpp`. Key naming: keys -2–5 chars; fixed-width `UInt128` identities = 32-char lowercase hex strings; blob digests = +in `CasFormat.cpp` (`TRAITS`), asserted complete by `gtest_cas_text_format.cpp`. Key naming uses +full descriptive names; fixed-width `UInt128` identities = 32-char lowercase hex strings; blob digests = algo-width hex (two chars per digest byte), rendered with their algo name (`sha256:ab12…`) wherever a bare hex would be ambiguous; unbounded u64 = decimal strings; bounded counts/lengths/ms-timestamps = numbers; units documented here per object as codecs land. ## Evolution rules (one screen) -- `v` (header line) is the ONLY version field; reader gate: `v > G_BUILD` → +- `version` (header line) is the ONLY version field; reader gate: `version > G_BUILD` → `UNKNOWN_FORMAT_VERSION`, checked before the body. -- Additive change = new tolerant key, no `v` bump; on MUTABLE objects the field is best-effort +- Additive change = new tolerant key, no `version` bump; on MUTABLE objects the field is best-effort until the pool floor rises (an old writer's fresh re-encode drops it). -- Breaking change = `v` bump + `changePoints` + write-down-to-floor; the floor raise is what - fences old builds out (mount gates: `min_reader_generation` forward, pool-meta `v` backward). +- Breaking change = `version` bump + `changePoints` + write-down-to-floor; the floor raise is what + fences old builds out (mount gates: `min_reader_generation` forward, pool-meta `version` backward). - Deterministic formats (`cas_fold_seal`, `cas_run`): strict keys, pinned raw, and the adoption - pin — on a `putDeterministicArtifact` conflict, re-encode at the `v` of the EXISTING object. + pin — on a `putDeterministicArtifact` conflict, re-encode at the `version` of the EXISTING object. - A key prefixed `!` is critical: a reader that does not understand it fails closed. - Padding zones (blob header pad, manifest banners) are deterministic and verified — no unaccounted bytes in any object. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.h index 0c16098f18ea..8fc080dc32a9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.h @@ -99,7 +99,7 @@ CondemnedRow decodeCondemnedRow(std::string_view payload); class SourceEdgeRunView { public: - /// false once the run's `{"n"}` trailer is consumed (the trailer count is verified there). `key` is + /// false once the run's `{"record_count"}` trailer is consumed (the trailer count is verified there). `key` is /// the reconstructed `SourceEdgeKeyCodec::key(ref, source_id)`; `payload` is the original marker byte /// or `encodeCondemnedRow` bytes. bool next(String & key, String & payload); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index 1ea5c2d567cb..ddae15d8b80f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -1605,8 +1605,9 @@ namespace /// a tolerant, read-only peek at the /// `cas_ref_log` TEXT object (codecs-v3 phase 3) WITHOUT `decodeRefLogTxn`'s expected-value cross-check /// -- the whole point of this diagnostic is that the body is NOT expected to match this key's identity. -/// It `openObject`s the stored `.zst`, skips the header line, and reads `ns`/`we`/`rs` off the meta -/// line (`we`/`rs` are decimal u64 strings). Never validates the header `v`, never reads past the meta +/// It `openObject`s the stored `.zst`, skips the header line, and reads `namespace`/`writer_epoch`/ +/// `ref_sequence` off the meta line (the ids are decimal u64 strings). Never validates the header +/// `version`, never reads past the meta /// line (the ops are irrelevant to identifying the writer), and swallows any truncation/garbage: this /// is a background diagnostic only, never a decode anything else depends on. struct ForeignRefLogHeaderPeek @@ -1634,9 +1635,9 @@ std::optional peekForeignRefLogHeader(const String & by String key; while (r.nextKey(key)) { - if (key == "ns") { peek.ns = r.readString(); saw_ns = true; } - else if (key == "we") { peek.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "rs") { peek.ref_sequence = r.readU64String(); saw_rs = true; } + if (key == "namespace") { peek.ns = r.readString(); saw_ns = true; } + else if (key == "writer_epoch") { peek.writer_epoch = r.readU64String(); saw_we = true; } + else if (key == "ref_sequence") { peek.ref_sequence = r.readU64String(); saw_rs = true; } else r.skipUnknown(key); } if (!saw_ns || !saw_we || !saw_rs) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index 69eba5061fd3..c450b6e4b461 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -55,7 +55,7 @@ struct PoolConfig /// (clean relative path); `ServerUUID`/`server_id` is demoted to an /// owner token. Validated via `Cas::validateServerRootId`. String server_root_id; - uint64_t blob_header_len = 256; /// creation-time only; the pool is authoritative on reopen + uint64_t blob_header_len = 384; /// creation-time only; the pool is authoritative on reopen /// CAS mixed-algo pools: the /// NODE-LOCAL algo this Pool writes NEW content with (`Pool::writeAlgo()`). NOT durable pool /// state -- two live nodes may intentionally write with different (already-admitted) algos, so diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp index 1428be282f30..d0f0dc646498 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp @@ -284,9 +284,9 @@ String renderMountLease(const MountLease & m) .add("server_uuid", jsonHex(m.server_uuid)) .add("writer_epoch", jsonUInt(m.writer_epoch)) .add("hostname", jsonEscape(m.hostname)) - .add("pid", jsonUInt(m.pid)) + .add("process_id", jsonUInt(m.pid)) .add("started_at_ms", jsonUInt(m.started_at_ms)) - .add("seq", jsonUInt(m.seq)) + .add("sequence", jsonUInt(m.seq)) .add("expires_at_ms", jsonUInt(m.expires_at_ms)) .add("min_active", jsonUInt(m.min_active)) .add("gc_fenced", jsonBool(m.gc_fenced)) diff --git a/src/Disks/tests/cas_format_test_battery.h b/src/Disks/tests/cas_format_test_battery.h index 173bfcc5c4df..2cef4b1dd636 100644 --- a/src/Disks/tests/cas_format_test_battery.h +++ b/src/Disks/tests/cas_format_test_battery.h @@ -33,7 +33,7 @@ struct FormatBatteryCase /// self-derived expectation. inline String currentFormatHeader(std::string_view type) { - return fmt::format("{{\"type\":\"{}\",\"v\":{}}}\n", type, DB::Cas::currentCompatibilityVersion()); + return fmt::format("{{\"type\":\"{}\",\"version\":{}}}\n", type, DB::Cas::currentCompatibilityVersion()); } namespace cas_battery_detail @@ -85,8 +85,8 @@ inline void runFormatBattery(const FormatBatteryCase & c) [&] { c.decode(text.substr(0, i)); }, fmt::format("{}: cut inside header at {}", t.type, i)); /// v+1 gate. - const String v_now = fmt::format("\"v\":{}", currentCompatibilityVersion()); - const String v_next = fmt::format("\"v\":{}", currentCompatibilityVersion() + 1); + const String v_now = fmt::format("\"version\":{}", currentCompatibilityVersion()); + const String v_next = fmt::format("\"version\":{}", currentCompatibilityVersion() + 1); String future; if (c.make_future_version) future = c.make_future_version(text); diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index e7b5f6221c18..419d9ddbd9f7 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -749,7 +749,7 @@ inline void seedPoolMetaForRestart( { const DB::Cas::Layout layout(pool_prefix); DB::Cas::PoolMeta::createOrValidate( - backend, layout, /*blob_header_len=*/256, gc_shards, + backend, layout, /*blob_header_len=*/384, gc_shards, DB::Cas::BlobHashAlgo::CityHash128, /*allow_new=*/false, /*allow_mint=*/true); if (!backend.get(layout.refCatalogKey())) DB::Cas::CasRefCatalog::initializeEmptyForNewPool(backend, layout); @@ -761,7 +761,7 @@ inline void seedPoolMetaForRestart( /// are arbitrary (GC never reads them); the hash is what the manifest entry references. inline void writeBlobBody( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::UInt128 & hash, - uint64_t blob_header_len = 256) + uint64_t blob_header_len = 384) { DB::Cas::EnvelopeHeader header; header.kind = DB::Cas::ObjectKind::Blob; diff --git a/src/Disks/tests/gtest_cas_blob_digest.cpp b/src/Disks/tests/gtest_cas_blob_digest.cpp index 9a87f08ccda2..0da2ba6043ef 100644 --- a/src/Disks/tests/gtest_cas_blob_digest.cpp +++ b/src/Disks/tests/gtest_cas_blob_digest.cpp @@ -73,7 +73,7 @@ TEST(CASBlobDigest, ShardOfViaPoolMetaConstructedCodecMatchesOldBlobShard) { auto backend = std::make_shared(); const Layout layout("p"); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 512, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); ASSERT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::CityHash128)})); const DigestCodec codec = codecFor(BlobHashAlgo::CityHash128); @@ -227,26 +227,26 @@ TEST(CASBlobDigest, PoolMetaRecordsCreatingAlgoAndWidthDerivesFromIt) { auto backend = std::make_shared(); const Layout layout("p1"); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::CityHash128)})); EXPECT_EQ(blobHashLenFor(BlobHashAlgo::CityHash128), 16u); } { auto backend = std::make_shared(); const Layout layout("p2"); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::XXH3_128, /*allow_new*/ false, /*allow_mint*/ true); + const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::XXH3_128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::XXH3_128)})); EXPECT_EQ(blobHashLenFor(BlobHashAlgo::XXH3_128), 16u); } { auto backend = std::make_shared(); const Layout layout("p3"); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256, /*allow_new*/ false, /*allow_mint*/ true); + const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::Sha256, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::Sha256)})); EXPECT_EQ(blobHashLenFor(BlobHashAlgo::Sha256), 32u); /// Reopen (decode path) must re-derive the same recorded algo. - const PoolMeta reopened = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256); + const PoolMeta reopened = PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::Sha256); EXPECT_EQ(reopened.algos_used, (std::vector{static_cast(BlobHashAlgo::Sha256)})); } } diff --git a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp index 1119a5be0a64..68dacebcca43 100644 --- a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp @@ -20,15 +20,15 @@ EnvelopeHeader sampleHeader(const String & ref) h.intended_ref = ref; return h; } -constexpr uint32_t L = 256; +constexpr uint32_t L = 512; /// The envelope has a fixed physical length. At generation 9 there is no unsupported one-digit /// version, so replacing `9` with `10` must consume one byte from the space pad rather than silently -/// turning the 256-byte fixture into a different wire shape. +/// turning the 512-byte fixture into a different wire shape. String blobEnvelopeWithFutureVersion(std::string_view text) { - const String v_now = fmt::format("\"v\":{}", currentCompatibilityVersion()); - const String v_next = fmt::format("\"v\":{}", currentCompatibilityVersion() + 1); + const String v_now = fmt::format("\"version\":{}", currentCompatibilityVersion()); + const String v_next = fmt::format("\"version\":{}", currentCompatibilityVersion() + 1); String future(text); const size_t version_pos = future.find(v_now); if (version_pos == String::npos || v_next.size() < v_now.size()) @@ -51,11 +51,11 @@ TEST(CASBlobEnvelopeFormat, FixedLengthAndPadZone) const String head = encodeEnvelopeHeader(h, L); ASSERT_EQ(head.size(), L); /// exactly blob_header_len EXPECT_EQ(head[L - 1], '\n'); /// terminator at byte 255 - const String json = fmt::format(R"({{"type":"cas_blob","v":{},)", currentCompatibilityVersion()) + - "\"tag\":\"0102030405060708090a0b0c0d0e0f10\"," - "\"bld\":\"1112131415161718191a1b1c1d1e1f20\",\"ts\":1752537600123," - "\"by\":\"2122232425262728292a2b2c2d2e2f30\",\"op\":\"merge\",\"ch\":26006001," - "\"ref\":\"t-abc/all_1_2_0\"}"; + const String json = fmt::format(R"({{"type":"cas_blob","version":{},)", currentCompatibilityVersion()) + + "\"incarnation_tag\":\"0102030405060708090a0b0c0d0e0f10\"," + "\"build_id\":\"1112131415161718191a1b1c1d1e1f20\",\"created_at_ms\":1752537600123," + "\"creator_server_id\":\"2122232425262728292a2b2c2d2e2f30\",\"operation\":\"merge\",\"clickhouse_version\":26006001," + "\"intended_ref\":\"t-abc/all_1_2_0\"}"; ASSERT_LT(json.size(), L); EXPECT_EQ(head.substr(0, json.size()), json); /// '/' UNescaped (local escaper) EXPECT_EQ(head.substr(json.size(), (L - 1) - json.size()), String((L - 1) - json.size(), ' ')); /// pad = spaces @@ -75,8 +75,8 @@ TEST(CASBlobEnvelopeFormat, FixedLengthAndPadZone) TEST(CASBlobEnvelopeFormat, RefTruncatedToExactBudget) { - /// A 200-char ref cannot fit; it is truncated so the header is EXACTLY 256 bytes and the pad holds. - EnvelopeHeader h = sampleHeader(String(200, 'a')); + /// A 600-char ref cannot fit; it is truncated so the header is EXACTLY 512 bytes and the pad holds. + EnvelopeHeader h = sampleHeader(String(600, 'a')); const String head = encodeEnvelopeHeader(h, L); ASSERT_EQ(head.size(), L); EXPECT_EQ(head[L - 1], '\n'); @@ -117,7 +117,7 @@ TEST(CASBlobEnvelopeFormat, GatesAndCriticalKey) String wrong_type = head; wrong_type.replace(wrong_type.find("cas_blob"), 8, "cas_xxxx"); EXPECT_THROW(decodeEnvelopeHeader(wrong_type, wrong_type.size(), ObjectKind::Blob), DB::Exception); - const String current_version = fmt::format("\"v\":{}", currentCompatibilityVersion()); + const String current_version = fmt::format("\"version\":{}", currentCompatibilityVersion()); String future = blobEnvelopeWithFutureVersion(head); cas_battery_detail::expectCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, [&] { decodeEnvelopeHeader(future, future.size(), ObjectKind::Blob); }, "future blob-envelope version"); @@ -125,7 +125,7 @@ TEST(CASBlobEnvelopeFormat, GatesAndCriticalKey) String out_of_range = head; const size_t out_of_range_version_at = out_of_range.find(current_version); ASSERT_NE(out_of_range_version_at, String::npos); - out_of_range.replace(out_of_range_version_at, current_version.size(), "\"v\":4294967299"); + out_of_range.replace(out_of_range_version_at, current_version.size(), "\"version\":4294967299"); try { decodeEnvelopeHeader(out_of_range, out_of_range.size(), ObjectKind::Blob); @@ -146,11 +146,11 @@ TEST(CASBlobEnvelopeFormat, RefEscaperAlphabetPinned) { /// Pins the LOCAL escaper's alphabet (§ref-escaper): " and \ escape, control chars -> \uXXXX, /// '/' passes VERBATIM. Goes RED if anyone "unifies" this with writeStringValue/FormatSettings — - /// the 256-byte budget arithmetic depends on this alphabet being codec-owned and frozen. + /// the 512-byte budget arithmetic depends on this alphabet being codec-owned and frozen. EnvelopeHeader h = sampleHeader(String("a/b\"c\\d") + '\x01' + "e"); const String head = encodeEnvelopeHeader(h, L); const String expected_ref_json = R"("a/b\"c\\d\u0001e")"; - EXPECT_NE(head.find("\"ref\":" + expected_ref_json), String::npos) + EXPECT_NE(head.find("\"intended_ref\":" + expected_ref_json), String::npos) << "escaper alphabet drifted: '/' must be verbatim, quote/backslash escaped, control -> \\uXXXX"; } @@ -159,11 +159,11 @@ TEST(CASFormatBattery, BlobEnvelope) /// The golden is CONSTRUCTED from the hand-pinned json literal (same one FixedLengthAndPadZone /// asserts) + the derived pad — NOT self-computed via encodeEnvelopeHeader, which would compare /// the encoder to itself and pin nothing. - const String json = fmt::format(R"({{"type":"cas_blob","v":{},)", currentCompatibilityVersion()) + - "\"tag\":\"0102030405060708090a0b0c0d0e0f10\"," - "\"bld\":\"1112131415161718191a1b1c1d1e1f20\",\"ts\":1752537600123," - "\"by\":\"2122232425262728292a2b2c2d2e2f30\",\"op\":\"merge\",\"ch\":26006001," - "\"ref\":\"t-abc/all_1_2_0\"}"; + const String json = fmt::format(R"({{"type":"cas_blob","version":{},)", currentCompatibilityVersion()) + + "\"incarnation_tag\":\"0102030405060708090a0b0c0d0e0f10\"," + "\"build_id\":\"1112131415161718191a1b1c1d1e1f20\",\"created_at_ms\":1752537600123," + "\"creator_server_id\":\"2122232425262728292a2b2c2d2e2f30\",\"operation\":\"merge\",\"clickhouse_version\":26006001," + "\"intended_ref\":\"t-abc/all_1_2_0\"}"; const String golden = json + String((L - 1) - json.size(), ' ') + '\n'; runFormatBattery(FormatBatteryCase{ .id = FormatId::Blob, diff --git a/src/Disks/tests/gtest_cas_blob_meta_format.cpp b/src/Disks/tests/gtest_cas_blob_meta_format.cpp index bb055d2cef71..0bdfce8e1335 100644 --- a/src/Disks/tests/gtest_cas_blob_meta_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_meta_format.cpp @@ -16,8 +16,8 @@ TEST(CASFormatBattery, BlobMeta) .id = FormatId::BlobMeta, .encode = [&] { return sealObject(FormatId::BlobMeta, encodeBlobMeta(m)); }, .decode = [](std::string_view s) { decodeBlobMeta(std::string(openObject(FormatId::BlobMeta, s))); }, - .golden = "{\"type\":\"cas_blob_meta\",\"v\":10}\n" - "{\"st\":\"clean\",\"cr\":\"0\",\"sz\":\"12345\"}\n"}); + .golden = "{\"type\":\"cas_blob_meta\",\"version\":10}\n" + "{\"state\":\"clean\",\"condemn_round\":\"0\",\"size\":\"12345\"}\n"}); } TEST(CASBlobMetaFormat, CondemnedRoundTripAllFields) @@ -31,7 +31,7 @@ TEST(CASBlobMetaFormat, CondemnedRoundTripAllFields) EXPECT_EQ(back.condemn_round, 7u); EXPECT_EQ(back.size, 4096u); EXPECT_EQ(encodeBlobMeta(m), - "{\"type\":\"cas_blob_meta\",\"v\":10}\n{\"st\":\"condemned\",\"cr\":\"7\",\"sz\":\"4096\"}\n"); + "{\"type\":\"cas_blob_meta\",\"version\":10}\n{\"state\":\"condemned\",\"condemn_round\":\"7\",\"size\":\"4096\"}\n"); } TEST(CASBlobMetaFormat, FailsClosedOnUnknownStateAndTruncation) @@ -39,11 +39,11 @@ TEST(CASBlobMetaFormat, FailsClosedOnUnknownStateAndTruncation) /// Unknown state word -> CORRUPTED_DATA (mirrors the old `state > Condemned` reject). /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes /// the header gate, which is the point — the BODY is what has to fail here. - const String bad_state = "{\"type\":\"cas_blob_meta\",\"v\":3}\n{\"st\":\"zombie\",\"cr\":\"0\",\"sz\":\"0\"}\n"; + const String bad_state = "{\"type\":\"cas_blob_meta\",\"version\":3}\n{\"state\":\"zombie\",\"condemn_round\":\"0\",\"size\":\"0\"}\n"; EXPECT_THROW(decodeBlobMeta(bad_state), DB::Exception); /// Missing state key -> CORRUPTED_DATA. - const String no_state = "{\"type\":\"cas_blob_meta\",\"v\":3}\n{\"cr\":\"0\",\"sz\":\"0\"}\n"; + const String no_state = "{\"type\":\"cas_blob_meta\",\"version\":3}\n{\"condemn_round\":\"0\",\"size\":\"0\"}\n"; EXPECT_THROW(decodeBlobMeta(no_state), DB::Exception); /// Truncated (header only) -> CORRUPTED_DATA. - EXPECT_THROW(decodeBlobMeta("{\"type\":\"cas_blob_meta\",\"v\":3}\n"), DB::Exception); + EXPECT_THROW(decodeBlobMeta("{\"type\":\"cas_blob_meta\",\"version\":3}\n"), DB::Exception); } diff --git a/src/Disks/tests/gtest_cas_encoding_pins.cpp b/src/Disks/tests/gtest_cas_encoding_pins.cpp index f4c3a913fda7..dba028e80bcc 100644 --- a/src/Disks/tests/gtest_cas_encoding_pins.cpp +++ b/src/Disks/tests/gtest_cas_encoding_pins.cpp @@ -9,11 +9,9 @@ using namespace DB; using namespace DB::Cas; -/// These literals pin the CANONICAL BYTES of the CAS text encoders as of the commit that -/// introduced this file. The CasJsonWriter migration (2026-07-20 spec) must keep every one of -/// them green UNMODIFIED: canonical text is byte-compared on retries and deterministic adoption, -/// and the incremental ref budget counters assume these exact sizes. Never edit an expected -/// string here to make a test pass — that means the encoder's bytes drifted, which is the bug. +/// These literals pin the CANONICAL BYTES of the CAS text encoders. Canonical text is byte-compared +/// on retries and deterministic adoption, and the incremental ref budget counters assume these +/// exact sizes. Update an expected string only for an intentional format change. TEST(CASEncodingPins, RefLogTxnAllOpKinds) { @@ -48,16 +46,16 @@ TEST(CASEncodingPins, RefLogTxnAllOpKinds) removal.kind = RefOpKind::RemoveNamespace; txn.ops.push_back(removal); - const String expected = fmt::format("{{\"type\":\"cas_ref_log\",\"v\":{}}}\n", currentCompatibilityVersion()) + - "{\"ns\":\"roots/pin\",\"we\":\"7\",\"rs\":\"9\"}\n" - "{\"op\":\"namespace_birth\"}\n" - "{\"op\":\"owner_transition\",\"obk\":\"precommit\",\"orn\":\"20260101_0_1_1_1\"," - "\"ome\":\"1\",\"omb\":\"2\",\"omo\":3,\"nbk\":\"committed\",\"nrn\":\"20260101_0_1_1_1\"," - "\"nme\":\"1\",\"nmb\":\"2\",\"nmo\":3}\n" - "{\"op\":\"set_published_at\",\"rn\":\"20260101_0_1_1_1\\\"c\\nd\\u0001e\\u2028f\"," - "\"me\":\"1\",\"mb\":\"2\",\"mo\":3,\"ts\":1234}\n" - "{\"op\":\"remove_namespace\"}\n" - "{\"n\":4}\n"; + const String expected = fmt::format("{{\"type\":\"cas_ref_log\",\"version\":{}}}\n", currentCompatibilityVersion()) + + "{\"namespace\":\"roots/pin\",\"writer_epoch\":\"7\",\"ref_sequence\":\"9\"}\n" + "{\"operation\":\"namespace_birth\"}\n" + "{\"operation\":\"owner_transition\",\"old_binding_kind\":\"precommit\",\"old_ref_name\":\"20260101_0_1_1_1\"," + "\"old_writer_epoch\":\"1\",\"old_build_sequence\":\"2\",\"old_manifest_ordinal\":3,\"new_binding_kind\":\"committed\",\"new_ref_name\":\"20260101_0_1_1_1\"," + "\"new_writer_epoch\":\"1\",\"new_build_sequence\":\"2\",\"new_manifest_ordinal\":3}\n" + "{\"operation\":\"set_published_at\",\"ref_name\":\"20260101_0_1_1_1\\\"c\\nd\\u0001e\\u2028f\"," + "\"writer_epoch\":\"1\",\"build_sequence\":\"2\",\"manifest_ordinal\":3,\"published_at_ms\":1234}\n" + "{\"operation\":\"remove_namespace\"}\n" + "{\"record_count\":4}\n"; EXPECT_EQ(encodeRefLogTxn(txn), expected); } @@ -75,11 +73,11 @@ TEST(CASEncodingPins, RefSnapshotLive) snap.precommits.push_back(RefOwnerBinding{RefOwnerKind::Precommit, "20260102_0_2_2_2", ManifestRef{4, 5, 6}}); - const String expected = fmt::format("{{\"type\":\"cas_ref_snap\",\"v\":{}}}\n", currentCompatibilityVersion()) + - "{\"ns\":\"roots/pin\",\"we\":\"7\",\"rs\":\"9\",\"lc\":\"live\"}\n" - "{\"k\":\"c\",\"rn\":\"20260101_0_1_1_1\",\"me\":\"1\",\"mb\":\"2\",\"mo\":3,\"ts\":5}\n" - "{\"k\":\"p\",\"rn\":\"20260102_0_2_2_2\",\"me\":\"4\",\"mb\":\"5\",\"mo\":6}\n" - "{\"n\":2}\n"; + const String expected = fmt::format("{{\"type\":\"cas_ref_snap\",\"version\":{}}}\n", currentCompatibilityVersion()) + + "{\"namespace\":\"roots/pin\",\"writer_epoch\":\"7\",\"ref_sequence\":\"9\",\"lifecycle\":\"live\"}\n" + "{\"kind\":\"c\",\"ref_name\":\"20260101_0_1_1_1\",\"writer_epoch\":\"1\",\"build_sequence\":\"2\",\"manifest_ordinal\":3,\"published_at_ms\":5}\n" + "{\"kind\":\"p\",\"ref_name\":\"20260102_0_2_2_2\",\"writer_epoch\":\"4\",\"build_sequence\":\"5\",\"manifest_ordinal\":6}\n" + "{\"record_count\":2}\n"; EXPECT_EQ(encodeRefTableSnapshot(snap), expected); } @@ -97,13 +95,13 @@ TEST(CASEncodingPins, SourceEdgeRunLines) writer.finish(); out.finalize(); - /// The exact "b" rendering (algo byte + digest hex) is pinned as a whole line; the point is + /// The exact `blob` rendering (algo byte + digest hex) is pinned as a whole line; the point is /// that Task 8's line-scratch rewrite must reproduce it byte-for-byte. const String text = out.str(); - const String header = fmt::format("{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n", currentCompatibilityVersion()); + const String header = fmt::format("{{\"type\":\"cas_run\",\"version\":{},\"kind\":\"source_edge\"}}\n", currentCompatibilityVersion()); const String expected_record = - "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n"; - const String trailer = "{\"n\":1}\n"; + "{\"blob_ref\":\"0100000000000000000000000000000002\",\"source_id\":\"00000000000000000000000000000005\",\"marker\":\"edge\"}\n"; + const String trailer = "{\"record_count\":1}\n"; /// There is exactly one record, so the whole buffer must be byte-identical to header + record + trailer. const String expected_full = header + expected_record + trailer; EXPECT_EQ(text, expected_full) << text; diff --git a/src/Disks/tests/gtest_cas_envelope.cpp b/src/Disks/tests/gtest_cas_envelope.cpp index 7dac7ddf6eed..d908096839b9 100644 --- a/src/Disks/tests/gtest_cas_envelope.cpp +++ b/src/Disks/tests/gtest_cas_envelope.cpp @@ -6,7 +6,7 @@ using namespace DB; using namespace DB::Cas; -/// The v3 blob-envelope shape (256-byte JSON header + payload). Full round-trip / gate / pad-zone / +/// The v3 blob-envelope shape (512-byte JSON header + payload). Full round-trip / gate / pad-zone / /// budget / critical-key coverage lives in gtest_cas_blob_envelope_format.cpp; these two keep the /// cases that file does not exercise: a header with NO provenance/ref, and the incarnation-zone /// independence of the payload. @@ -18,14 +18,14 @@ TEST(CASEnvelope, BlobRoundTripNoExtensions) h.kind = ObjectKind::Blob; h.incarnation_tag = 0x22; h.build_id = 0x33; - const std::string obj = encodeEnvelopeHeader(h, 256) + payload; + const std::string obj = encodeEnvelopeHeader(h, 512) + payload; const EnvelopeHeader d = decodeEnvelopeHeader(obj, obj.size(), ObjectKind::Blob); EXPECT_EQ(d.kind, ObjectKind::Blob); EXPECT_EQ(d.compatibility_version, G_BUILD); EXPECT_FALSE(d.provenance.has_value()); /// none set -> the ts/by/op/ch keys are absent EXPECT_FALSE(d.intended_ref.has_value()); /// none set -> the ref key is omitted - EXPECT_EQ(d.header_len, 256u); + EXPECT_EQ(d.header_len, 512u); /// payload starts right after the fixed-length header. EXPECT_EQ(obj.substr(payloadOffset(d)), payload); } @@ -44,8 +44,8 @@ TEST(CASEnvelope, IncarnationZoneDoesNotAffectPayload) b.incarnation_tag = 0xCCCC; b.build_id = 0xDDDD; - const std::string ha = encodeEnvelopeHeader(a, 256); - const std::string hb = encodeEnvelopeHeader(b, 256); + const std::string ha = encodeEnvelopeHeader(a, 512); + const std::string hb = encodeEnvelopeHeader(b, 512); EXPECT_NE(ha, hb); /// headers differ (incarnation zone) const EnvelopeHeader da = decodeEnvelopeHeader(ha + payload, ha.size() + payload.size(), ObjectKind::Blob); diff --git a/src/Disks/tests/gtest_cas_fold_seal_format.cpp b/src/Disks/tests/gtest_cas_fold_seal_format.cpp index afe6331a4ed8..6c6f41274166 100644 --- a/src/Disks/tests/gtest_cas_fold_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_fold_seal_format.cpp @@ -42,11 +42,11 @@ TEST(CASFormatBattery, FoldSeal) [&] { return sealObject(FormatId::FoldSeal, encodeFoldSeal(seal)); }, [](std::string_view s) { decodeFoldSeal(std::string(openObject(FormatId::FoldSeal, s))); }, currentFormatHeader("cas_fold_seal") + - "{\"g\":\"5\",\"pg\":\"4\"}\n" - "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":2,\"lfe\":\"7\",\"lfs\":\"11\"}\n" - "{\"k\":\"btr\",\"key\":\"r0\",\"ck\":\"0000000000000000000000000000000f\",\"shard\":0,\"gen\":\"5\"}\n" - "{\"k\":\"cnd\",\"shard\":0,\"ct\":3,\"pt\":1,\"ocr\":\"4\"}\n" - "{\"n\":3}\n"}); + "{\"generation\":\"5\",\"parent_generation\":\"4\"}\n" + "{\"kind\":\"rfl\",\"life_id\":\"00000000000000000000000000000001\",\"classification\":2,\"last_folded_writer_epoch\":\"7\",\"last_folded_ref_sequence\":\"11\"}\n" + "{\"kind\":\"btr\",\"key\":\"r0\",\"checksum\":\"0000000000000000000000000000000f\",\"shard\":0,\"generation\":\"5\"}\n" + "{\"kind\":\"cnd\",\"shard\":0,\"condemned_total\":3,\"pending_total\":1,\"oldest_nonpending_condemn_round\":\"4\"}\n" + "{\"record_count\":3}\n"}); } TEST(CASFoldSealFormat, RoundTripsAllFields) @@ -123,11 +123,11 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRequiresEveryBlobTargetAndSummaryFiel for (const std::string_view field : { R"(,"key":"p/gc/gen/7/attempt/1/blob_target/0/0")", - R"(,"ck":"00000000000000000000000000000001")", - R"(,"gen":"7")", - ",\"ct\":0", - ",\"pt\":0", - R"(,"ocr":"18446744073709551615")"}) + R"(,"checksum":"00000000000000000000000000000001")", + R"(,"generation":"7")", + ",\"condemned_total\":0", + ",\"pending_total\":0", + R"(,"oldest_nonpending_condemn_round":"18446744073709551615")"}) { String malformed = valid; eraseRequiredField(malformed, field); @@ -291,11 +291,11 @@ TEST(CASFoldSealFormat, UnifiedRefLifeRowRoundTripsCoverageHoldAndCleanupEvidenc .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{9, 10}}}); const String expected = currentFormatHeader("cas_fold_seal") + - "{\"g\":\"8\",\"pg\":\"7\"}\n" - "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000001234\",\"cls\":4," - "\"lfe\":\"3\",\"lfs\":\"4\",\"hr\":\"manifest_body_missing\",\"hpe\":\"5\"," - "\"hps\":\"6\",\"hrc\":7,\"hnr\":\"8\",\"rte\":\"9\",\"rts\":\"10\"}\n" - "{\"n\":1}\n"; + "{\"generation\":\"8\",\"parent_generation\":\"7\"}\n" + "{\"kind\":\"rfl\",\"life_id\":\"00000000000000000000000000001234\",\"classification\":4," + "\"last_folded_writer_epoch\":\"3\",\"last_folded_ref_sequence\":\"4\",\"hold_reason\":\"manifest_body_missing\",\"hold_position_writer_epoch\":\"5\"," + "\"hold_position_ref_sequence\":\"6\",\"hold_retry_count\":7,\"hold_next_retry_round\":\"8\",\"remove_txn_writer_epoch\":\"9\",\"remove_txn_ref_sequence\":\"10\"}\n" + "{\"record_count\":1}\n"; EXPECT_EQ(encodeFoldSeal(seal), expected); EXPECT_EQ(decodeFoldSeal(expected), seal); @@ -306,10 +306,10 @@ TEST(CASFoldSealFormat, UnifiedRefLifeRowRoundTripsCoverageHoldAndCleanupEvidenc TEST(CASFoldSealFormat, UnifiedCodecRejectsLegacyCoverageRecord) { const String old = - "{\"type\":\"cas_fold_seal\",\"v\":7}\n" - "{\"g\":\"8\",\"pg\":\"7\"}\n" - "{\"k\":\"cov\",\"key\":\"name/0\",\"cls\":2,\"lfe\":\"3\",\"lfs\":\"4\"}\n" - "{\"n\":1}\n"; + "{\"type\":\"cas_fold_seal\",\"version\":7}\n" + "{\"generation\":\"8\",\"parent_generation\":\"7\"}\n" + "{\"kind\":\"cov\",\"key\":\"name/0\",\"classification\":2,\"last_folded_writer_epoch\":\"3\",\"last_folded_ref_sequence\":\"4\"}\n" + "{\"record_count\":1}\n"; cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(old); }, "legacy coverage"); } @@ -318,10 +318,10 @@ TEST(CASFoldSealFormat, UnifiedCodecRejectsLegacyCoverageRecord) TEST(CASFoldSealFormat, UnifiedCodecRejectsLegacyNamespaceCleanupRecord) { const String old = - "{\"type\":\"cas_fold_seal\",\"v\":7}\n" - "{\"g\":\"8\",\"pg\":\"7\"}\n" - "{\"k\":\"nsc\",\"ns\":\"name\",\"rte\":\"3\",\"rts\":\"4\",\"st\":\"completed\"}\n" - "{\"n\":1}\n"; + "{\"type\":\"cas_fold_seal\",\"version\":7}\n" + "{\"generation\":\"8\",\"parent_generation\":\"7\"}\n" + "{\"kind\":\"nsc\",\"namespace\":\"name\",\"remove_txn_writer_epoch\":\"3\",\"remove_txn_ref_sequence\":\"4\",\"status\":\"completed\"}\n" + "{\"record_count\":1}\n"; cas_battery_detail::expectCode( DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(old); }, "legacy namespace cleanup"); } diff --git a/src/Disks/tests/gtest_cas_format.cpp b/src/Disks/tests/gtest_cas_format.cpp index 9aa93e99f76f..ae4a61eb7792 100644 --- a/src/Disks/tests/gtest_cas_format.cpp +++ b/src/Disks/tests/gtest_cas_format.cpp @@ -78,15 +78,15 @@ TEST(CASPoolMeta, GenerationNinePoolIsRejectedAtReaderFloor) { PoolMeta meta; meta.pool_id = UInt128{1}; - meta.blob_header_len = 256; + meta.blob_header_len = 512; meta.gc_shards = 1; meta.min_reader_generation = 10; meta.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; String encoded = encodePoolMeta(meta); - const String current = "\"v\":10"; + const String current = "\"version\":10"; const size_t version = encoded.find(current); ASSERT_NE(version, String::npos); - encoded.replace(version, current.size(), "\"v\":9"); + encoded.replace(version, current.size(), "\"version\":9"); try { diff --git a/src/Disks/tests/gtest_cas_format_battery.cpp b/src/Disks/tests/gtest_cas_format_battery.cpp index f6ba9bcdbcd0..fedcc200e334 100644 --- a/src/Disks/tests/gtest_cas_format_battery.cpp +++ b/src/Disks/tests/gtest_cas_format_battery.cpp @@ -11,7 +11,7 @@ TEST(CASFormatBattery, PoolMeta) { PoolMeta pm; pm.pool_id = hexToU128("00112233445566778899aabbccddeeff"); - pm.blob_header_len = 256; + pm.blob_header_len = 512; pm.min_reader_generation = 3; pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; runFormatBattery(FormatBatteryCase{ @@ -19,5 +19,5 @@ TEST(CASFormatBattery, PoolMeta) .encode = [&] { return sealObject(FormatId::PoolMeta, encodePoolMeta(pm)); }, .decode = [](std::string_view s) { decodePoolMeta(std::string(openObject(FormatId::PoolMeta, s))); }, .golden = currentFormatHeader("cas_pool_meta") + - "{\"pid\":\"00112233445566778899aabbccddeeff\",\"hln\":256,\"gcs\":1,\"mrg\":3,\"alg\":\"ch128\"}\n"}); + "{\"pool_id\":\"00112233445566778899aabbccddeeff\",\"blob_header_len\":512,\"gc_shards\":1,\"min_reader_generation\":3,\"algorithms_used\":\"ch128\"}\n"}); } diff --git a/src/Disks/tests/gtest_cas_fsck.cpp b/src/Disks/tests/gtest_cas_fsck.cpp index 71abb917eaa2..41935191ed7a 100644 --- a/src/Disks/tests/gtest_cas_fsck.cpp +++ b/src/Disks/tests/gtest_cas_fsck.cpp @@ -811,10 +811,10 @@ TEST(CASFsckAuthority, MissingBurnedEpochSealIsChainBroken) /// intermediate epoch is reported rather than treated as a sparse legal transition. String skipped_bytes = encodeRefLogTxn(RefLogTxn{ .ns = ns.string(), .txn_id = RefTxnId{7, 1}, .ops = {}, .prev_epoch_seal = RefTxnId{6, 1}}); - const String old_epoch_token = R"("!pse":"6")"; + const String old_epoch_token = R"("!previous_seal_writer_epoch":"6")"; const auto old_epoch = skipped_bytes.find(old_epoch_token); ASSERT_NE(old_epoch, String::npos); - skipped_bytes.replace(old_epoch, old_epoch_token.size(), R"("!pse":"1")"); + skipped_bytes.replace(old_epoch, old_epoch_token.size(), R"("!previous_seal_writer_epoch":"1")"); ASSERT_EQ(backend->putIfAbsent(layout.refLogKey(life, RefTxnId{7, 1}), sealObject(FormatId::RefLog, skipped_bytes)).outcome, PutOutcome::Done); diff --git a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp index 13dc97ab826c..b186d0b5694e 100644 --- a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp +++ b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp @@ -1132,9 +1132,9 @@ TEST(CASGCFrontierGate, AMalformedCatalogNeverDecodesIntoAnEmptyProof) const String type_needle = fmt::format("\"type\":\"{}\"", traitsFor(FormatId::RefCatalog).type); ASSERT_NE(empty_base.find(type_needle), String::npos); - const String version_needle = fmt::format("\"v\":{}", currentCompatibilityVersion()); + const String version_needle = fmt::format("\"version\":{}", currentCompatibilityVersion()); ASSERT_NE(empty_base.find(version_needle), String::npos); - ASSERT_NE(base.find("\"n\":1"), String::npos); + ASSERT_NE(base.find("\"record_count\":1"), String::npos); const auto replaceOnce = [](const String & haystack, const String & needle, const String & replacement) -> String { @@ -1150,10 +1150,10 @@ TEST(CASGCFrontierGate, AMalformedCatalogNeverDecodesIntoAnEmptyProof) {"wrong-type", replaceOnce(empty_base, type_needle, "\"type\":\"cas_ref_ckpt\"")}, /// The one version case the CURRENT (unmodified) gate actually enforces: a version ABOVE /// `G_BUILD` is refused by `checkCompatibility` before decode proceeds. - {"future-version", replaceOnce(empty_base, version_needle, "\"v\":999999")}, - {"trailer-count-mismatch", replaceOnce(base, "\"n\":1", "\"n\":2")}, + {"future-version", replaceOnce(empty_base, version_needle, "\"version\":999999")}, + {"trailer-count-mismatch", replaceOnce(base, "\"record_count\":1", "\"record_count\":2")}, /// The trailer line entirely gone: decode's post-entry loop expects another line and hits EOF. - {"missing-trailer", base.substr(0, base.rfind("{\"n\":1}\n"))}, + {"missing-trailer", base.substr(0, base.rfind("{\"record_count\":1}\n"))}, /// The trailer present but its own line has no terminator: EOF strictly inside a line. {"truncated-mid-line", base.substr(0, base.size() - 2)}, }; diff --git a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp index 19a63bc6842a..d1195054390d 100644 --- a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp +++ b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp @@ -268,7 +268,7 @@ String sealTextWith(const String & prototype, const std::vector & record String text = headerAndMetaOf(prototype); for (const String & record : records) text += record + "\n"; - return text + "{\"n\":" + std::to_string(records.size()) + "}\n"; + return text + "{\"record_count\":" + std::to_string(records.size()) + "}\n"; } /// Replace the coverage row's `cls` value with `raw`, VERBATIM. The point is to write integers no @@ -277,9 +277,9 @@ String sealTextWith(const String & prototype, const std::vector & record /// the value always ends at a comma. String withRawClassification(const String & encoded, std::string_view raw) { - const size_t at = encoded.find("\"cls\":"); + const size_t at = encoded.find("\"classification\":"); EXPECT_NE(at, String::npos); - const size_t begin = at + strlen("\"cls\":"); + const size_t begin = at + strlen("\"classification\":"); const size_t end = encoded.find(',', begin); EXPECT_NE(end, String::npos); return encoded.substr(0, begin) + String{raw} + encoded.substr(end); @@ -437,7 +437,7 @@ TEST(CASGCHoldGrammar, AHoldOnAnyOtherClassificationIsRefusedByTheDecoder) .retry_count = 0, .next_retry_round = 1}; fixtureCoverage(seal, "ns/0") = cov; String text = encodeFoldSeal(seal); - const size_t at = text.find("\"cls\":4"); + const size_t at = text.find("\"classification\":4"); ASSERT_NE(at, String::npos); text[at + 6] = '2'; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(text); }); @@ -456,8 +456,8 @@ TEST(CASGCHoldGrammar, ClassificationFourWithoutAHoldIsRefusedByTheDecoder) .retry_count = 3, .next_retry_round = 4}; fixtureCoverage(seal, "ns/0") = cov; const String whole = encodeFoldSeal(seal); - for (const String & field : {String(R"("hr":"body_undecodable")"), String(R"("hpe":"1")"), - String(R"("hps":"2")"), String(R"("hrc":3)"), String(R"("hnr":"4")")}) + for (const String & field : {String(R"("hold_reason":"body_undecodable")"), String(R"("hold_position_writer_epoch":"1")"), + String(R"("hold_position_ref_sequence":"2")"), String(R"("hold_retry_count":3)"), String(R"("hold_next_retry_round":"4")")}) { SCOPED_TRACE("without " + field); const size_t at = whole.find(field); @@ -479,12 +479,12 @@ TEST(CASGCHoldGrammar, DuplicateHoldKeyIsCorruptedData) fixtureCoverage(seal, "ns/0") = cov; const String whole = encodeFoldSeal(seal); - const String field = R"("hr":"gap_below_witness")"; + const String field = R"("hold_reason":"gap_below_witness")"; const size_t at = whole.find(field); ASSERT_NE(at, String::npos); /// The same key twice, with a DIFFERENT value: last-wins would silently rewrite the reason. String doubled = whole; - doubled.insert(at, R"("hr":"witness_disappeared",)"); + doubled.insert(at, R"("hold_reason":"witness_disappeared",)"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(doubled); }); } @@ -548,10 +548,10 @@ TEST(CASGCHoldGrammar, AClassificationOutsideTheGrammarIsCorruptedData) TEST(CASGCHoldGrammar, ACoverageRowWithoutAClassificationIsCorruptedData) { const String clean = encodeFoldSeal(cleanSeal("ns/0")); - const size_t at = clean.find("\"cls\":2,"); + const size_t at = clean.find("\"classification\":2,"); ASSERT_NE(at, String::npos); String without = clean; - without.erase(at, strlen("\"cls\":2,")); + without.erase(at, strlen("\"classification\":2,")); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(without); }); } @@ -567,13 +567,13 @@ TEST(CASGCHoldGrammar, AHoldWhoseOffendingPositionHasAZeroComponentIsCorruptedDa expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - decodeFoldSeal(withField(withField(held, R"("hpe":"4")", R"("hpe":"0")"), - R"("hps":"6")", R"("hps":"0")")); + decodeFoldSeal(withField(withField(held, R"("hold_position_writer_epoch":"4")", R"("hold_position_writer_epoch":"0")"), + R"("hold_position_ref_sequence":"6")", R"("hold_position_ref_sequence":"0")")); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(held, R"("hpe":"4")", R"("hpe":"0")")); }); + [&] { decodeFoldSeal(withField(held, R"("hold_position_writer_epoch":"4")", R"("hold_position_writer_epoch":"0")")); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(held, R"("hps":"6")", R"("hps":"0")")); }); + [&] { decodeFoldSeal(withField(held, R"("hold_position_ref_sequence":"6")", R"("hold_position_ref_sequence":"0")")); }); } /// (3) The duplicate row. Two `cov` records for the same (namespace, shard) — held first, clean second — @@ -647,12 +647,12 @@ TEST(CASGCHoldGrammar, CleanupEvidenceWithAZeroRemovalIdIsCorruptedData) const String encoded = encodeFoldSeal(seal); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(encoded, R"("rte":"2")", R"("rte":"0")")); }); + [&] { decodeFoldSeal(withField(encoded, R"("remove_txn_writer_epoch":"2")", R"("remove_txn_writer_epoch":"0")")); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(encoded, R"("rts":"3")", R"("rts":"0")")); }); + [&] { decodeFoldSeal(withField(encoded, R"("remove_txn_ref_sequence":"3")", R"("remove_txn_ref_sequence":"0")")); }); /// Omitted entirely is the same thing: the fields default to zero. expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(encoded, R"("rte":"2",)", "")); }); + [&] { decodeFoldSeal(withField(encoded, R"("remove_txn_writer_epoch":"2",)", "")); }); } /// The OBJECT cap bounds the whole seal. Nothing on the fold-seal READ path enforces it (the seal @@ -1398,7 +1398,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWithAnUndecodablePriorSeal) const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); const String seal_key = layout.foldSealKey(st.snap_generation, st.snap_attempt); - backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":4}\nthis is not a seal body\n", + backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"version\":4}\nthis is not a seal body\n", backend->head(seal_key).token); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { gc.rebuildBaseline(/*force=*/true); }); @@ -1470,7 +1470,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenTheNewestSealIsUnreadableAndTheStateIsL const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); const String seal_key = layout.foldSealKey(st.snap_generation, st.snap_attempt); - backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":4}\nthis is not a seal body\n", + backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"version\":4}\nthis is not a seal body\n", backend->head(seal_key).token); const HeadResult sh = backend->head(layout.gcStateKey()); ASSERT_EQ(backend->deleteExact(layout.gcStateKey(), sh.token).kind, DeleteOutcome::Kind::Deleted); diff --git a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp index 158040be38c9..2f38d4189a4b 100644 --- a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp @@ -48,7 +48,7 @@ TEST(CASGCMaintenanceStateFormat, RegistryLayoutAndCanonicalCodec) const GcMaintenanceState empty; EXPECT_EQ(encodeGcMaintenanceState(empty), fmt::format( - "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"cur\":\"\"}}\n", currentCompatibilityVersion())); + "{{\"type\":\"cas_gc_maintenance_state\",\"version\":{}}}\n{{\"janitor_cursor\":\"\"}}\n", currentCompatibilityVersion())); const GcMaintenanceState state{.janitor_cursor = R"(cas/ns/a/"quoted"\\next)"}; EXPECT_EQ(decodeGcMaintenanceState(encodeGcMaintenanceState(state)), state); } @@ -57,28 +57,28 @@ TEST(CASGCMaintenanceStateFormat, RejectsMalformedAndBoundsCursor) { const auto bad = [](std::string_view body) { - return "{\"type\":\"cas_gc_maintenance_state\",\"v\":7}\n" + String(body); + return "{\"type\":\"cas_gc_maintenance_state\",\"version\":7}\n" + String(body); }; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeGcMaintenanceState(bad("{}\n")); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeGcMaintenanceState(bad("{\"cur\":\"a\",\"cur\":\"b\"}\n")); }); + [&] { (void)decodeGcMaintenanceState(bad("{\"janitor_cursor\":\"a\",\"janitor_cursor\":\"b\"}\n")); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeGcMaintenanceState(bad("{\"cur\":\"a\",\"extra\":1}\n")); }); + [&] { (void)decodeGcMaintenanceState(bad("{\"janitor_cursor\":\"a\",\"extra\":1}\n")); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeGcMaintenanceState(bad("{\"cur\":\"a\"}\nx")); }); + [&] { (void)decodeGcMaintenanceState(bad("{\"janitor_cursor\":\"a\"}\nx")); }); const GcMaintenanceState at_limit{.janitor_cursor = String(kMaxGcMaintenanceCursorBytes, 'x')}; EXPECT_EQ(decodeGcMaintenanceState(encodeGcMaintenanceState(at_limit)), at_limit); const GcMaintenanceState over_limit{.janitor_cursor = String(kMaxGcMaintenanceCursorBytes + 1, 'x')}; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LIMIT_EXCEEDED, [&] { (void)encodeGcMaintenanceState(over_limit); }); - const String raw = "{\"type\":\"cas_gc_maintenance_state\",\"v\":7}\n{\"cur\":\"" + over_limit.janitor_cursor + "\"}\n"; + const String raw = "{\"type\":\"cas_gc_maintenance_state\",\"version\":7}\n{\"janitor_cursor\":\"" + over_limit.janitor_cursor + "\"}\n"; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeGcMaintenanceState(raw); }); - String oversized = R"({"type":"cas_gc_maintenance_state","v":7,"pad":")"; + String oversized = R"({"type":"cas_gc_maintenance_state","version":7,"pad":")"; oversized.append(448 * 1024, 'x'); - oversized += "\"}\n{\"cur\":\""; + oversized += "\"}\n{\"janitor_cursor\":\""; oversized.append(kMaxGcMaintenanceCursorBytes, 'y'); oversized += "\"}\n"; ASSERT_GT(oversized.size(), traitsFor(FormatId::GcMaintenanceState).object_cap); @@ -172,7 +172,7 @@ TEST(CASGCMaintenanceState, FutureVersionPropagatesInsteadOfResetting) const Layout layout("p"); const String key = layout.gcMaintenanceStateKey(); ASSERT_EQ(backend.putIfAbsent(key, fmt::format( - "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"cur\":\"\"}}\n", currentCompatibilityVersion() + 1)).outcome, + "{{\"type\":\"cas_gc_maintenance_state\",\"version\":{}}}\n{{\"janitor_cursor\":\"\"}}\n", currentCompatibilityVersion() + 1)).outcome, PutOutcome::Done); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, [&] { (void)readGcMaintenanceState(backend, layout); }); diff --git a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp index e6530f131299..a2733d982eb4 100644 --- a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp @@ -40,8 +40,8 @@ TEST(CASFormatBattery, GcOutcomes) [&] { return sealObject(FormatId::GcOutcomes, encodeOutcomeLog(log)); }, [](std::string_view d) { decodeOutcomeLog(std::string(openObject(FormatId::GcOutcomes, d))); }, currentFormatHeader("cas_gc_outcomes") + - "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddeeff\"," - "\"tt\":\"etag\",\"tv\":\"e-1\",\"oc\":\"deleted\"}\n{\"n\":1}\n"}); + "{\"kind\":\"blob\",\"hash_algorithm\":\"ch128\",\"hash\":\"00112233445566778899aabbccddeeff\"," + "\"token_type\":\"etag\",\"token_value\":\"e-1\",\"outcome\":\"deleted\"}\n{\"record_count\":1}\n"}); } TEST(CASGCOutcomesFormat, EmptyRoundTrips) @@ -80,12 +80,12 @@ TEST(CASGCOutcomesFormat, GarbageAndUnknownWordsFailClosed) EXPECT_THROW(decodeOutcomeLog(String("")), DB::Exception); EXPECT_THROW(decodeOutcomeLog(String("not a cas object\n")), DB::Exception); /// A record with an unknown outcome word fails closed. - const String bad = "{\"type\":\"cas_gc_outcomes\",\"v\":3}\n" - "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddeeff\"," - "\"tt\":\"etag\",\"tv\":\"x\",\"oc\":\"bogus\"}\n{\"n\":1}\n"; + const String bad = "{\"type\":\"cas_gc_outcomes\",\"version\":3}\n" + "{\"kind\":\"blob\",\"hash_algorithm\":\"ch128\",\"hash\":\"00112233445566778899aabbccddeeff\"," + "\"token_type\":\"etag\",\"token_value\":\"x\",\"outcome\":\"bogus\"}\n{\"record_count\":1}\n"; EXPECT_THROW(decodeOutcomeLog(bad), DB::Exception); /// A trailer count mismatch fails closed. - const String miscount = "{\"type\":\"cas_gc_outcomes\",\"v\":3}\n{\"n\":5}\n"; + const String miscount = "{\"type\":\"cas_gc_outcomes\",\"version\":3}\n{\"record_count\":5}\n"; EXPECT_THROW(decodeOutcomeLog(miscount), DB::Exception); } @@ -94,8 +94,8 @@ TEST(CASGCOutcomesFormat, DigestWidthMismatchFailsClosedWithCorruptedData) /// `ch128` (CityHash128) digests are 16 bytes = 32 hex chars; here the "h" field is truncated /// to 30 hex chars. Must surface as CORRUPTED_DATA (malformed serialized input), not /// `fromHex`'s BAD_ARGUMENTS. - const String bad = "{\"type\":\"cas_gc_outcomes\",\"v\":3}\n" - "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddee\"," - "\"tt\":\"etag\",\"tv\":\"x\",\"oc\":\"deleted\"}\n{\"n\":1}\n"; + const String bad = "{\"type\":\"cas_gc_outcomes\",\"version\":3}\n" + "{\"kind\":\"blob\",\"hash_algorithm\":\"ch128\",\"hash\":\"00112233445566778899aabbccddee\"," + "\"token_type\":\"etag\",\"token_value\":\"x\",\"outcome\":\"deleted\"}\n{\"record_count\":1}\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeOutcomeLog(bad); }); } diff --git a/src/Disks/tests/gtest_cas_gc_state_format.cpp b/src/Disks/tests/gtest_cas_gc_state_format.cpp index aa661429f813..c6b13fba9860 100644 --- a/src/Disks/tests/gtest_cas_gc_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_state_format.cpp @@ -24,8 +24,8 @@ TEST(CASFormatBattery, GcState) [&] { return sealObject(FormatId::GcState, encodeGcState(s)); }, [](std::string_view d) { decodeGcState(std::string(openObject(FormatId::GcState, d))); }, currentFormatHeader("cas_gc_state") + - "{\"rnd\":\"4\",\"gcs\":1,\"sg\":\"9\",\"spt\":\"7\",\"sa\":\"3\",\"msc\":\"\"," - "\"lo\":\"00000000000000000000000000000001\",\"ls\":\"12\"}\n"}); + "{\"round\":\"4\",\"gc_shards\":1,\"snapshot_generation\":\"9\",\"snapshot_pruned_through\":\"7\",\"snapshot_attempt\":\"3\",\"manifest_sweep_cursor\":\"\"," + "\"lease_owner\":\"00000000000000000000000000000001\",\"lease_sequence\":\"12\"}\n"}); } TEST(CASFormatBattery, GcHeartbeat) @@ -35,7 +35,7 @@ TEST(CASFormatBattery, GcHeartbeat) [&] { return sealObject(FormatId::GcHeartbeat, encodeGcHeartbeat(hb)); }, [](std::string_view d) { decodeGcHeartbeat(std::string(openObject(FormatId::GcHeartbeat, d))); }, currentFormatHeader("cas_gc_hb") + - "{\"by\":\"00000000000000000000000000000001\",\"seq\":\"1741\"}\n"}); + "{\"owner\":\"00000000000000000000000000000001\",\"heartbeat_sequence\":\"1741\"}\n"}); } /// ---------- field round-trips (migrated from gtest_cas_gc_formats.cpp, re-pointed at the text codec) ---------- @@ -85,9 +85,9 @@ TEST(CASGCStateFormat, RejectsZeroGcShards) { /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes /// the header gate, which is the point — the BODY is what has to fail here. - const String bad = "{\"type\":\"cas_gc_state\",\"v\":3}\n" - "{\"rnd\":\"0\",\"gcs\":0,\"sg\":\"0\",\"spt\":\"0\",\"sa\":\"0\",\"msc\":\"\"," - "\"lo\":\"00000000000000000000000000000000\",\"ls\":\"0\"}\n"; + const String bad = "{\"type\":\"cas_gc_state\",\"version\":3}\n" + "{\"round\":\"0\",\"gc_shards\":0,\"snapshot_generation\":\"0\",\"snapshot_pruned_through\":\"0\",\"snapshot_attempt\":\"0\",\"manifest_sweep_cursor\":\"\"," + "\"lease_owner\":\"00000000000000000000000000000000\",\"lease_sequence\":\"0\"}\n"; EXPECT_THROW(decodeGcState(bad), DB::Exception); } @@ -127,9 +127,9 @@ TEST(CASGCStateFormat, RejectsAbsentGcShards) /// to the struct's gc_shards = 1 — a missing shard count means a corrupt object, not "use the floor". /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes /// the header gate, which is the point — the BODY is what has to fail here. - const String bad = "{\"type\":\"cas_gc_state\",\"v\":3}\n" - "{\"rnd\":\"0\",\"sg\":\"0\",\"spt\":\"0\",\"sa\":\"0\",\"msc\":\"\"," - "\"lo\":\"00000000000000000000000000000000\",\"ls\":\"0\"}\n"; + const String bad = "{\"type\":\"cas_gc_state\",\"version\":3}\n" + "{\"round\":\"0\",\"snapshot_generation\":\"0\",\"snapshot_pruned_through\":\"0\",\"snapshot_attempt\":\"0\",\"manifest_sweep_cursor\":\"\"," + "\"lease_owner\":\"00000000000000000000000000000000\",\"lease_sequence\":\"0\"}\n"; EXPECT_THROW(decodeGcState(bad), DB::Exception); } @@ -159,7 +159,7 @@ TEST(CASGCHeartbeatFormat, RejectsMissingIdentityFields) { /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes /// the header gate, which is the point — the BODY is what has to fail here. - const String header = "{\"type\":\"cas_gc_hb\",\"v\":3}\n"; + const String header = "{\"type\":\"cas_gc_hb\",\"version\":3}\n"; const auto expectCorrupted = [](const String & data) { @@ -174,6 +174,6 @@ TEST(CASGCHeartbeatFormat, RejectsMissingIdentityFields) } }; - expectCorrupted(header + "{\"seq\":\"1741\"}\n"); - expectCorrupted(header + "{\"by\":\"00000000000000000000000000000001\"}\n"); + expectCorrupted(header + "{\"heartbeat_sequence\":\"1741\"}\n"); + expectCorrupted(header + "{\"owner\":\"00000000000000000000000000000001\"}\n"); } diff --git a/src/Disks/tests/gtest_cas_json_writer.cpp b/src/Disks/tests/gtest_cas_json_writer.cpp index f04b89255818..3b58ee5aa43f 100644 --- a/src/Disks/tests/gtest_cas_json_writer.cpp +++ b/src/Disks/tests/gtest_cas_json_writer.cpp @@ -15,17 +15,18 @@ TEST(CASJsonWriter, KeyValueSequenceMatchesCanonicalShape) { CasJsonWriter w; bool first = true; - w.key("we", first); + w.key("writer_epoch", first); w.u64StringValue(7); - w.key("mo", first); + w.key("manifest_ordinal", first); w.u64Number(3); w.key("ok", first); w.boolValue(true); - w.key("o", "me", first); + w.key("old_", "writer_epoch", first); w.u64StringValue(1); w.closeObject(first); w.newline(); - EXPECT_EQ(std::move(w).take(), "{\"we\":\"7\",\"mo\":3,\"ok\":true,\"ome\":\"1\"}\n"); + EXPECT_EQ(std::move(w).take(), + "{\"writer_epoch\":\"7\",\"manifest_ordinal\":3,\"ok\":true,\"old_writer_epoch\":\"1\"}\n"); } TEST(CASJsonWriter, EmptyObjectAndClear) diff --git a/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp b/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp index 901b223cb90f..b6b84b19a614 100644 --- a/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp +++ b/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp @@ -247,12 +247,12 @@ TEST(CASNsFileIncarnation, LegacyUnqualifiedFileKeyIsRefusedAtOpen) /// one, so every other byte is exactly what that generation really wrote. PoolMeta meta; meta.pool_id = hexToU128("0123456789abcdef0123456789abcdef"); - meta.blob_header_len = 256; + meta.blob_header_len = 512; meta.min_reader_generation = kNamespaceLifeKeyedGeneration - 1; meta.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; String encoded = encodePoolMeta(meta); - const String current_v = "\"v\":" + std::to_string(G_BUILD); - const String legacy_v = "\"v\":" + std::to_string(kNamespaceLifeKeyedGeneration); + const String current_v = "\"version\":" + std::to_string(G_BUILD); + const String legacy_v = "\"version\":" + std::to_string(kNamespaceLifeKeyedGeneration); const size_t at = encoded.find(current_v); /// Guard the substitution itself: a silent no-op here would leave a CURRENT-generation pool and the /// test would pass by opening a pool it believes it downgraded. diff --git a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp index 45f82c1a9e33..75901d44f632 100644 --- a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp +++ b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp @@ -553,11 +553,11 @@ TEST(CASOrphanManifestSweep, MissingImmediateEpochAfterCleanedCursorCannotBeSkip .ops = publishCommittedOps("phantom", phantom), .prev_epoch_seal = RefTxnId{6, 1}}; String malformed_later_link = encodeRefLogTxn(direct_later_link); - const String encoded_predecessor{R"("!pse":"6")"}; + const String encoded_predecessor{R"("!previous_seal_writer_epoch":"6")"}; const size_t predecessor_pos = malformed_later_link.find(encoded_predecessor); ASSERT_NE(predecessor_pos, String::npos); malformed_later_link.replace( - predecessor_pos, encoded_predecessor.size(), R"("!pse":"2")"); + predecessor_pos, encoded_predecessor.size(), R"("!previous_seal_writer_epoch":"2")"); ASSERT_EQ(backend->putIfAbsent( store->layout().refLogKey(life, RefTxnId{7, 1}), sealObject(FormatId::RefLog, malformed_later_link)).outcome, PutOutcome::Done); diff --git a/src/Disks/tests/gtest_cas_part_manifest_format.cpp b/src/Disks/tests/gtest_cas_part_manifest_format.cpp index abb81c1928aa..ef6c7e6962c7 100644 --- a/src/Disks/tests/gtest_cas_part_manifest_format.cpp +++ b/src/Disks/tests/gtest_cas_part_manifest_format.cpp @@ -65,10 +65,10 @@ TEST(CASFormatBattery, PartManifest) /// stays self-consistent with whatever sample() produces, now that decode verifies payload_digest. const String golden = currentFormatHeader("cas_part_manifest") + - "{\"me\":\"5\",\"mb\":\"15\",\"mo\":1,\"ns\":\"00/aa@cas@\",\"pd\":\"" + u128ToHex(m.payload_digest) + "\"}\n" // NOLINT(modernize-raw-string-literal): mixes '\"' quoting with '\n' line endings across this concatenated literal; a raw string can't hold the newline as-is. - "{\"p\":\"a/b.bin\",\"pm\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddeeff\",\"sz\":4096}\n" - "{\"p\":\"c/small.txt\",\"pm\":\"inline\",\"il\":12}\n" - "{\"n\":2}\n" + "{\"writer_epoch\":\"5\",\"build_sequence\":\"15\",\"manifest_ordinal\":1,\"namespace\":\"00/aa@cas@\",\"payload_digest\":\"" + u128ToHex(m.payload_digest) + "\"}\n" // NOLINT(modernize-raw-string-literal): mixes '\"' quoting with '\n' line endings across this concatenated literal; a raw string can't hold the newline as-is. + "{\"path\":\"a/b.bin\",\"placement\":\"blob\",\"hash_algorithm\":\"ch128\",\"hash\":\"00112233445566778899aabbccddeeff\",\"size\":4096}\n" + "{\"path\":\"c/small.txt\",\"placement\":\"inline\",\"inline_length\":12}\n" + "{\"record_count\":2}\n" "==> \"c/small.txt\" il=12 <==\n" "hello world!\n"; runFormatBattery({FormatId::PartManifest, @@ -113,14 +113,14 @@ TEST(CASPartManifestFormat, EmptyEntriesRoundTrips) TEST(CASPartManifestFormat, PlacementWordsRenderAndRejectUnknown) { const String text = encodePartManifest(sample()); - EXPECT_NE(text.find("\"pm\":\"blob\""), String::npos); - EXPECT_NE(text.find("\"pm\":\"inline\""), String::npos); + EXPECT_NE(text.find("\"placement\":\"blob\""), String::npos); + EXPECT_NE(text.find("\"placement\":\"inline\""), String::npos); /// An unknown placement word fails closed. String bad = text; - const size_t pos = bad.find(R"("pm":"blob")"); + const size_t pos = bad.find(R"("placement":"blob")"); ASSERT_NE(pos, String::npos); - bad.replace(pos, String(R"("pm":"blob")").size(), R"("pm":"bogus")"); + bad.replace(pos, String(R"("placement":"blob")").size(), R"("placement":"bogus")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } @@ -320,8 +320,8 @@ TEST(CASPartManifestFormat, DecodeRejectsOutOfOrderEntries) m.payload_digest = computePayloadDigest(m); const String text = encodePartManifest(m); - const size_t pos_a = text.find(R"("p":"a/one.bin")"); - const size_t pos_b = text.find(R"("p":"b/two.bin")"); + const size_t pos_a = text.find(R"("path":"a/one.bin")"); + const size_t pos_b = text.find(R"("path":"b/two.bin")"); ASSERT_NE(pos_a, String::npos); ASSERT_NE(pos_b, String::npos); @@ -364,10 +364,10 @@ TEST(CASPartManifestFormat, DecodeRejectsNonAdjacentDuplicatePath) m.payload_digest = computePayloadDigest(m); String forged = encodePartManifest(m); - const String needle = R"("p":"ccc/three.bin")"; + const String needle = R"("path":"ccc/three.bin")"; const size_t pos = forged.find(needle); ASSERT_NE(pos, String::npos); - forged.replace(pos, needle.size(), R"("p":"aaa/one.bin")"); + forged.replace(pos, needle.size(), R"("path":"aaa/one.bin")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(forged); }); } @@ -375,10 +375,10 @@ TEST(CASPartManifestFormat, DecodeRejectsNonAdjacentDuplicatePath) TEST(CASPartManifestFormat, UnknownEntryAlgoFailsClosed) { String bad = encodePartManifest(sample()); - const String needle = R"("ha":"ch128")"; + const String needle = R"("hash_algorithm":"ch128")"; const size_t pos = bad.find(needle); ASSERT_NE(pos, String::npos); - bad.replace(pos, needle.size(), R"("ha":"bogus")"); + bad.replace(pos, needle.size(), R"("hash_algorithm":"bogus")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } @@ -388,7 +388,7 @@ TEST(CASPartManifestFormat, UnknownEntryAlgoFailsClosed) TEST(CASPartManifestFormat, DigestHexWidthMismatchFailsClosedNotBadArguments) { String bad = encodePartManifest(sample()); - const String key = R"("h":")"; + const String key = R"("hash":")"; const size_t key_pos = bad.find(key); ASSERT_NE(key_pos, String::npos); const size_t hex_start = key_pos + key.size(); @@ -435,10 +435,10 @@ TEST(CASPartManifestFormat, TrailingByteAfterPayloadZoneFailsClosed) TEST(CASPartManifestFormat, InlineRecordIlMismatchWithPayloadZoneBannerFailsClosed) { String bad = encodePartManifest(sample()); - const String needle = "\"il\":12"; + const String needle = "\"inline_length\":12"; const size_t pos = bad.find(needle); ASSERT_NE(pos, String::npos); - bad.replace(pos, needle.size(), "\"il\":13"); + bad.replace(pos, needle.size(), "\"inline_length\":13"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } diff --git a/src/Disks/tests/gtest_cas_part_write.cpp b/src/Disks/tests/gtest_cas_part_write.cpp index 7a426676cbc8..b527f55e9fd6 100644 --- a/src/Disks/tests/gtest_cas_part_write.cpp +++ b/src/Disks/tests/gtest_cas_part_write.cpp @@ -341,7 +341,7 @@ TEST(CASPartWriteTxn, PutBlobWritesEnvelopeWithFixedHeader) auto raw = b->get(s->layout().blobKey(ref.ref)); ASSERT_TRUE(raw.has_value()); auto h = decodeEnvelopeHeader(raw->bytes, raw->bytes.size(), ObjectKind::Blob); - EXPECT_EQ(h.header_len, s->poolMeta().blob_header_len); /// 256 + EXPECT_EQ(h.header_len, s->poolMeta().blob_header_len); /// 512 /// `logical_size`/`logical_hash` were dropped 2026-07-11, and `domain_id` in codecs-v3 phase 7 /// (the pool id no longer travels in the envelope) — identity is the content key and the payload /// starts at the fixed offset `header_len`. diff --git a/src/Disks/tests/gtest_cas_pluggable_hash.cpp b/src/Disks/tests/gtest_cas_pluggable_hash.cpp index dca4081c0041..6d2795afec10 100644 --- a/src/Disks/tests/gtest_cas_pluggable_hash.cpp +++ b/src/Disks/tests/gtest_cas_pluggable_hash.cpp @@ -127,12 +127,12 @@ TEST(CASPluggableHash, PoolMetaRoundTripsAlgosUsed) { PoolMeta pm; pm.pool_id = u128Of("pool-a"); - pm.blob_header_len = 256; + pm.blob_header_len = 512; pm.algos_used = {static_cast(BlobHashAlgo::CityHash128), static_cast(BlobHashAlgo::XXH3_128)}; const PoolMeta back = decodePoolMeta(encodePoolMeta(pm)); EXPECT_EQ(back.algos_used, pm.algos_used); - EXPECT_EQ(back.blob_header_len, 256u); + EXPECT_EQ(back.blob_header_len, 512u); } TEST(CASPluggableHash, CreateOrValidateRecordsConfigAlgoOnFreshPool) @@ -140,11 +140,11 @@ TEST(CASPluggableHash, CreateOrValidateRecordsConfigAlgoOnFreshPool) auto backend = std::make_shared(); const Layout layout("p"); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 256, BlobHashAlgo::XXH3_128, /*allow_new*/ false, /*allow_mint*/ true); + const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 512, BlobHashAlgo::XXH3_128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::XXH3_128)})); /// Reopening with the SAME algo is a no-op reopen: the recorded value comes back unchanged. - const PoolMeta reopened = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::XXH3_128); + const PoolMeta reopened = PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::XXH3_128); EXPECT_EQ(reopened.algos_used, (std::vector{static_cast(BlobHashAlgo::XXH3_128)})); EXPECT_EQ(reopened.pool_id, pm.pool_id); } @@ -154,7 +154,7 @@ TEST(CASPluggableHash, CreateOrValidateDefaultsToCityHash128) auto backend = std::make_shared(); const Layout layout("p"); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::CityHash128)})); } @@ -167,19 +167,19 @@ TEST(CASPluggableHash, CreateOrValidateFailsClosedOnAlgoMismatchWithoutFlag) auto backend = std::make_shared(); const Layout layout("p"); - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); expectThrowsCodeWithMessage( DB::ErrorCodes::BAD_ARGUMENTS, "1", [&] { - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::XXH3_128, /*allow_new*/ false); + PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::XXH3_128, /*allow_new*/ false); }); /// The pool is untouched by the refused reopen: a subsequent open with the ORIGINAL algo still /// succeeds and returns the same pool_id. - const PoolMeta reopened = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128); + const PoolMeta reopened = PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::CityHash128); EXPECT_EQ(reopened.algos_used, (std::vector{static_cast(BlobHashAlgo::CityHash128)})); } @@ -189,18 +189,18 @@ TEST(CASPluggableHash, AdmissionIsFlagGated) { auto backend = std::make_shared(); const Layout layout("p"); - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); /// without the flag: refuse, pool untouched expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] - { PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256, false); }); + { PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::Sha256, false); }); /// with the flag: admitted - const PoolMeta admitted = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256, true); + const PoolMeta admitted = PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::Sha256, true); EXPECT_EQ(admitted.algos_used, (std::vector{1, 3})); /// steady state: admitted algo reopens WITHOUT the flag - const PoolMeta steady = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256, false); + const PoolMeta steady = PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::Sha256, false); EXPECT_EQ(steady.algos_used, (std::vector{1, 3})); } @@ -208,10 +208,10 @@ TEST(CASPluggableHash, ConcurrentAdmissionUnions) { auto backend = std::make_shared(); const Layout layout("p"); - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, false, /*allow_mint*/ true); - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::XXH3_128, true); - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256, true); - const PoolMeta final_pm = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, false); + PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::CityHash128, false, /*allow_mint*/ true); + PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::XXH3_128, true); + PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::Sha256, true); + const PoolMeta final_pm = PoolMeta::createOrValidate(*backend, layout, 512, BlobHashAlgo::CityHash128, false); EXPECT_EQ(final_pm.algos_used, (std::vector{1, 2, 3})); /// union, sorted, nothing lost } @@ -714,7 +714,7 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) { auto backend = std::make_shared(); const Layout layout("p"); - PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 512, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); pm.min_reader_generation = G_BUILD + 1; ASSERT_TRUE(backend->casPut(layout.poolMetaKey(), encodePoolMeta(pm), backend->get(layout.poolMetaKey())->token).outcome == CasOutcome::Committed); @@ -722,7 +722,7 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) { Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); }); } - /// BACKWARD floor: a pool whose header `v` (compatibility_version) is BELOW `G_BUILD` was written + /// BACKWARD floor: a pool whose header `version` is BELOW `G_BUILD` was written /// by an older build this reader can no longer trust -- today that is one generation short of /// `kNamespaceLifeKeyedGeneration`, a pool whose ref-object keys carry no incarnation segment, /// which this build's parsers refuse as corruption rather than read. Craft it at the text layer: @@ -731,11 +731,11 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) { auto backend = std::make_shared(); const Layout layout("p"); - PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 512, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); const String fresh_bytes = encodePoolMeta(pm); - const String from = "\"v\":" + std::to_string(G_BUILD); - const String to = "\"v\":" + std::to_string(G_BUILD - 1); + const String from = "\"version\":" + std::to_string(G_BUILD); + const String to = "\"version\":" + std::to_string(G_BUILD - 1); const auto pos = fresh_bytes.find(from); ASSERT_NE(pos, String::npos); // sanity: a fresh pool stamps the header at the floor String downgraded = fresh_bytes; diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index b0ad1b340afd..1959fe45e917 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -440,12 +440,12 @@ TEST(CASPoolMeta, CreateThenReopen) { auto b = std::make_shared(); Layout layout("p"); - PoolMeta created = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 256, + PoolMeta created = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 512, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_NE(created.pool_id, UInt128{}); - PoolMeta reopened = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 512); + PoolMeta reopened = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 1024); EXPECT_EQ(reopened.pool_id, created.pool_id); /// pool is authoritative — config ignored on reopen - EXPECT_EQ(reopened.blob_header_len, 256u); + EXPECT_EQ(reopened.blob_header_len, 512u); } TEST(CASPoolMeta, FailClosed) @@ -457,14 +457,14 @@ TEST(CASPoolMeta, FailClosed) auto b2 = std::make_shared(); b2->putIfAbsent(layout.poolMetaKey(), "garbage"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { PoolMeta::createOrValidate(*b2, layout, 256); }); + [&] { PoolMeta::createOrValidate(*b2, layout, 512); }); } TEST(CASPoolMeta, RoundTripAndReadability) { PoolMeta pm; pm.pool_id = hexToU128("0123456789abcdeffedcba9876543210"); - pm.blob_header_len = 256; + pm.blob_header_len = 512; pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; const String encoded = encodePoolMeta(pm); @@ -485,13 +485,13 @@ TEST(CASPoolMeta, RejectsBadConstantsAtCreation) auto b = std::make_shared(); Layout layout("p"); - /// not 8-aligned (above the floor, so it is the alignment rule that rejects it) + /// not 8-aligned (above the full-key envelope floor, so it is the alignment rule that rejects it) expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, - [&] { PoolMeta::createOrValidate(*b, layout, 250); }); - /// below the v3 envelope floor (240) but 8-aligned: rejected by the floor, not the alignment rule. + [&] { PoolMeta::createOrValidate(*b, layout, 506); }); + /// below the full-key envelope floor but 8-aligned: rejected by the floor, not the alignment rule. /// Without the raised floor this pool would pass creation and LOGICAL_ERROR on the first blob write. expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, - [&] { PoolMeta::createOrValidate(*b, layout, 128); }); + [&] { PoolMeta::createOrValidate(*b, layout, 256); }); /// well below the floor expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] { PoolMeta::createOrValidate(*b, layout, 64); }); @@ -513,7 +513,7 @@ TEST(CASPoolMeta, RejectsBadConstantsOnDecode) bad_pm.blob_header_len = 100; /// violates 8-alignment invariant b->putIfAbsent(layout.poolMetaKey(), encodePoolMeta(bad_pm)); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { PoolMeta::createOrValidate(*b, layout, 256); }); + [&] { PoolMeta::createOrValidate(*b, layout, 512); }); } TEST(CASPoolMeta, DecodeGarbageFails) @@ -534,13 +534,13 @@ TEST(CASPoolMeta, ConcurrentCreateRace) const UInt128 foreign = hexToU128("0123456789abcdeffedcba9876543210"); PoolMeta foreign_pm; foreign_pm.pool_id = foreign; - foreign_pm.blob_header_len = 256; + foreign_pm.blob_header_len = 512; foreign_pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; b->putIfAbsent(layout.poolMetaKey(), encodePoolMeta(foreign_pm)); - PoolMeta result = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 512); + PoolMeta result = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 1024); EXPECT_EQ(result.pool_id, foreign); - EXPECT_EQ(result.blob_header_len, 256u); /// the foreign pool's constants win + EXPECT_EQ(result.blob_header_len, 512u); /// the foreign pool's constants win } TEST(CASPoolMeta, CasConflictReReadsWinner) @@ -575,18 +575,18 @@ TEST(CASPoolMeta, CasConflictReReadsWinner) const UInt128 winner = hexToU128("0123456789abcdeffedcba9876543210"); PoolMeta winner_pm; winner_pm.pool_id = winner; - winner_pm.blob_header_len = 256; + winner_pm.blob_header_len = 512; winner_pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; auto b = std::make_shared(); b->winner_bytes = encodePoolMeta(winner_pm); Layout layout("p"); - /// Our config (512) is what we WOULD have minted, but we lose the race and inherit the winner. - PoolMeta result = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 512, + /// Our config (1024) is what we WOULD have minted, but we lose the race and inherit the winner. + PoolMeta result = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 1024, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(result.pool_id, winner); - EXPECT_EQ(result.blob_header_len, 256u); + EXPECT_EQ(result.blob_header_len, 512u); } TEST(CASPool, OpenFailsClosedOnNonEnforcingBackend) diff --git a/src/Disks/tests/gtest_cas_record_stream_format.cpp b/src/Disks/tests/gtest_cas_record_stream_format.cpp index 42e44585c357..d66f3b341712 100644 --- a/src/Disks/tests/gtest_cas_record_stream_format.cpp +++ b/src/Disks/tests/gtest_cas_record_stream_format.cpp @@ -70,7 +70,7 @@ TEST(CASRecordStream, EmptyRunRoundTripsAndChecksumMatches) { const String bytes = encodeRun({}); EXPECT_EQ(bytes, fmt::format( - "{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n{{\"n\":0}}\n", currentCompatibilityVersion())); + "{{\"type\":\"cas_run\",\"version\":{},\"kind\":\"source_edge\"}}\n{{\"record_count\":0}}\n", currentCompatibilityVersion())); ReadBufferFromMemory in(bytes.data(), bytes.size()); SourceEdgeRunReader reader(in); @@ -159,9 +159,9 @@ TEST(CASRecordStream, SourceIdRendersAs32Hex) { const String bytes = encodeRun({edge(chRef(1), 10)}); /// The source id 10 is a 32-char lowercase hex string ending in 'a'. - EXPECT_NE(bytes.find("\"s\":\"0000000000000000000000000000000a\""), String::npos); - /// The record key `b` for a ch128 ref is the algo byte 01 + a 32-hex digest (34 chars total). - EXPECT_NE(bytes.find("\"b\":\"01"), String::npos); + EXPECT_NE(bytes.find("\"source_id\":\"0000000000000000000000000000000a\""), String::npos); + /// The record key `blob_ref` for a `ch128` ref is the algorithm byte 01 + a 32-hex digest (34 chars total). + EXPECT_NE(bytes.find("\"blob_ref\":\"01"), String::npos); } TEST(CASRecordStream, SealChecksumMismatchFailsClosed) @@ -195,8 +195,8 @@ TEST(CASRecordStream, TrailerCountMismatchIsCorruptData) { String bytes = encodeRun({edge(chRef(1), 10)}); /// Rewrite the trailer count 1 -> 2. - const String from = "{\"n\":1}\n"; - const String to = "{\"n\":2}\n"; + const String from = "{\"record_count\":1}\n"; + const String to = "{\"record_count\":2}\n"; const size_t at = bytes.rfind(from); ASSERT_NE(at, String::npos); bytes.replace(at, from.size(), to); @@ -207,7 +207,7 @@ TEST(CASRecordStream, TruncationAtLineBoundaryFailsClosed) { const String bytes = encodeRun({edge(chRef(1), 10), edge(chRef(1), 20)}); /// Drop the trailer line entirely (truncate after the last record's newline). - const size_t trailer = bytes.rfind("{\"n\":"); + const size_t trailer = bytes.rfind("{\"record_count\":"); ASSERT_NE(trailer, String::npos); EXPECT_THROW(decodeRun(bytes.substr(0, trailer)), DB::Exception); } @@ -216,18 +216,18 @@ TEST(CASRecordStream, HeaderGates) { /// Wrong type. { - const String s = "{\"type\":\"cas_pool_meta\",\"v\":3,\"kind\":\"source_edge\"}\n{\"n\":0}\n"; + const String s = "{\"type\":\"cas_pool_meta\",\"version\":3,\"kind\":\"source_edge\"}\n{\"record_count\":0}\n"; EXPECT_THROW(decodeRun(s), DB::Exception); } /// Wrong kind. { - const String s = "{\"type\":\"cas_run\",\"v\":3,\"kind\":\"blob_delta\"}\n{\"n\":0}\n"; + const String s = "{\"type\":\"cas_run\",\"version\":3,\"kind\":\"blob_delta\"}\n{\"record_count\":0}\n"; EXPECT_THROW(decodeRun(s), DB::Exception); } /// Future version -> UNKNOWN_FORMAT_VERSION. { const String s = fmt::format( - "{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n{{\"n\":0}}\n", currentCompatibilityVersion() + 1); + "{{\"type\":\"cas_run\",\"version\":{},\"kind\":\"source_edge\"}}\n{{\"record_count\":0}}\n", currentCompatibilityVersion() + 1); ReadBufferFromMemory in(s.data(), s.size()); try { @@ -241,7 +241,7 @@ TEST(CASRecordStream, HeaderGates) } /// An out-of-range version must not narrow to a valid low u32 value. { - const String s = "{\"type\":\"cas_run\",\"v\":4294967299,\"kind\":\"source_edge\"}\n{\"n\":0}\n"; + const String s = "{\"type\":\"cas_run\",\"version\":4294967299,\"kind\":\"source_edge\"}\n{\"record_count\":0}\n"; try { decodeRun(s); diff --git a/src/Disks/tests/gtest_cas_recovery_grounding.cpp b/src/Disks/tests/gtest_cas_recovery_grounding.cpp index 5362813bdfa6..34dcd3fbcbb5 100644 --- a/src/Disks/tests/gtest_cas_recovery_grounding.cpp +++ b/src/Disks/tests/gtest_cas_recovery_grounding.cpp @@ -269,9 +269,9 @@ TEST(CASRecoveryGrounding, RejectsLifeEpochAboveCommittedFrontierOnDecodeAndGrou { const RefCkpt invalid = ckpt(2, RefTxnId{1, 5}); String encoded = encodeRefCkpt(ckpt(1, RefTxnId{1, 5})); - const size_t life_epoch = encoded.find(R"("le":"1")"); + const size_t life_epoch = encoded.find(R"("life_epoch":"1")"); ASSERT_NE(life_epoch, String::npos); - encoded.replace(life_epoch, String{R"("le":"1")"}.size(), R"("le":"2")"); + encoded.replace(life_epoch, String{R"("life_epoch":"1")"}.size(), R"("life_epoch":"2")"); expectCode([&] { (void)decodeRefCkpt(encoded); }, DB::ErrorCodes::CORRUPTED_DATA); expectCode([&] { (void)chooseRecoveryGrounding(catalog(NsState::Live), invalid); }, @@ -568,9 +568,9 @@ TEST(CASRecoveryGrounding, SameEpochFrontierAfterDecodedEpochSealIsCorruption) .committed_through = RefTxnId{1, 2}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = RefTxnId{1, 2}}); - const size_t frontier_sequence = malformed_ckpt.find(R"("cts":"2")"); + const size_t frontier_sequence = malformed_ckpt.find(R"("committed_through_ref_sequence":"2")"); ASSERT_NE(frontier_sequence, String::npos); - malformed_ckpt.replace(frontier_sequence, String{R"("cts":"2")"}.size(), R"("cts":"3")"); + malformed_ckpt.replace(frontier_sequence, String{R"("committed_through_ref_sequence":"2")"}.size(), R"("committed_through_ref_sequence":"3")"); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), malformed_ckpt).outcome, PutOutcome::Done); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); @@ -635,9 +635,9 @@ TEST(CASRecoveryGrounding, TerminalGapBelowFrontierIsCorruptionNotARebirth) .committed_through = RefTxnId{1, 1}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); - const size_t frontier_epoch = malformed_ckpt.find(R"("cte":"1")"); + const size_t frontier_epoch = malformed_ckpt.find(R"("committed_through_writer_epoch":"1")"); ASSERT_NE(frontier_epoch, String::npos); - malformed_ckpt.replace(frontier_epoch, String{R"("cte":"1")"}.size(), R"("cte":"2")"); + malformed_ckpt.replace(frontier_epoch, String{R"("committed_through_writer_epoch":"1")"}.size(), R"("committed_through_writer_epoch":"2")"); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), malformed_ckpt).outcome, PutOutcome::Done); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index 11a4c029f139..036e4fa0ab67 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -62,9 +62,9 @@ String rawEntLine(const String & ns, const String & state, const String & inc_he std::optional> creator = std::nullopt) { if (!creator) - return fmt::format(R"({{"k":"ent","ns":"{}","st":"{}","inc":"{}"}})", ns, state, inc_hex); + return fmt::format(R"({{"kind":"ent","namespace":"{}","state":"{}","incarnation":"{}"}})", ns, state, inc_hex); const auto & [srid, we, fg] = *creator; - return fmt::format(R"({{"k":"ent","ns":"{}","st":"{}","inc":"{}","csr":"{}","cwe":"{}","cfg":"{}"}})", + return fmt::format(R"({{"kind":"ent","namespace":"{}","state":"{}","incarnation":"{}","creator_server_root_id":"{}","creator_writer_epoch":"{}","creator_fence_generation":"{}"}})", ns, state, inc_hex, srid, we, fg); } @@ -73,10 +73,10 @@ String rawEntLine(const String & ns, const String & state, const String & inc_he /// `gtest_cas_fold_seal_format.cpp`'s `RejectsOutOfRangeNsCleanupState` uses for the same reason. String rawCatalog(const std::vector & ent_lines) { - String out = R"({"type":"cas_ref_catalog","v":1})" "\n"; + String out = R"({"type":"cas_ref_catalog","version":1})" "\n"; for (const String & l : ent_lines) out += l + "\n"; - out += fmt::format("{{\"n\":{}}}\n", ent_lines.size()); + out += fmt::format("{{\"record_count\":{}}}\n", ent_lines.size()); return out; } @@ -84,7 +84,7 @@ String withRemovalStartedRound(String line, uint64_t round) { const size_t close = line.rfind('}'); EXPECT_NE(close, String::npos); - line.insert(close, fmt::format(R"(,"rsr":"{}")", round)); + line.insert(close, fmt::format(R"(,"removal_started_round":"{}")", round)); return line; } @@ -221,10 +221,10 @@ TEST(CASFormatBattery, RefCatalog) [&] { return sealObject(FormatId::RefCatalog, encodeRefCatalog(c)); }, [](std::string_view s) { decodeRefCatalog(std::string(openObject(FormatId::RefCatalog, s))); }, currentFormatHeader("cas_ref_catalog") + - "{\"k\":\"ent\",\"ns\":\"a\",\"st\":\"creating\",\"inc\":\"00000000000000000000000000000001\"," - "\"csr\":\"srv1\",\"cwe\":\"5\",\"cfg\":\"2\"}\n" - "{\"k\":\"ent\",\"ns\":\"b\",\"st\":\"live\",\"inc\":\"00000000000000000000000000000002\"}\n" - "{\"n\":2}\n"}); + "{\"kind\":\"ent\",\"namespace\":\"a\",\"state\":\"creating\",\"incarnation\":\"00000000000000000000000000000001\"," + "\"creator_server_root_id\":\"srv1\",\"creator_writer_epoch\":\"5\",\"creator_fence_generation\":\"2\"}\n" + "{\"kind\":\"ent\",\"namespace\":\"b\",\"state\":\"live\",\"incarnation\":\"00000000000000000000000000000002\"}\n" + "{\"record_count\":2}\n"}); } /// ---------- codec round-trip ---------- @@ -260,7 +260,7 @@ TEST(CASRefCatalogFormat, RemovalStartedRoundIsRequiredExactlyForRemoving) .removal_started_round = 19}; const RefCatalog catalog{.entries = {removing}}; const String encoded = encodeRefCatalog(catalog); - EXPECT_NE(encoded.find("\"rsr\":\"19\""), String::npos); + EXPECT_NE(encoded.find("\"removal_started_round\":\"19\""), String::npos); EXPECT_EQ(decodeRefCatalog(encoded), catalog); const String inc = "00000000000000000000000000000009"; @@ -546,7 +546,7 @@ TEST(CASRefCatalogFormat, DecodeRejectsEmptyNamespace) TEST(CASRefCatalogFormat, DecodeRejectsMissingNamespaceKey) { /// No "ns" key at all -- must be refused exactly like an explicit empty one, not read as "". - const String bad = rawCatalog({R"({"k":"ent","st":"live","inc":")" + u128ToHex(UInt128(1)) + "\"}"}); + const String bad = rawCatalog({R"({"kind":"ent","state":"live","incarnation":")" + u128ToHex(UInt128(1)) + "\"}"}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } diff --git a/src/Disks/tests/gtest_cas_ref_ckpt.cpp b/src/Disks/tests/gtest_cas_ref_ckpt.cpp index 9b63fa4dfc92..b6b24c1ea223 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt.cpp @@ -12,8 +12,6 @@ #include #include #include -#include -#include #include @@ -275,8 +273,8 @@ TEST(CASRefCheckpoint, CommittedThroughHasCanonicalExactWireEncoding) .committed_through = RefTxnId{9, 11}, .checkpoint_snapshot_id = RefTxnId{9, 10}, .last_epoch_seal = RefTxnId{8, 12}}; - const String expected = R"({"type":"cas_ref_ckpt","v":10} -{"le":"7","cte":"9","cts":"11","cse":"9","css":"10","lse":"8","lss":"12"} + const String expected = R"({"type":"cas_ref_ckpt","version":10} +{"life_epoch":"7","committed_through_writer_epoch":"9","committed_through_ref_sequence":"11","checkpoint_snapshot_writer_epoch":"9","checkpoint_snapshot_ref_sequence":"10","last_epoch_seal_writer_epoch":"8","last_epoch_seal_ref_sequence":"12"} )"; EXPECT_EQ(encodeRefCkpt(ckpt), expected); @@ -307,9 +305,9 @@ TEST(CASRefCheckpoint, CodecRejectsIncoherentCommittedFrontierAndSealEpochs) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { encodeRefCkpt(unsealed_non_genesis); }); String malformed = encodeRefCkpt(valid); - const size_t cte = malformed.find(R"("cte":"8")"); + const size_t cte = malformed.find(R"("committed_through_writer_epoch":"8")"); ASSERT_NE(cte, String::npos); - malformed.replace(cte, String{R"("cte":"8")"}.size(), R"("cte":"10")"); + malformed.replace(cte, String{R"("committed_through_writer_epoch":"8")"}.size(), R"("committed_through_writer_epoch":"10")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(malformed); }); } @@ -337,7 +335,7 @@ TEST(CASRefCheckpoint, RejectsADuplicateKey) const String good = encodeRefCkpt(RefCkpt{.life_epoch = std::optional{7}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); String duplicated = good; - duplicated.replace(duplicated.rfind('}'), 1, R"(,"le":"9"})"); + duplicated.replace(duplicated.rfind('}'), 1, R"(,"life_epoch":"9"})"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(duplicated); }); } @@ -361,13 +359,13 @@ TEST(CASRefCheckpoint, RejectsTruncation) const String empty_body = good.substr(0, good.find('\n') + 1) + "{}\n"; EXPECT_EQ(decodeRefCkpt(empty_body), RefCkpt{}); - const String half_pair = good.substr(0, good.find('\n') + 1) + R"({"le":"7","cse":"1"})" + "\n"; + const String half_pair = good.substr(0, good.find('\n') + 1) + R"({"life_epoch":"7","checkpoint_snapshot_writer_epoch":"1"})" + "\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(half_pair); }); - const String other_half = good.substr(0, good.find('\n') + 1) + R"({"le":"7","lss":"2"})" + "\n"; + const String other_half = good.substr(0, good.find('\n') + 1) + R"({"life_epoch":"7","last_epoch_seal_ref_sequence":"2"})" + "\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(other_half); }); - const String frontier_half = good.substr(0, good.find('\n') + 1) + R"({"le":"7","cte":"1"})" + "\n"; + const String frontier_half = good.substr(0, good.find('\n') + 1) + R"({"life_epoch":"7","committed_through_writer_epoch":"1"})" + "\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(frontier_half); }); } @@ -398,9 +396,9 @@ TEST(CASRefCheckpoint, RejectsInvalidFieldsOnEncodeAndOnDecode) const String header = encodeRefCkpt(RefCkpt{.life_epoch = std::optional{7}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); const String prefix = header.substr(0, header.find('\n') + 1); - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(prefix + R"({"le":"0"})" + "\n"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(prefix + R"({"life_epoch":"0"})" + "\n"); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeRefCkpt(prefix + R"({"le":"7","cse":"1","css":"0"})" + "\n"); }); + [&] { decodeRefCkpt(prefix + R"({"life_epoch":"7","checkpoint_snapshot_writer_epoch":"1","checkpoint_snapshot_ref_sequence":"0"})" + "\n"); }); } /// The registry row is part of the contract: Control/Strict decides how the decoder treats unknown @@ -1187,7 +1185,7 @@ TEST(CASRefCheckpoint, APublishFencedOutMidAttemptDoesNotAdvanceTheCheckpoint) /// backend and the real append lane, which this suite already drives through `publishRef` (including /// the namespace birth, the one chunk shape whose first durable effect is the `_ckpt` and not the /// ref-log `PUT`). -TEST(CASRefCheckpoint, CommitRefChunkDurableBytesUnchangedByExtraction) +TEST(CASRefCheckpoint, CommitRefChunkHasCanonicalDurableBytes) { auto backend = std::make_shared(); auto store = openPool(backend); @@ -1208,31 +1206,20 @@ TEST(CASRefCheckpoint, CommitRefChunkDurableBytesUnchangedByExtraction) + "/_log/0000000000000001-0000000000000001.zst") << "the canonical ref-log key the append lane derives"; - /// The BODY is checked as exact length plus a 128-bit SipHash of it -- not literally byte for byte, - /// but any change that survives both is a 128-bit collision at a fixed length, which is the trade for - /// keeping the assertion readable. It is a function of `{format generation, ns, id, ops, - /// chain_link}` only -- no incarnation reaches it. Generation 10 changed the shared format header; - /// the plaintext discriminator below removes only that change and pins every remaining byte to the - /// generation-9 fixture before accepting the new deterministic compressed size and hash. + /// The decoded body is compared byte for byte. It is a function of `{format generation, namespace, + /// id, operations, chain_link}` only -- no incarnation reaches it. const auto got = backend->get(key); ASSERT_TRUE(got.has_value()) << "the birth chunk must be durable at its canonical key"; - String as_generation_9 = openObject(FormatId::RefLog, got->bytes); - const String generation_10_header = R"({"type":"cas_ref_log","v":10})"; - ASSERT_TRUE(as_generation_9.starts_with(generation_10_header)); - as_generation_9.replace(0, generation_10_header.size(), R"({"type":"cas_ref_log","v":9})"); - EXPECT_EQ(as_generation_9, R"({"type":"cas_ref_log","v":9} -{"ns":"test/golden@cas@","we":"1","rs":"1"} -{"op":"namespace_birth"} -{"op":"owner_transition","nbk":"precommit","nrn":"gold_ref","nme":"1","nmb":"7","nmo":1} -{"op":"owner_transition","obk":"precommit","orn":"gold_ref","ome":"1","omb":"7","omo":1,"nbk":"committed","nrn":"gold_ref","nme":"1","nmb":"7","nmo":1} -{"n":3} -)") << "generation 10 must change only the self-describing header of this ref-log fixture"; - EXPECT_EQ(got->bytes.size(), 179u) << "the sealed ref-log body changed size"; - SipHash body_hash; - body_hash.update(got->bytes.data(), got->bytes.size()); - EXPECT_EQ(getHexUIntLowercase(body_hash.get128()), "ada75a83638e933c98d731183a46b7b7") - << "the sealed ref-log body changed content -- preparation must seal the same bytes it sealed " - "before the extraction"; + const String body = openObject(FormatId::RefLog, got->bytes); + const String generation_10_header = R"({"type":"cas_ref_log","version":10})"; + ASSERT_TRUE(body.starts_with(generation_10_header)); + EXPECT_EQ(body, R"({"type":"cas_ref_log","version":10} +{"namespace":"test/golden@cas@","writer_epoch":"1","ref_sequence":"1"} +{"operation":"namespace_birth"} +{"operation":"owner_transition","new_binding_kind":"precommit","new_ref_name":"gold_ref","new_writer_epoch":"1","new_build_sequence":"7","new_manifest_ordinal":1} +{"operation":"owner_transition","old_binding_kind":"precommit","old_ref_name":"gold_ref","old_writer_epoch":"1","old_build_sequence":"7","old_manifest_ordinal":1,"new_binding_kind":"committed","new_ref_name":"gold_ref","new_writer_epoch":"1","new_build_sequence":"7","new_manifest_ordinal":1} +{"record_count":3} +)") << "the durable ref-log body must use the canonical readable keys"; } /// The directive's "preserve backend request counts", asserted rather than assumed: preparation is pure, diff --git a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp index bc911e4ad6db..b6b300ca805a 100644 --- a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp +++ b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp @@ -296,7 +296,7 @@ TEST(CASRefContiguousAlloc, OldPoolFormatIsRefusedNamingRecreation) { PoolMeta pm; pm.pool_id = UInt128{1, 2}; - pm.blob_header_len = 256; + pm.blob_header_len = 512; pm.min_reader_generation = G_BUILD; pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; @@ -305,8 +305,8 @@ TEST(CASRefContiguousAlloc, OldPoolFormatIsRefusedNamingRecreation) /// Rewrite the header-line generation to the last pre-contiguous one, exactly as an older build /// would have stamped it. - const String from = "\"v\":" + std::to_string(G_BUILD); - const String to = "\"v\":" + std::to_string(kContiguousRefStreamsGeneration - 1); + const String from = "\"version\":" + std::to_string(G_BUILD); + const String to = "\"version\":" + std::to_string(kContiguousRefStreamsGeneration - 1); const size_t at = current.find(from); ASSERT_NE(at, String::npos); String old_format = current; @@ -333,7 +333,7 @@ TEST(CASRefContiguousAlloc, GenerationFiveNamespaceBearingPoolIsRefusedNamingRec { PoolMeta pm; pm.pool_id = UInt128{1, 2}; - pm.blob_header_len = 256; + pm.blob_header_len = 512; pm.min_reader_generation = G_BUILD; pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; @@ -342,8 +342,8 @@ TEST(CASRefContiguousAlloc, GenerationFiveNamespaceBearingPoolIsRefusedNamingRec /// Rewrite the header to the immediately preceding generation, which used /// `cas/refs///...`. - const String from = "\"v\":" + std::to_string(G_BUILD); - const String to = "\"v\":" + std::to_string(kNamespaceLifeKeyedGeneration); + const String from = "\"version\":" + std::to_string(G_BUILD); + const String to = "\"version\":" + std::to_string(kNamespaceLifeKeyedGeneration); const size_t at = current.find(from); ASSERT_NE(at, String::npos); String old_format = current; @@ -372,13 +372,13 @@ TEST(CASRefContiguousAlloc, GenerationSixSplitFoldSealPoolIsRefusedNamingRecreat { PoolMeta pm; pm.pool_id = UInt128{1, 2}; - pm.blob_header_len = 256; + pm.blob_header_len = 512; pm.min_reader_generation = G_BUILD; pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; const String current = encodePoolMeta(pm); - const String from = "\"v\":" + std::to_string(G_BUILD); - const String to = "\"v\":6"; + const String from = "\"version\":" + std::to_string(G_BUILD); + const String to = "\"version\":6"; const size_t at = current.find(from); ASSERT_NE(at, String::npos); String old_format = current; @@ -402,12 +402,12 @@ TEST(CASPoolMeta, GcShardsIsPersistedAndOverridesMismatchedReopenConfig) InMemoryBackend backend; const Layout layout("p"); const PoolMeta created = PoolMeta::createOrValidate( - backend, layout, /*blob_header_len=*/256, /*gc_shards=*/4, + backend, layout, /*blob_header_len=*/512, /*gc_shards=*/4, BlobHashAlgo::CityHash128, /*allow_new=*/false, /*allow_mint=*/true); EXPECT_EQ(created.gc_shards, 4u); const PoolMeta reopened = PoolMeta::createOrValidate( - backend, layout, /*blob_header_len=*/256, /*gc_shards=*/1, + backend, layout, /*blob_header_len=*/512, /*gc_shards=*/1, BlobHashAlgo::CityHash128, /*allow_new=*/false, /*allow_mint=*/false); EXPECT_EQ(reopened.gc_shards, 4u); EXPECT_EQ(decodePoolMeta(backend.get(layout.poolMetaKey())->bytes).gc_shards, 4u); diff --git a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp index 5b6fa2070c46..5c7b53834859 100644 --- a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp @@ -165,16 +165,16 @@ TEST(CASRefEpochSealFormat, DecodeRejectsSealTxnWithTwoOpsSpliced) txn.ops.push_back(epochSealOp()); const String bytes = encodeRefLogTxn(txn); - const String op_line = "{\"op\":\"epoch_seal\"}\n"; + const String op_line = "{\"operation\":\"epoch_seal\"}\n"; const auto op_pos = bytes.find(op_line); ASSERT_NE(op_pos, String::npos); String tampered = bytes; tampered.insert(op_pos, op_line); /// two consecutive "epoch_seal" op lines now - const String old_trailer = "{\"n\":1}\n"; + const String old_trailer = "{\"record_count\":1}\n"; const auto trailer_pos = tampered.find(old_trailer); ASSERT_NE(trailer_pos, String::npos); - tampered.replace(trailer_pos, old_trailer.size(), "{\"n\":2}\n"); /// keep the trailer honest + tampered.replace(trailer_pos, old_trailer.size(), "{\"record_count\":2}\n"); /// keep the trailer honest expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -189,16 +189,16 @@ TEST(CASRefEpochSealFormat, DecodeRejectsSealTxnWithSecondNonSealOpSpliced) txn.ops.push_back(epochSealOp()); const String bytes = encodeRefLogTxn(txn); - const String op_line = "{\"op\":\"epoch_seal\"}\n"; + const String op_line = "{\"operation\":\"epoch_seal\"}\n"; const auto op_pos = bytes.find(op_line); ASSERT_NE(op_pos, String::npos); String tampered = bytes; - tampered.insert(op_pos + op_line.size(), "{\"op\":\"namespace_birth\"}\n"); + tampered.insert(op_pos + op_line.size(), "{\"operation\":\"namespace_birth\"}\n"); - const String old_trailer = "{\"n\":1}\n"; + const String old_trailer = "{\"record_count\":1}\n"; const auto trailer_pos = tampered.find(old_trailer); ASSERT_NE(trailer_pos, String::npos); - tampered.replace(trailer_pos, old_trailer.size(), "{\"n\":2}\n"); + tampered.replace(trailer_pos, old_trailer.size(), "{\"record_count\":2}\n"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -224,11 +224,11 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealAtNonUnitSequenceSpliced) txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"("rs":"2")"; + const String needle = R"("ref_sequence":"2")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; - tampered.insert(pos + needle.size(), R"(,"!pse":"1","!pss":"1")"); + tampered.insert(pos + needle.size(), R"(,"!previous_seal_writer_epoch":"1","!previous_seal_ref_sequence":"1")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -267,7 +267,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealMissingPssComponent) txn.ops.push_back(epochSealOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"(,"!pss":"9")"; + const String needle = R"(,"!previous_seal_ref_sequence":"9")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; @@ -326,11 +326,11 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealSkippingImmediateEpochSpli txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"("rs":"1")"; + const String needle = R"("ref_sequence":"1")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; - tampered.insert(pos + needle.size(), R"(,"!pse":"3","!pss":"1")"); + tampered.insert(pos + needle.size(), R"(,"!previous_seal_writer_epoch":"3","!previous_seal_ref_sequence":"1")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -346,11 +346,11 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealPointingAtSameOrFutureEpoc txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); /// valid: sequence 1, no prev_epoch_seal - const String needle = R"("rs":"1")"; + const String needle = R"("ref_sequence":"1")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; - tampered.insert(pos + needle.size(), R"(,"!pse":"5","!pss":"1")"); /// self-pointer + tampered.insert(pos + needle.size(), R"(,"!previous_seal_writer_epoch":"5","!previous_seal_ref_sequence":"1")"); /// self-pointer expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -437,7 +437,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsUnknownCriticalKeyInMetaLine) txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"("rs":"1")"; + const String needle = R"("ref_sequence":"1")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; @@ -484,8 +484,8 @@ TEST(CASRefEpochSealFormat, FormatBatteryEpochSeal) runFormatBattery({FormatId::RefLog, [txn] { return sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); }, [ns, id](std::string_view s) { decodeRefLogTxn(openObject(FormatId::RefLog, s), ns, id); }, - "{\"type\":\"cas_ref_log\",\"v\":10}\n" - "{\"ns\":\"ns\",\"we\":\"3\",\"rs\":\"1\",\"!pse\":\"2\",\"!pss\":\"9\"}\n" - "{\"op\":\"epoch_seal\"}\n" - "{\"n\":1}\n"}); + "{\"type\":\"cas_ref_log\",\"version\":10}\n" + "{\"namespace\":\"ns\",\"writer_epoch\":\"3\",\"ref_sequence\":\"1\",\"!previous_seal_writer_epoch\":\"2\",\"!previous_seal_ref_sequence\":\"9\"}\n" + "{\"operation\":\"epoch_seal\"}\n" + "{\"record_count\":1}\n"}); } diff --git a/src/Disks/tests/gtest_cas_ref_log_format.cpp b/src/Disks/tests/gtest_cas_ref_log_format.cpp index d5186e77aca1..4044ffc9b706 100644 --- a/src/Disks/tests/gtest_cas_ref_log_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_log_format.cpp @@ -10,7 +10,7 @@ /// `gtest_cas_ref_codecs.cpp` and re-pointed at the TEXT codec: the encoder-side validation tests are /// format-agnostic (they only assert `encodeRefLogTxn` throws) and carry over verbatim; the old /// binary-offset byte-patch decode tests (`bytes[k] = 99`) are gone — the shape-level corruption -/// classes (truncation, `v`+1 forward-gate, wrong type, leading garbage) are now covered by the +/// classes (truncation, `version`+1 forward-gate, wrong type, leading garbage) are now covered by the /// `CASFormatBattery.RefLog` row below. `RefTxnId` render/parse coverage lives here too (it rode in /// the same suite and is independent of either ref codec). @@ -205,10 +205,10 @@ TEST(CASRefCodec, DecodeRejectsRemovedPayloadFieldInOpRecord) const String bytes = encodeRefLogTxn(txn); /// Splice the retired `"pl"` field back into the op record, just before its `"ts"` field. - const String needle = ",\"ts\":"; + const String needle = ",\"published_at_ms\":"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); - const String tampered = bytes.substr(0, pos) + R"(,"pl":"deadbeef")" + bytes.substr(pos); + const String tampered = bytes.substr(0, pos) + R"(,"payload":"deadbeef")" + bytes.substr(pos); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -784,7 +784,7 @@ TEST(CASFormatBattery, RefLog) [txn] { return sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); }, [ns, id](std::string_view s) { decodeRefLogTxn(openObject(FormatId::RefLog, s), ns, id); }, currentFormatHeader("cas_ref_log") + - "{\"ns\":\"ns\",\"we\":\"1\",\"rs\":\"1\"}\n" - "{\"op\":\"set_published_at\",\"rn\":\"all_1_1_0\",\"me\":\"1\",\"mb\":\"1\",\"mo\":1,\"ts\":42}\n" - "{\"n\":1}\n"}); + "{\"namespace\":\"ns\",\"writer_epoch\":\"1\",\"ref_sequence\":\"1\"}\n" + "{\"operation\":\"set_published_at\",\"ref_name\":\"all_1_1_0\",\"writer_epoch\":\"1\",\"build_sequence\":\"1\",\"manifest_ordinal\":1,\"published_at_ms\":42}\n" + "{\"record_count\":1}\n"}); } diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp index 2293c0bc166c..1eb29898fa7b 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp @@ -7,10 +7,10 @@ /// v3 text codec tests for `cas_ref_snap` (codecs-v3 phase 3). Split out of the retired /// `gtest_cas_ref_codecs.cpp` and re-pointed at the TEXT codec. The encoder-side validation tests are /// format-agnostic and carry over verbatim; the old binary-offset byte-patch decode tests -/// (`bytes[k] = 99`) are gone -- the shape-level corruption classes (truncation, `v`+1 forward-gate, +/// (`bytes[k] = 99`) are gone -- the shape-level corruption classes (truncation, `version`+1 forward-gate, /// wrong type, leading garbage) are covered by the `CASFormatBattery.RefSnapshot` row below, which also /// subsumes the old `DecodeRejectsFutureFormatVersion`/`DecodeRejectsFormatVersionOne` pair (there is -/// no `format_version` byte any more -- the header `v` gate is the single forward-compat mechanism). +/// no `format_version` byte any more -- the header `version` gate is the single forward-compat mechanism). using namespace DB::Cas; using DB::Cas::tests::expectThrowsCode; @@ -65,7 +65,7 @@ TEST(CASRefSnapshotCodec, DecodeRequiresLifecycleField) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String field = R"(,"lc":"live")"; + const String field = R"(,"lifecycle":"live")"; const size_t at = bytes.find(field); ASSERT_NE(at, String::npos); bytes.erase(at, field.size()); @@ -78,10 +78,10 @@ TEST(CASRefSnapshotCodec, DecodeRejectsTerminalLifecycleWord) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); - bytes.replace(at, live.size(), R"("lc":"removed")"); + bytes.replace(at, live.size(), R"("lifecycle":"removed")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeRefTableSnapshot(bytes, s.ns, s.snapshot_id); }); @@ -91,10 +91,10 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnEpochField) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); - bytes.replace(at, live.size(), live + R"(,"rte":"7")"); + bytes.replace(at, live.size(), live + R"(,"remove_txn_writer_epoch":"7")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeRefTableSnapshot(bytes, s.ns, s.snapshot_id); }); @@ -104,10 +104,10 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnSequenceField) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); - bytes.replace(at, live.size(), live + R"(,"rts":"9")"); + bytes.replace(at, live.size(), live + R"(,"remove_txn_ref_sequence":"9")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeRefTableSnapshot(bytes, s.ns, s.snapshot_id); }); @@ -117,10 +117,10 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnFieldPair) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); - bytes.replace(at, live.size(), live + R"(,"rte":"7","rts":"9")"); + bytes.replace(at, live.size(), live + R"(,"remove_txn_writer_epoch":"7","remove_txn_ref_sequence":"9")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeRefTableSnapshot(bytes, s.ns, s.snapshot_id); }); @@ -143,10 +143,10 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRemovedPayloadFieldInCommittedRow) const String bytes = encodeRefTableSnapshot(s); /// Splice the retired `"pl"` field back into the committed record, just before its `"ts"` field. - const String needle = ",\"ts\":"; + const String needle = ",\"published_at_ms\":"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); - const String tampered = bytes.substr(0, pos) + R"(,"pl":"deadbeef")" + bytes.substr(pos); + const String tampered = bytes.substr(0, pos) + R"(,"payload":"deadbeef")" + bytes.substr(pos); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefTableSnapshot(tampered, s.ns, s.snapshot_id); }); @@ -207,7 +207,7 @@ TEST(CASRefSnapshotFormat, MaximalRefSequenceRoundTripsAsADecimalString) const String text = encodeRefTableSnapshot(m); const RefTableSnapshot back = decodeRefTableSnapshot(text, m.ns, m.snapshot_id); EXPECT_EQ(back.snapshot_id.ref_sequence, std::numeric_limits::max()); - EXPECT_NE(text.find("\"rs\":\"18446744073709551615\""), String::npos); + EXPECT_NE(text.find("\"ref_sequence\":\"18446744073709551615\""), String::npos); } /// =================================================================================== @@ -428,9 +428,9 @@ TEST(CASFormatBattery, RefSnapshot) [s] { return sealObject(FormatId::RefSnapshot, encodeRefTableSnapshot(s)); }, [ns, id](std::string_view d) { decodeRefTableSnapshot(openObject(FormatId::RefSnapshot, d), ns, id); }, currentFormatHeader("cas_ref_snap") + - "{\"ns\":\"srv1/db/table@cas@\",\"we\":\"5\",\"rs\":\"200\",\"lc\":\"live\"}\n" - "{\"k\":\"c\",\"rn\":\"all_1_1_0\",\"me\":\"5\",\"mb\":\"10\",\"mo\":1,\"ts\":1717000000000}\n" - "{\"k\":\"c\",\"rn\":\"all_2_2_0\",\"me\":\"5\",\"mb\":\"11\",\"mo\":1,\"ts\":1717000000001}\n" - "{\"k\":\"p\",\"rn\":\"all_3_3_0\",\"me\":\"5\",\"mb\":\"12\",\"mo\":1}\n" - "{\"n\":3}\n"}); + "{\"namespace\":\"srv1/db/table@cas@\",\"writer_epoch\":\"5\",\"ref_sequence\":\"200\",\"lifecycle\":\"live\"}\n" + "{\"kind\":\"c\",\"ref_name\":\"all_1_1_0\",\"writer_epoch\":\"5\",\"build_sequence\":\"10\",\"manifest_ordinal\":1,\"published_at_ms\":1717000000000}\n" + "{\"kind\":\"c\",\"ref_name\":\"all_2_2_0\",\"writer_epoch\":\"5\",\"build_sequence\":\"11\",\"manifest_ordinal\":1,\"published_at_ms\":1717000000001}\n" + "{\"kind\":\"p\",\"ref_name\":\"all_3_3_0\",\"writer_epoch\":\"5\",\"build_sequence\":\"12\",\"manifest_ordinal\":1}\n" + "{\"record_count\":3}\n"}); } diff --git a/src/Disks/tests/gtest_cas_s3_staging.cpp b/src/Disks/tests/gtest_cas_s3_staging.cpp index 92c4542b1a32..24417247602e 100644 --- a/src/Disks/tests/gtest_cas_s3_staging.cpp +++ b/src/Disks/tests/gtest_cas_s3_staging.cpp @@ -691,7 +691,7 @@ TEST(CASS3Staging, PublishOverCondemnedBlobUsesFreshTagNotVerbatim) const std::string staging_key = "p/staging/mount1/ccc.tmp"; /// The staging object holds `[header][payload]` (as `writeFile` now emits it). The staging header is - /// a fixed 256-byte CABL envelope with its OWN incarnation_tag. + /// a fixed 512-byte CABL envelope with its OWN `incarnation_tag`. DB::Cas::EnvelopeHeader staging_h; staging_h.kind = DB::Cas::ObjectKind::Blob; staging_h.incarnation_tag = DB::UInt128(0xC0FFEE); /// the create-time tag diff --git a/src/Disks/tests/gtest_cas_server_root_format.cpp b/src/Disks/tests/gtest_cas_server_root_format.cpp index c2d1474dc29b..ef0d25dcf23c 100644 --- a/src/Disks/tests/gtest_cas_server_root_format.cpp +++ b/src/Disks/tests/gtest_cas_server_root_format.cpp @@ -16,7 +16,7 @@ TEST(CASFormatBattery, Owner) OwnerObject o; o.server_uuid = hexToU128("0123456789abcdeffedcba9876543210"); const String golden = currentFormatHeader("cas_owner") + - "{\"su\":\"0123456789abcdeffedcba9876543210\"}\n"; + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\"}\n"; EXPECT_EQ(encodeOwner(o), golden); EXPECT_FALSE(decodeOwner(golden).retired_at_ms.has_value()); runFormatBattery({FormatId::Owner, @@ -43,7 +43,7 @@ TEST(CASFormatBattery, ServerEpoch) runFormatBattery({FormatId::ServerEpoch, [&] { return sealObject(FormatId::ServerEpoch, encodeServerEpoch(e)); }, [](std::string_view s) { decodeServerEpoch(std::string(openObject(FormatId::ServerEpoch, s))); }, - currentFormatHeader("cas_epoch") + "{\"nwe\":\"7\"}\n"}); + currentFormatHeader("cas_epoch") + "{\"next_writer_epoch\":\"7\"}\n"}); } TEST(CASFormatBattery, MountLease) @@ -55,8 +55,8 @@ TEST(CASFormatBattery, MountLease) [&] { return sealObject(FormatId::MountLease, encodeMountLease(m)); }, [](std::string_view s) { decodeMountLease(std::string(openObject(FormatId::MountLease, s))); }, currentFormatHeader("cas_mount_lease") + - "{\"su\":\"0123456789abcdeffedcba9876543210\",\"we\":\"7\",\"hn\":\"host-1\",\"pid\":4242," - "\"sat\":1752537600000,\"seq\":\"5\",\"eat\":1752537630000,\"ma\":\"9\",\"fen\":false," + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"writer_epoch\":\"7\",\"hostname\":\"host-1\",\"process_id\":4242," + "\"started_at_ms\":1752537600000,\"sequence\":\"5\",\"expires_at_ms\":1752537630000,\"min_active\":\"9\",\"gc_fenced\":false," "\"write_attempt_id\":\"00112233445566778899aabbccddeeff\"}\n"}); } @@ -85,8 +85,8 @@ TEST(CASMountLeaseFormat, WriteAttemptIdIsRequiredAndCanonical) EXPECT_EQ(decodeMountLease(encoded).write_attempt_id, m.write_attempt_id); const String without_attempt_id = currentFormatHeader("cas_mount_lease") + - "{\"su\":\"0123456789abcdeffedcba9876543210\",\"we\":\"7\",\"hn\":\"\",\"pid\":0," - "\"sat\":0,\"seq\":\"0\",\"eat\":0,\"ma\":\"0\",\"fen\":false}\n"; + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"writer_epoch\":\"7\",\"hostname\":\"\",\"process_id\":0," + "\"started_at_ms\":0,\"sequence\":\"0\",\"expires_at_ms\":0,\"min_active\":\"0\",\"gc_fenced\":false}\n"; try { decodeMountLease(without_attempt_id); @@ -101,8 +101,8 @@ TEST(CASMountLeaseFormat, WriteAttemptIdIsRequiredAndCanonical) TEST(CASMountLeaseFormat, ZeroWriteAttemptIdIsRejected) { const String data = currentFormatHeader("cas_mount_lease") + - "{\"su\":\"0123456789abcdeffedcba9876543210\",\"we\":\"7\",\"hn\":\"\",\"pid\":0," - "\"sat\":0,\"seq\":\"0\",\"eat\":0,\"ma\":\"0\",\"fen\":false," + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"writer_epoch\":\"7\",\"hostname\":\"\",\"process_id\":0," + "\"started_at_ms\":0,\"sequence\":\"0\",\"expires_at_ms\":0,\"min_active\":\"0\",\"gc_fenced\":false," "\"write_attempt_id\":\"00000000000000000000000000000000\"}\n"; try { @@ -130,9 +130,9 @@ TEST(CASMountLeaseFormat, UnknownFieldsRemainTolerated) TEST(CASMountLeaseFormat, RejectsMissingIdentityFields) { - const String header = "{\"type\":\"cas_mount_lease\",\"v\":3}\n"; - const String fields = "\"hn\":\"host-1\",\"pid\":4242,\"sat\":1752537600000," - "\"seq\":\"5\",\"eat\":1752537630000,\"ma\":\"9\",\"fen\":false}"; + const String header = "{\"type\":\"cas_mount_lease\",\"version\":3}\n"; + const String fields = "\"hostname\":\"host-1\",\"process_id\":4242,\"started_at_ms\":1752537600000," + "\"sequence\":\"5\",\"expires_at_ms\":1752537630000,\"min_active\":\"9\",\"gc_fenced\":false}"; const auto expectCorrupted = [](const String & data) { @@ -147,6 +147,6 @@ TEST(CASMountLeaseFormat, RejectsMissingIdentityFields) } }; - expectCorrupted(header + R"({"we":"7",)" + fields + "\n"); - expectCorrupted(header + R"({"su":"0123456789abcdeffedcba9876543210",)" + fields + "\n"); + expectCorrupted(header + R"({"writer_epoch":"7",)" + fields + "\n"); + expectCorrupted(header + R"({"server_uuid":"0123456789abcdeffedcba9876543210",)" + fields + "\n"); } diff --git a/src/Disks/tests/gtest_cas_text_format.cpp b/src/Disks/tests/gtest_cas_text_format.cpp index 4371afb3f9e8..23a4ced88580 100644 --- a/src/Disks/tests/gtest_cas_text_format.cpp +++ b/src/Disks/tests/gtest_cas_text_format.cpp @@ -189,7 +189,7 @@ TEST(CASTextHeader, WriteExpectSniffGate) CasJsonWriter out; writeHeaderLine(out, FormatId::PoolMeta); const String rendered = std::move(out).take(); - EXPECT_EQ(rendered, fmt::format("{{\"type\":\"cas_pool_meta\",\"v\":{}}}\n", currentCompatibilityVersion())); + EXPECT_EQ(rendered, fmt::format("{{\"type\":\"cas_pool_meta\",\"version\":{}}}\n", currentCompatibilityVersion())); DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); const TextHeader h = expectHeaderLine(in, FormatId::PoolMeta); @@ -205,14 +205,14 @@ TEST(CASTextHeader, WriteExpectSniffGate) /// wrong type -> CORRUPTED_DATA; future v -> UNKNOWN_FORMAT_VERSION /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes /// the header gate, which is the point — the BODY is what has to fail here. - const String wrong = "{\"type\":\"cas_owner\",\"v\":3}\n"; + const String wrong = "{\"type\":\"cas_owner\",\"version\":3}\n"; DB::ReadBufferFromMemory in2(wrong.data(), wrong.size()); expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { expectHeaderLine(in2, FormatId::PoolMeta); }); - const String future = fmt::format("{{\"type\":\"cas_pool_meta\",\"v\":{}}}\n", currentCompatibilityVersion() + 1); + const String future = fmt::format("{{\"type\":\"cas_pool_meta\",\"version\":{}}}\n", currentCompatibilityVersion() + 1); DB::ReadBufferFromMemory in3(future.data(), future.size()); expectCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, [&] { expectHeaderLine(in3, FormatId::PoolMeta); }); - const String out_of_range = "{\"type\":\"cas_pool_meta\",\"v\":4294967299}\n"; + const String out_of_range = "{\"type\":\"cas_pool_meta\",\"version\":4294967299}\n"; DB::ReadBufferFromMemory in4(out_of_range.data(), out_of_range.size()); expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { expectHeaderLine(in4, FormatId::PoolMeta); }); } @@ -221,7 +221,7 @@ TEST(CASTextLines, ReadLineAndTrailer) { CasJsonWriter out; writeTrailerLine(out, 42); - EXPECT_EQ(std::move(out).take(), "{\"n\":42}\n"); + EXPECT_EQ(std::move(out).take(), "{\"record_count\":42}\n"); const String two = "abc\ndef\n"; DB::ReadBufferFromMemory in(two.data(), two.size()); @@ -247,13 +247,13 @@ TEST(CASZstdArm, SealOpenPolicyAndCaps) /// `changePoints` at decode time yet -- the gate is `v > G_BUILD` alone. Once a per-class floor is /// wired in, this literal must move to `G_BUILD`; the test's subject is the truncated BODY, not the /// version. - const String small = "{\"type\":\"cas_ref_snap\",\"v\":3}\n{}\n"; + const String small = "{\"type\":\"cas_ref_snap\",\"version\":3}\n{}\n"; const String sealed_small = sealObject(FormatId::RefSnapshot, small); ASSERT_TRUE(looksZstd(sealed_small)); EXPECT_EQ(openObject(FormatId::RefSnapshot, sealed_small), small); EXPECT_EQ(openObject(FormatId::RefSnapshot, small), small); - String big = "{\"type\":\"cas_ref_snap\",\"v\":3}\n{\"pad\":\""; + String big = "{\"type\":\"cas_ref_snap\",\"version\":3}\n{\"pad\":\""; big += String(8192, 'a'); big += "\"}\n"; const String sealed = sealObject(FormatId::RefSnapshot, big); diff --git a/src/Disks/tests/gtest_cas_wire_vocab.cpp b/src/Disks/tests/gtest_cas_wire_vocab.cpp index efbe3da7a0ae..fa77628a9df9 100644 --- a/src/Disks/tests/gtest_cas_wire_vocab.cpp +++ b/src/Disks/tests/gtest_cas_wire_vocab.cpp @@ -29,7 +29,7 @@ TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) closeObject(out, first); const String rendered = std::move(out).take(); EXPECT_EQ(rendered, - R"({"tt":"etag","tv":"etag-abc\"x","ha":"ch128","h":"00112233445566778899aabbccddeeff"})"); + R"({"token_type":"etag","token_value":"etag-abc\"x","hash_algorithm":"ch128","hash":"00112233445566778899aabbccddeeff"})"); DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); diff --git a/tests/integration/test_cas_gc_sharded/test.py b/tests/integration/test_cas_gc_sharded/test.py index a7c56edf2705..4c3a8d7dce79 100644 --- a/tests/integration/test_cas_gc_sharded/test.py +++ b/tests/integration/test_cas_gc_sharded/test.py @@ -122,13 +122,13 @@ def get_rustfs_object(key): # `gc/state`'s wire format is a plain JSON-like text object (CasGcStateFormat.cpp), not a binary -# blob: the two fields this test needs are literally spelled `"sg":""` (snap_generation) -# and `"sa":""` (snap_attempt) in the object bytes, so a direct regex read is exact without +# blob: the two fields this test needs are literally spelled `"snapshot_generation":""` +# and `"snapshot_attempt":""` in the object bytes, so a direct regex read is exact without # needing the C++ decoder. This mirrors the production reader that resolves "the adopted seal" # (Gc/CasOrphanManifestSweep.cpp): read gc/state, take (snap_generation, snap_attempt), then look up # that exact fold seal -- the only two-hop lookup that names one authoritative adopted pair. -_SNAP_GENERATION_RE = re.compile(r'"sg":"(\d+)"') -_SNAP_ATTEMPT_RE = re.compile(r'"sa":"(\d+)"') +_SNAP_GENERATION_RE = re.compile(r'"snapshot_generation":"(\d+)"') +_SNAP_ATTEMPT_RE = re.compile(r'"snapshot_attempt":"(\d+)"') def read_adopted_generation_and_attempt(): diff --git a/tests/integration/test_cas_gcs/gcs_mocks/server.py b/tests/integration/test_cas_gcs/gcs_mocks/server.py index 74294cf95fa5..2ce6a34fef68 100644 --- a/tests/integration/test_cas_gcs/gcs_mocks/server.py +++ b/tests/integration/test_cas_gcs/gcs_mocks/server.py @@ -784,7 +784,10 @@ def handle_control(path, method, query): return _no_such_key(meta_key) text = entry["body"].decode("utf-8", "strict") rewritten, replacements = re.subn( - r'"st":"clean","cr":"[0-9]+"', '"st":"condemned","cr":"1"', text, count=1 + r'"state":"clean","condemn_round":"[0-9]+"', + '"state":"condemned","condemn_round":"1"', + text, + count=1, ) if replacements != 1: return _bad_request("blob metadata is not Clean: " + meta_key) diff --git a/tests/integration/test_cas_gcs/test.py b/tests/integration/test_cas_gcs/test.py index 285da574ac17..7df52e4dbbd3 100644 --- a/tests/integration/test_cas_gcs/test.py +++ b/tests/integration/test_cas_gcs/test.py @@ -469,7 +469,7 @@ def test_blob_publication_request_budget_and_default_mode(disk): for r in meta if r["method"] == "PUT" and r["headers"].get("x-goog-if-generation-match") == "0" - and '"st":"clean"' in r["request_body"] + and '"state":"clean"' in r["request_body"] ] assert len(creates) == 1, (key, meta) @@ -524,7 +524,7 @@ def test_blob_publication_request_budget_and_default_mode(disk): for r in _meta_requests(retry, target) if r["method"] == "PUT" and r["headers"].get("x-goog-if-generation-match", "0") != "0" - and '"st":"clean"' in r["request_body"] + and '"state":"clean"' in r["request_body"] ] assert len(clean_cas) == 1, clean_cas diff --git a/tests/integration/test_cas_mount_renewal_retry/test.py b/tests/integration/test_cas_mount_renewal_retry/test.py index 31fa1529c0c3..af6629a14656 100644 --- a/tests/integration/test_cas_mount_renewal_retry/test.py +++ b/tests/integration/test_cas_mount_renewal_retry/test.py @@ -104,7 +104,7 @@ def _decode_mount(body): lines = body.decode().splitlines() assert len(lines) == 2, lines header = json.loads(lines[0]) - assert header["type"] == "cas_mount_lease" and int(header["v"]) > 0, header + assert header["type"] == "cas_mount_lease" and int(header["version"]) > 0, header return json.loads(lines[1]) @@ -226,7 +226,7 @@ def recovered_snapshot(): assert mount_after["state"] == "live", mount_after assert mount_after["lifecycle"] == "live", mount_after assert mount_after["gc_fenced"] == 0, mount_after - assert int(mount_body["seq"]) == sequence + assert int(mount_body["sequence"]) == sequence assert token_after != token_before assert stats["faults"] == 1, stats assert stats["by_mode"].get("503") == 1, stats @@ -311,7 +311,7 @@ def resolved_snapshot(): assert record["upstream_etag"].strip('"') == token_after, record assert body_after != body_before assert token_after != token_before - assert int(mount_body["seq"]) == sequence + assert int(mount_body["sequence"]) == sequence assert delta["CASMountRenewalAttempts"] == 1, delta assert delta["CASMountRenewalRetries"] == 0, delta diff --git a/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh b/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh index ef5e2377cc21..4abc0ea5412a 100755 --- a/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh +++ b/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh @@ -11,7 +11,7 @@ # SQL-level drop completed normally while the CAS catalog row leaked, one per create/drop cycle. # # The primary oracle is the pool's OWN plain-text `cas/ref_catalog` object, read directly off disk: the -# exact `st` (lifecycle) field recorded for the table's logical namespace. `SYSTEM CAS FSCK`'s +# exact `state` field recorded for the table's logical namespace. `SYSTEM CAS FSCK`'s # unreachable/dangling counts are a secondary check only -- fsck correctly regards a `live` leak as # CONSISTENT (nothing is unreachable; the row simply never dies), so it cannot detect this defect on its # own; `04290_cas_no_leftovers.sh`'s fsck-only oracle is exactly why FINDING #2 shipped unnoticed. @@ -31,10 +31,10 @@ CATALOG_FILE="${POOL_DIR}/ca/cas/ref_catalog" # The pool's own plain-text catalog line for namespace $1, or empty if the namespace has no row at all. catalog_line() { - grep -F "\"ns\":\"$1\"" "${CATALOG_FILE}" 2>/dev/null || true + grep -F "\"namespace\":\"$1\"" "${CATALOG_FILE}" 2>/dev/null || true } -# The `st` (lifecycle) word recorded for namespace $1: "live"/"creating"/"removing", or "absent" if the +# The `state` word recorded for namespace $1: "live"/"creating"/"removing", or "absent" if the # namespace has no catalog row (matches `04290`'s field-by-name discipline: never assume a position). catalog_state() { local line @@ -43,7 +43,7 @@ catalog_state() { echo "absent" return fi - echo "${line}" | grep -o '"st":"[a-z]*"' | head -1 | sed -E 's/"st":"([a-z]*)"/\1/' + echo "${line}" | grep -o '"state":"[a-z]*"' | head -1 | sed -E 's/"state":"([a-z]*)"/\1/' } # ClickHouse's own store// fanout with the CAS archive boundary marker, exactly as