Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/en/antalya/cas/architecture/blob-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ bytes, not inferred from occupancy alone as blob uploads do. Divergent bytes are
correct operation and fail closed as `CORRUPTED_DATA`.

This is the format-evolution **adoption pin**, documented in the persisted-format registry
(`Formats/README.md`): on a `putDeterministicArtifact` conflict, the writer re-encodes at the `v`
(`Formats/README.md`): on a `putDeterministicArtifact` conflict, the writer re-encodes at the `version`
of the *existing* object rather than at its own current build's version, so two writers on
different builds replaying the same deterministic round still land on byte-identical output.

Expand Down
10 changes: 5 additions & 5 deletions docs/en/antalya/cas/architecture/garbage-collection.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,30 +22,30 @@ race over one blob is covered on the
## Leadership {#leadership}

There is **no separate `GC` lease object**. The lease lives inside `gc/state` itself as
`{owner, seq}`.
`{lease_owner, lease_sequence}`.

```mermaid
stateDiagram-v2
[*] --> Reading: GET gc/state
Reading --> Creating: object absent, never observed before
Creating --> Leader: casPut create-if-absent, cas_gc_shards fixed here, once
Reading --> Renewing: lease owner is me
Renewing --> Leader: casPut seq+1, guarded by the observed token
Renewing --> Leader: casPut lease_sequence+1, guarded by the observed token
Reading --> Evaluating: foreign owner
Evaluating --> NotLeader: incumbent lease moved, or heartbeat moved, or steal not allowed
Evaluating --> Stealing: both frozen across a full observation window
Stealing --> Leader: casPut owner=me seq+1, on the observed token
Stealing --> Leader: casPut lease_owner=me lease_sequence+1, on the observed token
Stealing --> NotLeader: lost the CAS, re-read and re-arm
Leader --> [*]: run the round
```

Two independent liveness signals are consulted before a steal: whether `(owner, seq)` moved since
Two independent liveness signals are consulted before a steal: whether `(lease_owner, lease_sequence)` moved since
the last tick, and whether the separate `gc/hb` heartbeat moved. The heartbeat is compared only
under the same remembered heartbeat owner, deliberately not against `lease.owner` — a deposed
leader's heartbeat thread keeps pulsing, and that must not cause a live new leader's lease to be
stolen. The paced background loop may steal; a manual `SYSTEM CAS GC RUN` may not, because the
safety argument needs two observations separated by real wall time. Because every renew or steal
bumps `seq`, `seq` doubles as the round's attempt id.
bumps `lease_sequence`, `lease_sequence` doubles as the round's attempt id.

**A deposed leader that keeps running cannot corrupt anything**, and the argument does not rely on
exclusivity at all:
Expand Down
6 changes: 3 additions & 3 deletions docs/en/antalya/cas/architecture/mounts-and-leases.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Two failure modes this closes:

One object, `gc/server-roots/<server_root_id>/mount`, carries **both** the liveness lease and the build
watermark — there is no separate watermark object. `MountLease` fields: `server_uuid`,
`writer_epoch`, `write_attempt_id`, `hostname`, `pid`, `started_at_ms`, renewal `seq`,
`writer_epoch`, `write_attempt_id`, `hostname`, `process_id`, `started_at_ms`, renewal `sequence`,
`expires_at_ms`, `min_active` (the build-watermark floor), and `gc_fenced`.

- **Logical renewal identity.** Each holder-originated body has a fresh nonzero
Expand Down Expand Up @@ -165,8 +165,8 @@ the claim outcomes above and is shown here as behavior, not as a type in the cod
```mermaid
stateDiagram-v2
[*] --> Absent
Absent --> Live: claimMount putIfAbsent, seq=1
Live --> Live: keeper beat, putOverwrite seq+1
Absent --> Live: claimMount putIfAbsent, sequence=1
Live --> Live: keeper beat, putOverwrite sequence+1
Live --> Fenced: GC observes a stable token past threshold, gc_fenced=1, body preserved
Live --> Terminated: certified drain, terminal farewell (expires_at=now, min_active=MAX)
Fenced --> Live: same-uuid claim with a fresh writer_epoch, instant reclaim
Expand Down
2 changes: 1 addition & 1 deletion docs/en/antalya/cas/architecture/namespaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Each row (`CatalogEntry`) carries:

| Field | Meaning |
|---|---|
| `ns` | The namespace name |
| `namespace` | The namespace name |
| `state` | `Creating`, `Live`, or `Removing` — see below |
| `incarnation` | The `life_id` for this row, nonzero, never reused |
| `creator` | The mounted writer's fence identity (server root, writer epoch, admission fence generation) that is creating this row — **required** iff `state == Creating`, **forbidden** otherwise |
Expand Down
10 changes: 5 additions & 5 deletions docs/en/antalya/cas/architecture/storage-layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,24 +63,24 @@ touches — it is reclaimed only by its own server's next mount.
Every persisted CAS metadata object is text: a header line, a body, and an optional trailer.

```
{"type":"cas_<object>","v":N} <- header line, always present
{"type":"cas_<object>","version":N} <- header line, always present
<body> <- one JSON object, sorted NDJSON records,
or a descriptor + raw payload zone
{"n":…} <- optional trailer (record/entry count)
{"record_count":…} <- optional trailer (record/entry count)
```

`v` is the only version field; a reader rejects `v` above what the build supports with
`version` is the only version field; a reader rejects `version` above what the build supports with
`UNKNOWN_FORMAT_VERSION`, checked before the body. A `.zst` key suffix means, exactly, that the
object kind's compression policy is `Always`: the object is stored as one zstd frame with the
checksum flag on, and its declared content size is checked against a per-kind cap before
allocation. Always-small and deterministic kinds (`cas_ref_ckpt`, `cas_blob_meta`, `cas_fold_seal`,
`cas_run`, …) are stored raw, with no `.zst` suffix.

The blob envelope is a special case of the header/body shape: a JSON descriptor padded with ASCII
spaces to a pool-constant `blob_header_len` (256 bytes, a `cas_pool_meta` field), terminated by
spaces to the pool-constant `blob_header_length` (384 bytes by default, a `cas_pool_meta` field), terminated by
`\n`, so the raw payload always starts at that fixed offset with no header parse needed to locate
it. The part manifest is the other `PayloadHybrid` kind: text header, descriptor, sorted NDJSON
entry records, `{"n":…}` trailer, then a banner-framed raw payload zone for small inline file
entry records, `{"record_count":…}` trailer, then a banner-framed raw payload zone for small inline file
bytes.

## Codec table {#codec-table}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ ProvenanceOp opFromWord(std::string_view w)
if (w == "mutation") return ProvenanceOp::Mutation;
if (w == "attach") return ProvenanceOp::Attach;
if (w == "repack") return ProvenanceOp::Repack;
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: unknown op '{}'", w);
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: unknown operation '{}'", w);
}

/// The escaped byte-length of one raw ref char under the frozen envelope alphabet (see writeEnvelopeRefField).
Expand Down Expand Up @@ -109,30 +109,30 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len)
CasJsonWriter buf(256);
bool first = true;
writeKey(buf, "type", first); writeStringValue(buf, kBlobType);
writeKey(buf, "v", first); writeIntText(currentCompatibilityVersion(), buf);
writeKey(buf, "tag", first); writeHex128Value(buf, header.incarnation_tag);
writeKey(buf, "bld", first); writeHex128Value(buf, header.build_id);
writeKey(buf, "version", first); writeIntText(currentCompatibilityVersion(), buf);
writeKey(buf, "incarnation_tag", first); writeHex128Value(buf, header.incarnation_tag);
writeKey(buf, "build_id", first); writeHex128Value(buf, header.build_id);
if (header.provenance)
{
writeKey(buf, "ts", first); writeIntText(header.provenance->created_at_ms, buf);
writeKey(buf, "by", first); writeHex128Value(buf, header.provenance->creator_server_id);
writeKey(buf, "op", first); writeStringValue(buf, opToWord(header.provenance->op));
writeKey(buf, "ch", first); writeIntText(header.provenance->ch_version, buf);
writeKey(buf, "created_at_ms", first); writeIntText(header.provenance->created_at_ms, buf);
writeKey(buf, "creator_server_id", first); writeHex128Value(buf, header.provenance->creator_server_id);
writeKey(buf, "operation", first); writeStringValue(buf, opToWord(header.provenance->op));
writeKey(buf, "clickhouse_version", first); writeIntText(header.provenance->ch_version, buf);
}
/// Test-only critical extension: an unknown `!`-key BEFORE `ref`.
if (header.emit_unknown_critical_key)
{
writeKey(buf, "!x", first); writeStringValue(buf, "1");
}
json = std::move(buf).take(); /// e.g. {"type":"cas_blob","v":3,...,"ch":26006001 (no ref, no closing brace)
json = std::move(buf).take(); /// e.g. {"type":"cas_blob","version":3,...,"clickhouse_version":26006001
}

/// Optional `ref`, truncated to the exact remaining budget. Layout after this block:
/// json + `,"ref":` + `"` + <escaped ref, <= budget bytes> + `"` + `}` must be <= blob_header_len-1
/// json + `,"intended_ref":` + `"` + <escaped ref, <= budget bytes> + `"` + `}` must fit
/// (byte blob_header_len-1 is reserved for '\n'; the pad zone fills the gap with spaces).
if (header.intended_ref)
{
static constexpr std::string_view ref_key = ",\"ref\":";
static constexpr std::string_view ref_key = ",\"intended_ref\":";
/// +3 = opening quote + closing quote + closing brace.
const size_t fixed = json.size() + ref_key.size() + 3;
if (blob_header_len < 1 || fixed > static_cast<size_t>(blob_header_len) - 1)
Expand Down Expand Up @@ -181,45 +181,45 @@ EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*obje
"CAS blob envelope: object is a '{}', not a '{}'", t, kBlobType);
saw_type = true;
}
else if (key == "v")
else if (key == "version")
{
h.compatibility_version = r.readU32Number();
checkCompatibility(h.compatibility_version, "blob envelope");
saw_v = true;
}
else if (key == "tag")
else if (key == "incarnation_tag")
h.incarnation_tag = r.readHex128();
else if (key == "bld")
else if (key == "build_id")
h.build_id = r.readHex128();
else if (key == "ts")
else if (key == "created_at_ms")
{
prov.created_at_ms = r.readU64Number();
have_prov = true;
}
else if (key == "by")
else if (key == "creator_server_id")
{
prov.creator_server_id = r.readHex128();
have_prov = true;
}
else if (key == "op")
else if (key == "operation")
{
prov.op = opFromWord(r.readString());
have_prov = true;
}
else if (key == "ch")
else if (key == "clickhouse_version")
{
prov.ch_version = static_cast<uint32_t>(r.readU64Number());
have_prov = true;
}
else if (key == "ref")
else if (key == "intended_ref")
h.intended_ref = r.readString();
else
r.skipUnknown(key); /// `!`-key -> UNKNOWN_FORMAT_VERSION; unknown plain key -> skipped (tolerant)
}
if (!saw_type)
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: missing type");
if (!saw_v)
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: missing v");
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: missing version");
if (h.kind != expected_kind)
throw Exception(ErrorCodes::CORRUPTED_DATA,
"CAS blob envelope: kind {} does not match expected {}",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ struct Provenance
/// The blob envelope is a fixed-size JSON descriptor followed by the raw payload. The JSON object
/// occupies bytes [0, json_len), ASCII spaces occupy [json_len, blob_header_len-1), and '\n' is at byte
/// blob_header_len-1. The payload therefore begins at the pool-wide constant offset
/// `blob_header_len` (256 for blob pools, a `PoolMeta` parameter), allowing the locate path to use a
/// `blob_header_len` (384 by default for blob pools, a `PoolMeta` parameter), allowing the locate path to use a
/// constant shift without reading an object-specific header first. The header is also the incarnation
/// zone: it may differ between incarnations of one logical object, and each upload attempt gets a fresh
/// random u128 `tag`, which is used as the exact-token delete identity.
Expand All @@ -73,7 +73,7 @@ struct EnvelopeHeader
bool emit_unknown_critical_key = false;
};

/// Builds the fixed-length header for a pool whose `blob_header_len` is `blob_header_len` (256 for blob
/// Builds the fixed-length header for a pool whose `blob_header_len` is `blob_header_len` (384 by default for blob
/// pools). Sets `header.header_len = blob_header_len` and returns exactly that many bytes. The
/// diagnostic `ref` is the only truncatable field and is shortened, never dropped, when necessary to
/// preserve the fixed layout. The header is built without payload bytes, so an upload can stage the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,11 @@ String encodeBlobMeta(const BlobMeta & meta)
// `version` is represented by the header line. The JSON body contains only fields that describe
// the current marker and its accounting data.
bool first = true;
writeKey(out, "st", first);
writeKey(out, "state", first);
writeStringValue(out, metaStateToWord(meta.state));
writeKey(out, "cr", first);
writeKey(out, "condemn_round", first);
writeU64StringValue(out, meta.condemn_round);
writeKey(out, "sz", first);
writeKey(out, "size", first);
writeU64StringValue(out, meta.size);
closeObject(out, first);
writeChar('\n', out);
Expand All @@ -72,20 +72,20 @@ BlobMeta decodeBlobMeta(std::string_view bytes)
String key;
while (r.nextKey(key))
{
if (key == "st")
if (key == "state")
{
m.state = metaStateFromWord(r.readString());
saw_state = true;
}
else if (key == "cr")
else if (key == "condemn_round")
m.condemn_round = r.readU64String();
else if (key == "sz")
else if (key == "size")
m.size = r.readU64String();
else
r.skipUnknown(key);
}
if (!saw_state)
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: missing st");
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: missing state");
if (!body_in.eof() || !in.eof())
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: trailing bytes");
return m;
Expand Down
Loading
Loading