Skip to content

Commit 2bc5ae7

Browse files
dougqhclaude
andcommitted
Encode tag ids with a colored slot + one-long dense presence
Reframe of the dense-store presence layer. Replaces the earlier two-tier (group-decl mask + field-decl bloom) design with a single global colored slot: the tag-id middle 16 bits carry one graph-colored slot coordinate (SLOT_SHIFT=32, SLOT_MASK=0xFFFF), so the dense store tracks presence with one occupancy long instead of a group mask plus a field bloom. Adds the trace-level bit (LEVEL_TRACE) and level-bit read-through in the parent visibility check. KnownTags remains hand-maintained here (src/main); the tag-registry code generator that produces these colored ids lands in the following commit, which relocates the file to src/generated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent d758efe commit 2bc5ae7

6 files changed

Lines changed: 782 additions & 357 deletions

File tree

‎internal-api/src/main/java/datadog/trace/api/KnownTagCodec.java‎

Lines changed: 96 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -22,23 +22,33 @@ public static boolean isActive() {
2222
}
2323

2424
/*
25-
* tagId bit layout: [63 intercepted] [62-48 globalSerial (15 bits)] [47-32 fieldPos]
26-
* [31-0 nameHash]. Bit 63 (the sign bit) marks a tag the tag interceptor must see, so the check
27-
* is a single {@code tagId < 0}. globalSerial is globally unique per known tag; fieldPos is its
28-
* slot in the global positional layout (TagMap.knownEntries index); nameHash is
29-
* TagMap.Entry#_hash(name) and is layout-independent. Unknown (string-only) tags have the upper
30-
* 32 bits zero. NOTE: TagMap.Entry decodes nameHash inline as (int) tagId on its hot path, so the
31-
* low-32 encoding here must stay in sync with that.
25+
* tagId bit layout: [63 intercepted] [62-48 globalSerial (15 bits)] [47-32 slot (16 bits)] [31-0
26+
* reserved, zero]. Bit 63 (the sign bit) marks a tag the tag interceptor must see, so the check is
27+
* a single {@code tagId < 0}. globalSerial is globally unique per known tag. The middle 16 bits
28+
* carry the tag's SLOT: one globally stable coordinate assigned by graph-coloring the tag
29+
* co-occurrence graph (the resolved tag set of each concrete span type, plus the trace-level
30+
* tier, is a clique). Co-occurring tags always get distinct slots; slots are reused only between
31+
* tags that never appear together, so slotCount stays bounded by the largest clique (≤ 64) and
32+
* fits one {@code long} occupancy mask — the dense store's single-tier presence fast path (see
33+
* {@link TagMap}). The low 32 bits are unused for known ids (the whole id is fully determined by
34+
* serial + slot, so the generator can emit a literal). The low 32 bits are being carved for
35+
* cross-cutting flags; bit 2 is the trace/span LEVEL bit (set ⟹ trace-level), and bits 1-0 are
36+
* reserved for the dd/otel applicability flags that land with increment 1. The level bit lets
37+
* read-through skip the shadow check across the trace/span boundary — trace and span tags reuse
38+
* the same slots, so occupancy alone can't tell them apart, but a span map (no trace-level tags)
39+
* can never shadow a trace-level ancestor entry (see {@link TagMap}). Unknown (string-only) custom
40+
* tags are NOT known ids — they key off {@code TagMap.Entry#_hash(name)} in their own bucket path
41+
* and never enter here.
3242
*/
33-
public static int globalSerial(long tagId) {
43+
public static int serialNum(long tagId) {
3444
return (int) ((tagId >>> 48) & 0x7FFF);
3545
}
3646

3747
/**
38-
* Flag bit (the sign bit) marking a tag the tag interceptor must process — reserved/"virtual"
39-
* tags AND intercepted-but-stored tags (e.g. http.method, which the interceptor side-effects and
40-
* also stores). Encoded in the id so {@code DDSpanContext.setTag(long)} can route with a single
41-
* sign test ({@link #isIntercepted}) instead of resolving the name. Non-intercepted tags (peer.*,
48+
* Flag bit (the sign bit) marking a tag the tag interceptor must process — reserved tags AND
49+
* intercepted-but-stored tags (e.g. http.method, which the interceptor side-effects and also
50+
* stores). Encoded in the id so {@code DDSpanContext.setTag(long)} can route with a single sign
51+
* test ({@link #isIntercepted}) instead of resolving the name. Non-intercepted tags (peer.*,
4252
* base.service, …) leave it clear and take the fast store path. Must agree with the interceptor's
4353
* name-based {@code needsIntercept} for every assigned id.
4454
*/
@@ -54,78 +64,115 @@ public static long intercepted(long tagId) {
5464
return tagId | INTERCEPTED;
5565
}
5666

57-
public static int fieldPos(long tagId) {
58-
return (int) ((tagId >>> 32) & 0xFFFF);
67+
/**
68+
* Trace/span LEVEL bit (low-32 carve, bit 2). Set marks a trace-level tag (lives on the
69+
* TraceSegment's own TagMap); clear marks a span-level tag. Trace and span tags reuse the same
70+
* coloring slots, so this bit is what lets read-through tell the two levels apart — a span map
71+
* (no trace-level tags) can never shadow a trace-level ancestor entry, so its shadow check is
72+
* skipped (see {@link TagMap#parentDenseVisible}).
73+
*/
74+
public static final long LEVEL_TRACE = 1L << 2;
75+
76+
/** True if the tagId names a trace-level tag. */
77+
public static boolean isTraceLevel(long tagId) {
78+
return (tagId & LEVEL_TRACE) != 0L;
5979
}
6080

61-
public static int nameHash(long tagId) {
62-
return (int) tagId;
81+
/** Returns the tagId with the {@link #LEVEL_TRACE} flag set. */
82+
public static long traceLevel(long tagId) {
83+
return tagId | LEVEL_TRACE;
6384
}
6485

86+
// The middle 16 bits [47-32] hold the tag's SLOT: one globally stable coordinate from graph
87+
// coloring the co-occurrence graph. Co-occurring tags get distinct slots and slotCount stays
88+
// bounded by the largest clique (<= 64), so the dense store's presence fast path is a single
89+
// occupancy long (1L << slot); a clear bit proves the tag absent and enables an O(1) append. See
90+
// TagMap's dense-store fast path.
91+
static final int SLOT_SHIFT = 32;
92+
static final int SLOT_MASK = 0xFFFF; // 16 bits
93+
6594
/**
66-
* globalSerial partition. {@code [1, FIRST_STORED_SERIAL)} is reserved for "virtual" tags that
67-
* are specially handled (redirected to span fields or processed by the tag interceptor) and are
68-
* NOT stored in the TagMap — these are hand-assigned in tracer core. {@code [FIRST_STORED_SERIAL,
69-
* ..]} is for generated convention tags that ARE stored (slotted/bucketed). {@code globalSerial
70-
* == 0} means unknown / string-only. Both core and the code generator must agree on this
71-
* boundary.
95+
* The tag's slot: its globally stable coloring coordinate, or {@link #NO_SLOT} when it has none
96+
* (reserved or deliberately bucket-only). Drives the dense store's single occupancy mask.
97+
*/
98+
public static int slot(long tagId) {
99+
return (int) ((tagId >>> SLOT_SHIFT) & SLOT_MASK);
100+
}
101+
102+
/**
103+
* globalSerial partition. {@code [1, FIRST_STORED_SERIAL)} is the RESERVED tier and {@code
104+
* [FIRST_STORED_SERIAL, ..]} is the STORED tier; {@code globalSerial == 0} means unknown /
105+
* string-only. Both core and the code generator must agree on this boundary.
106+
*
107+
* <p><b>Reserved</b> is the shared mechanism: the tracer reserves the key and handles it itself
108+
* instead of putting it in the TagMap. It says nothing about whether a value exists — that splits
109+
* into two kinds (the {@code kind:} in the overlay):
110+
*
111+
* <ul>
112+
* <li><b>structural</b> — the value <i>does</i> exist, it just lives in a first-class
113+
* span/trace field (service, resource.name, error, span.type, origin), not the tag map.
114+
* <li><b>directive</b> — there is <i>no</i> stored value; the key is a command that triggers
115+
* trace behavior (sampling.priority, manual.keep, measured).
116+
* </ul>
117+
*
118+
* "virtual" over-claims non-existence (wrong for structural) and "built-in" over-claims existence
119+
* (wrong for directive), so the tier is named for the mechanism they share: reserved. These are
120+
* hand-assigned in the overlay. <b>Stored</b> tags are the generated convention tags that ARE put
121+
* in the map (slotted/bucketed).
72122
*/
73123
public static final int FIRST_STORED_SERIAL = 256;
74124

75-
/** True if the tagId names a reserved "virtual"/specially-handled tag (not stored in the map). */
125+
/** True if the tagId names a reserved (structural/directive) tag — handled, not stored. */
76126
public static boolean isReserved(long tagId) {
77-
int globalSerial = globalSerial(tagId);
78-
return globalSerial > 0 && globalSerial < FIRST_STORED_SERIAL;
127+
int serialNum = serialNum(tagId);
128+
return serialNum > 0 && serialNum < FIRST_STORED_SERIAL;
79129
}
80130

81131
/** True if the tagId names a generated, map-stored (slotted/bucketed) tag. */
82132
public static boolean isStored(long tagId) {
83-
return globalSerial(tagId) >= FIRST_STORED_SERIAL;
133+
return serialNum(tagId) >= FIRST_STORED_SERIAL;
84134
}
85135

86136
/**
87-
* Sentinel {@code fieldPos} meaning "no positional slot". It is the maximum value the 16-bit
88-
* fieldPos field can hold, so it always compares {@code >= slotCount()} and routes to the hash
89-
* buckets rather than the fast positional array. Two kinds of tagId use it:
137+
* Sentinel {@code slot} meaning "no positional slot". It is the maximum value the 16-bit slot
138+
* field can hold, so it always compares {@code >= slotCount()} and routes to the hash buckets
139+
* rather than the fast positional array. Two kinds of tagId use it:
90140
*
91141
* <ul>
92-
* <li>Reserved/virtual tags ({@code globalSerial < FIRST_STORED_SERIAL}) — not stored at all;
93-
* the sentinel just guarantees an incidental store never lands in a slot.
142+
* <li>Reserved tags ({@code globalSerial < FIRST_STORED_SERIAL}) — not stored at all; the
143+
* sentinel just guarantees an incidental store never lands in a slot.
94144
* <li>Unslotted stored tags ({@code globalSerial >= FIRST_STORED_SERIAL}) — "low-priority" tags
95145
* that get a stable id (and so {@code keyOf}/{@code nameOf} unification with their string
96-
* form) but are deliberately not given a slot, so they live in the buckets and don't widen
97-
* {@code knownEntries[]} for every span. {@code getEntry(long)} for these resolves the name
98-
* and rehashes — the cost of not owning a slot.
146+
* form) but are deliberately not given a slot, so they live in the buckets. {@code
147+
* getEntry(long)} for these resolves the name and rehashes — the cost of not owning a slot.
99148
* </ul>
100149
*/
101-
public static final int NO_SLOT = 0xFFFF;
150+
public static final int NO_SLOT = SLOT_MASK; // slot all-ones sentinel (16 bits)
102151

103152
/**
104153
* True if the tagId names a stored tag that deliberately has no positional slot (bucket-only).
105154
*/
106155
public static boolean isUnslotted(long tagId) {
107-
return isStored(tagId) && fieldPos(tagId) == NO_SLOT;
156+
return isStored(tagId) && slot(tagId) == NO_SLOT;
108157
}
109158

110159
/**
111-
* Builds a tagId from its parts: {@code globalSerial} (globally unique per known tag), {@code
112-
* fieldPos} (the tag's slot within its span type's positional table), and the tag {@code name}
113-
* (whose hash is computed via the same function the runtime uses, so the low 32 bits match {@link
114-
* TagMap.Entry#hash()}). Inverse of {@link #globalSerial}/{@link #fieldPos}/{@link #nameHash}.
115-
* Intended for the code generator and tests.
160+
* Builds a tagId from its {@code serialNum} (globally unique per known tag) and {@code slot} (its
161+
* coloring coordinate, or {@link #NO_SLOT}). The low 32 bits are zero, so the id is fully
162+
* determined by these parts — the generator emits it as a literal. Inverse of {@link
163+
* #serialNum}/{@link #slot}. Intended for the code generator and tests.
116164
*/
117-
public static long tagId(int globalSerial, int fieldPos, String name) {
118-
long nameHash = TagMap.Entry._hash(name) & 0xFFFFFFFFL;
119-
return ((long) globalSerial << 48) | ((long) (fieldPos & 0xFFFF) << 32) | nameHash;
165+
public static long makeTagId(int serialNum, int slot) {
166+
return ((long) serialNum << 48) | ((long) (slot & SLOT_MASK) << SLOT_SHIFT);
120167
}
121168

122169
/**
123-
* Builds a tagId with no positional slot ({@code fieldPos == }{@link #NO_SLOT}). Use for reserved
124-
* "virtual" tags and for "low-priority" stored tags that get a stable id but are intentionally
125-
* kept out of the fast slot array (they route to the hash buckets). See {@link #NO_SLOT}.
170+
* Builds a tagId with no positional slot ({@code slot == }{@link #NO_SLOT}). Use for reserved
171+
* tags and for "low-priority" stored tags that get a stable id but are intentionally kept out of
172+
* the fast slot array (they route to the hash buckets). See {@link #NO_SLOT}.
126173
*/
127-
public static long tagId(int globalSerial, String name) {
128-
return tagId(globalSerial, NO_SLOT, name);
174+
public static long makeTagId(int serialNum) {
175+
return makeTagId(serialNum, NO_SLOT);
129176
}
130177

131178
// Number of positional slots in the global layout = (max stored fieldPos) + 1, declared by the

0 commit comments

Comments
 (0)