Repository navigation
Conversation
Mirrors Go's BenchmarkParsePromProto: reads every entry with its accessor, then labels, start timestamp and all exemplars, over the Go createTestProtoBuf payload and testdata/expfmt/protobuf. The Layer 0 parsers are crate-internal, so a non-default bench-internals feature exposes the parse module, doc-hidden, to the benchmark only. Signed-off-by: Mateusz Kiersnowski <mateusz.kiersnowski@gmail.com>
ProtobufParser decoded each MetricFamily with prost, so every label name and value was an owned String, and labels() and exemplar() copied them again on every call. Replace the prost decode with a small zero-copy decoder for the io.prometheus.client messages, like Go's MetricStreamingDecoder: - MetricFamily::decode reads name, help, type and unit as &'a str and records each encoded metric, without decoding it. - Metric::decode decodes one metric when the parser reaches it, into buffers that are cleared and reused, so steady-state parsing does not allocate per metric. Exemplar labels share one buffer per metric. - labels() and exemplar() return Cow::Borrowed slices of the input. Only the summary _count/_sum metric names and the quantile label value are owned, because they are not in the input. Unlike Go's yoloString, the borrows are checked by the 'a lifetime and strings are still validated as UTF-8, with no unsafe code. Decoding one metric at a time also removes the old TODO: a malformed metric now fails only when the parser reaches it, after the earlier metrics of its family were returned, which matches Go. Decoding follows prost and gogo-protobuf: wrong wire types on known fields are errors, unknown fields and groups are skipped, repeated scalars may be packed or not, and repeated message fields are merged. Unit tests compare the decoder with prost on the same bytes. Closes prometheus#26 Signed-off-by: Mateusz Kiersnowski <mateusz.kiersnowski@gmail.com>
saswatamcode
left a comment
There was a problem hiding this comment.
Some comments from AI review.
The bench results seem great, I'm just not sure if we want to move to custom proto parser right at this stage.
| labels_end: labels.len(), | ||
| ..Self::default() | ||
| }); | ||
| if exemplar.labels_end != labels.len() { |
There was a problem hiding this comment.
Exemplar::merge keeps merged labels contiguous by copying the existing range to the end of the shared exemplar_labels vec and abandoning the original
entries. If a Metric's wire bytes alternate field 3 (counter{exemplar{label{}}}) and field 7 (histogram{bucket{exemplar{label{}}}}), every counter
merge sees labels_end != labels.len() and relocates, so round i copies i pairs. That's ~18 input bytes per round, so a 1 MB body gives ~58k rounds ≈
1.7e9 tuples — OOM/hang. prost grew linearly here because each Exemplar owned its own Vec.
Both encodings are valid protobuf, so this is reachable from a hostile or buggy target, not just a corrupt file. Cheapest fix is probably a dedicated
small buffer for the counter exemplar rather than the shared arena; a cap on exemplar_labels length would also do.
| self.metric_index = self.next_metric_index; | ||
| /// Go's `NextMetric`: decodes the next metric of the family, so a malformed metric only | ||
| /// fails when the parser reaches it. | ||
| fn next_metric(&mut self) -> Result<bool, ParseError> { |
There was a problem hiding this comment.
next_metric increments next_metric_index after self.metric.decode(message)?, and next_entry leaves state untouched on that error. A caller that logs
the ParseError and calls next_entry() again gets the same error forever. Before this PR the whole family was decoded in next_metric_family with
self.pos already advanced, so a retrying caller skipped ahead. No in-tree caller retries today (go_diff_tests::render and the bench both stop), so
it's latent — but incrementing before decode, like Go's MetricStreamingDecoder.NextMetric, restores it for free.
Verified clean: field numbers/wire types against metrics.proto for every message; the 10th-byte varint overflow rule matches prost's
decode_varint_slow; zigzag, packed/unpacked, group-skip recursion bound; the Option→default migration in series, start_timestamp, exemplar,
magic_label_value, update_series preserves old None semantics; decode/clear reset every field; the series_name.len() == family.name.len() borrow
heuristic in labels() is equivalent to "empty suffix"; every &'a str in LabelsBuffer<'a> points into input, so the reuse contract holds.
Two cosmetic notes: family.metric.clear() at protobufparse.rs:1250 is dead (family isn't read after encode_to_vec()), and labels() / label_at()
duplicate the index→label mapping — they agree now, but editing one would silently desync entry_bytes from labels().
ProtobufParsernow decodes with a small zero-copy decoder instead of prost, solabels()andexemplar()borrow label names and values from the input.Closes #26
What changed
parse/protowire.rsdecodes theio.prometheus.clientmessages the parser needs.MetricFamily::decodereadsname,help,typeandunitas&'a strand records where each metric is.Metric::decodedecodes one metric when the parser reaches it.labels()returnsCow::Borrowedfor everything in the input. Only the summary_count/_sumnames and thequantilevalue are owned, because they are not in the input.Parsertrait,ProtobufParser::newand the parser output did not change.go_diff_testsstill matches Go byte for byte.Compared to Go's
MetricStreamingDecoder: Go borrows withyoloString(unsafe.String) and unmarshals counter, summary and histogram into gogo structs. Here the'alifetime checks the borrows, strings are still UTF-8 validated like prost did, and there is nounsafe.Decoding follows prost and gogo-protobuf: a known field with the wrong wire type is an error, unknown fields and groups are skipped, repeated scalars may be packed or not, and a repeated message field is merged. Unit tests in
protowire.rsdecode the same bytes with prost and compare every field.Behavior change
Decoding per metric removes the old TODO in
next_metric_family. A malformed metric now fails only when the parser reaches it, after the earlier metrics of its family were returned. This matches Go'sNextMetric. Covered byreturns_metrics_before_a_malformed_one.Benchmark
The first commit adds
benches/parse.rs(benchfmt), which does what Go'sBenchmarkParsePromProtodoes: every entry with its accessor, then labels, start timestamp and all exemplars.textparse.TestProtobufParseis the payload of Go'screateTestProtoBuf.parseis crate-internal, so the bench needs a way in. I added a non-defaultbench-internalsfeature that makesparsepublic with#[doc(hidden)]. Happy to change it if there is a better option.Before: 7ace352 (bench only), after: 244af16.
Go v0.315.1
BenchmarkParsePromProtoon the same machine and the same payload ascase=textparse.TestProtobufParse:This is not quite apples to apples: Go builds
labels.Labelsthrough aSymbolTable, and Rust fills aLabelsBuffer.What's left
The remaining time on
expfmt.protobufis mostly not the decoder. That payload has 133 summary quantile series, and formatting eachquantilevalue withformat_openmetrics_floattakes about half ofnext_entry. With formatting stubbed out,next_entryalone went from 54µs to 28µs.float.rsis shared with the text parsers, so I'd fix it in a separate PR.The remaining 229 allocs/op on
expfmt.protobufare the 84 summary_count/_sumnames and the 133quantilevalues.LabelsBufferholdsCow<'a, str>, so they cannot borrow from the parser.Tested
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features -- -D warningscargo clippy --workspace --all-targets --no-default-features -- -D warningscargo test --workspace --all-featurescargo test --workspace --no-default-featurescrates/expfmt/src/arrowis not touched, so I did not run the OTAP suite.