Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements publish hedging support in the Pub/Sub publisher, integrating a hedging scheduler, token bucket, and cancellation coordinator into the message queues, while ensuring ordered queues do not use hedging. It also adds support for abort signals in API requests to handle cancellations immediately. The review feedback suggests replacing shallow copies of gaxOpts with deep copies using extend to prevent accidental mutation of nested configuration options by underlying libraries.
d12ddf1 to
f675ae1
Compare
f675ae1 to
3c11c08
Compare
3c11c08 to
e954f8d
Compare
e95b3c8 to
b6382d8
Compare
e89fc18 to
02582a7
Compare
02582a7 to
1231c0c
Compare
1231c0c to
d55d984
Compare
d55d984 to
7c8bffb
Compare
danieljbruce
left a comment
There was a problem hiding this comment.
LGTM. Just one nit/question about an interface change, but otherwise looks fine.
| flowControl: FlowControl; | ||
| hedgingOptions?: ResolvedHedgingOptions; | ||
| private tokenBucket_?: HedgingTokenBucket; | ||
| hedgingScheduler?: HedgingScheduler; |
There was a problem hiding this comment.
nit: Can this be private or is it needed elsewhere?
Overview
This PR is Layer 5 of 7 in Stack #9467, which implements Publish Hedging for
@google-cloud/pubsubto mitigate high-percentile (p99/p99.9) publish tail latencies by sending backup (hedged) publish RPCs when an in-flight batch takes longer than a configurablehedgeDelaythreshold.Stack Structure
feat(gax): support AbortSignal cancellation in CallOptions and createApiCallfeat(pubsub): add HedgingOptions validation and HedgingTokenBucketfeat(pubsub): add CancellationSharer and HedgingSchedulerfeat(pubsub): add OpenTelemetry events and logging for hedged publishfeat(pubsub): integrate publish hedging into MessageQueue(this PR)test(pubsub): add unit tests for publish hedging queue integrationtest(pubsub): add publish hedging system testsSummary of Changes in This PR
This layer wires
CancellationSharer,HedgingScheduler,AbortSignalcancellation, and OpenTelemetry/logging intoPublisher,MessageQueue._publish, andPubSub.prototype.request:PubSub.prototype.request(handwritten/pubsub/src/pubsub.ts):config.gaxOpts?.signal?.abortedbefore and afterthis.getClient_, immediately invoking the callback with aCANCELLEDServiceErrorif aborted before the GAPIC method is called.PublisherLifecycle (handwritten/pubsub/src/publisher/index.ts):HedgingSchedulerinsetOptionswhenhedgingis configured (logging'Hedging rate limited due to lack of tokens.'at debug level when rate-limited) and clears any pending timer onflush().MessageQueue._publish&OrderedQueue(handwritten/pubsub/src/publisher/message-queues.ts):OrderedQueuenever hedges (allowsHedging()returnsfalse).Queue:0with its ownAbortControllerand full GAX retry settings.hedgingScheduler.scheduleFirstHedge: each hedged attempt logs'Publishing hedged attempt %i', recordsPubsubEvents.publishStartHedged, and dispatches a non-retrying (retry: null) publish RPC withtimeout: attemptTimeout.millisecondsand its ownAbortController.PubsubEvents.publishEndHedgedwhen a hedged attempt wins (orPubsubEvents.publishEndotherwise) and resolves message callbacks with the winning attempt'smessageIds.