Repository navigation
TLS stage silently discards received inbound data when the transport output is cancelled #3739
Description
Motivation
Recent work hardened cancellation behaviour on the plaintext-output side of the TLS operators (#3727, #3731, #3738). While auditing cancellation symmetry across the TLS stage's four teardown paths, I found a gap in the opposite direction: when the transport (ciphertext) output is cancelled while inbound records have already been received but not yet decrypted, both TLS engines discard that data silently.
The other three teardown paths all have an observable contract:
- transport input truncated without
close_notify: received data is delivered andSessionTruncatedis emitted (documented, covered byTlsSpec); - transport input failure: the stage fails via
failTls, visible to the application; - plaintext output cancellation: the graph-stage engine drains accepted plaintext (Preserve accepted TLS plaintext on output cancellation #3727, Keep draining TLS plaintext after a TLS 1.3 peer close_notify #3731, Keep draining TLS plaintext after a TLS 1.3 KeyUpdate #3738), documented in the stream IO guide.
Only "transport output cancelled while inbound data awaits plaintext demand" ends in a clean onComplete with the received data dropped and no signal at all.
Scenario
Observed on main (2.0.0-M4+ snapshot), both pekko.stream.materializer.tls.engine = legacy-actor and graph-stage.
- A TLS client stage with an established session. The peer sends an application data record (e.g. a final response); the client's transport input receives the ciphertext.
- The client's plaintext output has no outstanding demand at that moment (the consumer has not requested yet).
- The transport output is then cancelled while the read side stays healthy (e.g. a kill switch on the send path, or an application that deliberately stops sending but still expects to read the response).
Observed
- The stage completes immediately; the plaintext output receives
onCompletewith noSessionBytes. - The already-received record is dropped undecrypted, and no
SessionTruncatedis emitted, so the application cannot tell that data received at the record layer was thrown away. - Identical behaviour with both engines.
- If plaintext demand existed before the cancellation, the data is delivered normally, so the loss is specific to "cancellation while inbound data awaits demand".
Internal notes
- Graph stage: inbound decryption is gated on plaintext-output demand (
inboundReadyrequiresuserOutput.isDemandAvailableinTlsGraphStage). A cancelled transport output makesoutboundCompleted(and thusbidirectionalCompleted) true immediately, so the pump skips every phase and reachespumpFinished, which cancels the transport input queue — the inbound close path never runs, hence nocloseInbound, noSessionTruncated. - Legacy actor: same demand gating in the inbound transfer state (
demandAvailableFor(UserOut)); a cancelled transport output jumps straight to the completed phase (TLSActor.doOutbound).
Expected
Either of the following, plus a sentence in the stream IO guide (the current text only covers the plaintext-output-cancellation/drain direction):
- Deliver already-received inbound records to the plaintext output as demand arrives before completing — the read side is independent of the cancelled write side; or
- At minimum, run the inbound close path so
SessionTruncatedis emitted before completion, making the loss observable.
Option 2 is a small, low-risk change (signal only, no completion-timing change). Option 1 mirrors the outbound drain philosophy but needs a decision on how long the stage may wait for plaintext demand, similar to the trade-off already accepted and documented for the outbound drain.
Activity
He-Pin commentedon Oct 30, 2026
I re-checked this on
main(f785e2f) with probe-level tests against both engines. The scenario as described does not reproduce, so I am closing this. The reasoning is below so it stays auditable, and the related defect I did find on the same teardown path is fixed in #3740.What I measured
Harness: the operator under test with all four ports on test probes, driven by a plain
SSLEnginepeer. The session is established first, then the peer sends an application record while the plaintext output has no outstanding demand, then the transport (ciphertext) output is cancelled.SessionBytes("response")All six then complete normally and cancel the transport input.
Where the internal note goes wrong
bidirectionalCompletedisoutboundCompleted && inboundCompleted. A cancelled transport output does makeoutboundCompletedtrue, butinboundCompletedadditionally requires(transportInput.isCompleted || userOutput.isClosed) && (engine.isInboundDone || userOutput.isErrored). With the transport input still open and the engine's inbound side not done, that is false, so the bidirectional phase is neither executable nor completed: the pump idles instead of reachingpumpFinished, and the transport input queue is not cancelled. When plaintext demand arrives,inboundReadybecomes true, the record is unwrapped and pushed, and only then doesdoOutboundobserve the cancelled transport output and move toCompletedPhase.The legacy actor behaves the same way:
inbound'sisCompletedneedsuserOutCancelled.isCompleted, i.e.engine.isInboundDone || outputBunch.isErrored(UserOut), sobidirectionaldoes not complete on a transport output cancellation either.Steps 1 and 2 of the scenario also cannot both hold here. Inbound decryption is gated on plaintext output demand (
inboundReadyrequiresuserOutput.isDemandAvailable; the actor requiresdemandAvailableFor(UserOut)), so a session can only be established if the plaintext output had demand at some point. With a plaintext output that never requested anything, the handshake itself never progresses — I hit exactly that while building the harness.The related defect that is real
Auditing the same teardown path did turn up a genuine loss, narrower than the one described: when two or more inbound transport elements are buffered while the plaintext output has no demand, cancelling the transport output delivers only the first and discards the rest with a clean
onCompleteand noSessionTruncated. Reproduced on both engines and both protocols. The control, with the transport output left open, delivers both records.Cause:
doOutboundmoves straight to the completed phase on a cancelled transport output, so the pump performs at most one inbound round beforepumpFinished()cancels the transport input.#3740 fixes that by reporting
SessionTruncatedon this path in both engines, with a directional spec where 14 of 18 cases fail without the production change.