Repository navigation
connector: a wedged name lookup returns within the dial deadline (money-security L2042/L2043) - #963
Merged
Merged
Conversation
…y-security L2042/L2043) SystemNames::resolve joined its lookup thread with no deadline, so a getaddrinfo that wedges held the kernel's resolve-then-pin caller for as long as it did. The lookup now answers over a channel the caller waits on for at most compose::DEFAULT_OPEN_TIMEOUT (the open's bound when the caller states none); no answer by then is Err, which the egress guard already reads as Unresolvable. The lookup thread is never joined. The Resolver seam is unchanged. RED: hostio_tests a_wedged_name_lookup_fails_within_its_bound_and_an_answer_in_time_is_kept plants a lookup that blocks far past a 300 ms bound and asserts Err with elapsed in [bound, bound + 1 s).
A wedged lookup's thread is let go, never joined, so a resolver that wedges again and again left one more thread behind each time. A lookup now takes a place in an in-flight count BEFORE its thread starts; with compose::MAX_PENDING_LOOKUPS (64) places taken it is refused at once (Err, which the egress guard reads as Unresolvable), no thread started. The thread gives its place back through a drop guard when its lookup ends (answers, errs or panics), even after its caller stopped waiting. The public path passes the const and a shared count; within_capped takes both as parameters. RED: hostio_tests wedged_name_lookups_are_capped_and_an_ended_one_gives_its_place_back plants two wedges under a cap of 2 with the test's own count, asserts the next lookup errs in under 1 s (bound 10 s) and never runs, then releases one wedge and asserts a new lookup is answered, and a panicking lookup gives its place back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defect (money-security L2042/L2043).
hostio::SystemNames::resolve(the kernel's resolve-then-pin lookup behind the egress-trustResolverseam) ranlookup_hoston a scoped thread and.join()ed it with no deadline.getaddrinfocan wedge; the caller then blocked for as long as it did.Bound source. The connector's existing open bound,
compose::DEFAULT_OPEN_TIMEOUT(crates/busbar-core-connector/src/compose.rs:52, 10 s: "the open's bound when the caller states none"). TheResolverseam carries no deadline, so this lookup states none. No new config knob; theResolversignature is unchanged.Change. The lookup runs on a detached thread and answers over a
sync_channel(1); the callerrecv_timeouts for at most the bound. No answer by then isErr("name lookup: no answer within 10s"), whichtrust::netalready maps toAddressRefusal::Unresolvable. The wedged thread is never joined; it ends when its lookup does.RED.
hostio_tests::a_wedged_name_lookup_fails_within_its_bound_and_an_answer_in_time_is_kept: a planted lookup that blocks far past a 300 ms bound returnsErrwith elapsed in [bound, bound + 1 s). Against the old join it never returns. It also checks an in-time answer is kept and a panicking lookup isErr.Base. On predev, not stacked on #865: #865's branch does not hold
SystemNames(it landed on predev after #865 branched). This hunk and the test's hunk are disjoint from #865's;git merge-treeagainstlane-row166-spawn-envis clean. Draft per the money-lands-alone rule.