Differentiate errors when the DNS resolver is unreachable versus a DNS error during validation #9018

Activity

1. mcpherrinm commented on Sep 25, 2026

   

   mcpherrinm

   on Sep 25, 2026

   ContributorAuthor

   More actions

   After discussion with @aarongable, I think it would be good to also split validation metrics based on the type of identifier (dns, IP), because while the challenge types are shared, there are some failure modes unique to each.
2. jsha

   self-assigned this

   on Oct 13, 2026
3. jsha

   added this to the 2026-10-13 milestone on Oct 13, 2026
4. jsha commented on Oct 15, 2026

   

   jsha

   on Oct 15, 2026

   Contributor

   More actions

   Worked on this for a while tonight. We do need a refactor here and we definitely should plumb through the info about "internal errors" (i.e. reaching a resolver) vs "user errors" (slow or broken authoritatives) farther.

   However, we do actually have a metric in the bdns package that shows what we want: rate(dns_query_time_count{result="failed"}[$__rate_interval])).

   boulder/bdns/dns.go

   Lines 216 to 227 in cf6406d

   	resp, rtt, err = c.exchanger.ExchangeContext(ctx, req, chosenServer)
   	// Do some metrics handling before we do error handling.
   	result := "failed"
   	if resp != nil {
   	result = dns.RcodeToString[resp.Rcode]
   	}
   	c.queryTime.With(prometheus.Labels{
   	"qtype": qtypeStr,
   	"result": result,
   	"resolver": chosenServerIP,
   	}).Observe(rtt.Seconds())

   Explanation: ExchangeContext is our DOH implementation. If it errors, it will return a non-nil err and a nil resp. The calling code increments dns_query_time_count with result="failed" in that case. The cases that are "user errors" are typically successful at the DOH level but failures at the DNS level and get result="SERVFAIL".

   There's a different metric in the va package, validation_latency_count, that has a label problem_type. This shows a misleading number: during the most recent outage (and presumably the earlier one as well), it showed validation_latency_count{problem_type="dns"} going up. That's the metric that #9019 seeks to change, by generating ProblemDetails with Type="serverInternal" for these cases. That's a good idea.

   The current diff I'm working on seeks to achieve that by changing the error contract for the bdns package so that it returns one specific error type for "user errors". Those will get turned into problem_type="dns", while all other errors will get turned into problem_type="serverInternal". This lets us take advantage of the VA's Error / InternalError logging (similar to the WFE's), which lets us be liberal about logging detailed errors for internal debugging while also being conservative about what we send to the user. However, as an FYI: it's turning into a big refactor, which carries some risk.
5. jsha commented on Oct 16, 2026

   

   jsha

   on Oct 16, 2026

   Contributor

   More actions

   I went ahead with an approach in #9103 that adds a new metric and doesn't attempt to change some of the user-visible ProblemTypes to serverInternal. I'd still like to do that as a followup but it was turning into a big spaghetti change.
6. jsha

   closed this as completedin #9103on Oct 19, 2026