feat(outbox): report outbox messages that stay undispatched

[guilhermeappel]

Description

This is the telemetry part of issue #4489. As I said on the issue, I'm starting it as a draft. It has the requirements, spec 0038, and two ADRs, ADR 0085-outbox-stale-message-telemetry and ADR 0086-outbox-per-topic-stale-count, which stay Proposed until you're happy with them. If you'd change the design, now is the cheapest time to say so.

The feature reports how many outbox messages have stayed undispatched for longer than an age you set, so an operator can alert on a message that was never sent, even in an app that runs without the sweeper. A new hosted service in Outbox.Hosting, registered with UseOutboxStaleMessageMonitor in the same way as UseOutboxArchiver, counts on a fixed interval, with a timeout. It makes two counts at each interval, and one never waits for the other. The overall count goes through the outbox's existing async count. The count per topic goes through a new optional outbox interface, which ADR 0086-outbox-per-topic-stale-count decides, so a topic whose circuit breaker is open shows up on its own. Each publication can set its own stale age, falling back to the monitor's, and a topic shared by several publications uses the smallest. A sealed meter in core, created by AddBrighterInstrumentation like PublishConfirmationMeter from PR #4562, publishes each count as an observable gauge, one series per topic for the second, and a counter of how each scheduled measurement ended, so "no recent reading" looks different from zero. The mediator, the sweeper and the existing meter roles don't change.

These are the answers I went with for the three questions in my last comment on the issue:

1. The loop lives in its own hosted service, so it keeps measuring when the sweeper or the posting path is stuck.
2. No age-aware count for now. It would change every relational outbox and MongoDb, and the monitor works without it. Until then those outboxes still load up to the ceiling's worth of rows on each overall count, 100 by default and at most 1,000, which ADR 0085-outbox-stale-message-telemetry lists as a known limitation for a follow-up.
3. The instruments are on a meter of their own rather than on OutboxMeter from PR Add outbox metrics #4484. They come from IMeterFactory like PublishConfirmationMeter, so they don't depend on when the mediator is built, and OutboxMeter and its role stay as they are. Alternative 12 in ADR 0085-outbox-stale-message-telemetry has the details. If you'd rather the outbox had one meter, I'll move them. The new meter has no role interface, so moving it later breaks no implementer.

After your review I added an age per publication and the count per topic, as described above. The count per topic only works on an outbox that implements the new interface. Only InMemoryOutbox does in this PR, and each of the other outbox families gets a follow-up issue, so until then they report the overall count only. The gauges don't tell instances apart: each process measures its own outbox, the series carries the outbox type in a new paramore.brighter.outbox.implementation attribute, and the docs will say to combine instances that share a database with max rather than sum (alternative 17 in ADR 0085-outbox-stale-message-telemetry).

I read "telemetry on the call that warns" as a metric you can alert on, counted on a schedule rather than on each Post or Clear, because a quiet app with no posts would never report a message that failed once. There's no Warning log for each count above zero either. Alternative 13 in ADR 0085-outbox-stale-message-telemetry has why, and alternative 23 there says why I didn't rely on added minus cleared from ADR 0084-outbox-metrics. The requirements still list my questions as Q-1 to Q-16, and the ADRs answer each of them. Q-7, Q-14 and Q-15 are the ones your review reopened.

Related Issues

Part of issue #4489. The channel for failed clears, the other part of the issue, isn't in this PR.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have read the Contributing Guide
• I have checked the documentation for relevant guidance
• I have added/updated XML documentation for any public API changes
• I have added/updated tests as appropriate
• My changes follow the existing code style and conventions

Additional Notes

• New public API: UseOutboxStaleMessageMonitor, TimedOutboxStaleMessageMonitor and its options in Outbox.Hosting, and in core the meter, a small readings type that the monitor writes and the meter reads, six names in BrighterSemanticConventions, the optional IAmATopicCountingOutbox interface and a nullable StaleMessageAge property on Publication. The readings type is public only because Outbox.Hosting is a separate assembly, the same reason AmbientScopeSuppression is public.
• The review findings for the requirements and the ADRs are in review-requirements.md and review-design.md in the spec folder.
• Regenerating docs/adr/index.md also adds the missing row for ADR 0082-isolate-azure-service-bus-subscription-retries.
• Please merge with a merge commit rather than squash, as CONTRIBUTING asks.

[iancooper]

There's one age for the whole outbox, not one per publication, because an age per publication needs a count and a series per topic (alternative 16)

Because we can circuit break a topic, we should have an age per publication, otherwise we won't see that clearly

[iancooper]

We might choose to support both this "overall outstanding count" and a "per topic outstanding count" to allow for topic circuit breakers to show up

[guilhermeappel]

Makes sense. While a topic's circuit is open, the sweeper skips that topic, so its messages stay in the outbox. One count over the whole outbox can't show which topic they belong to.

I'll keep the overall count, as you suggest, and add a count per topic on an instrument of its own, tagged with messaging.destination.name like the outbox metrics from ADR 0084-outbox-metrics. Each publication will be able to set its own age, falling back to the monitor's. Today only the MongoDb outbox can filter its count by topic, so I'd add the count per topic as a new optional capability on the outbox, and an outbox without it still reports the overall count. The ADR will also say how that capability and the ceiling work together, which outboxes get it in this PR, and which age a topic uses when two publications share it. I'll push the changes here and tag you when they're ready.

[guilhermeappel]

@iancooper I've pushed the revised requirements and ADRs for the age per publication and the count per topic. ADR 0085-outbox-stale-message-telemetry now covers both counts. The count per topic got an ADR of its own, ADR 0086-outbox-per-topic-stale-count, because the optional interface it adds to the outbox is a separate decision that each outbox family can then implement on its own.

On the points I said the ADRs would cover: the ceiling applies to each topic on its own, so a busy topic can't hide a quiet one, and a topic shared by two publications uses the smaller age. In this PR only InMemoryOutbox will implement the interface. The database outboxes, MongoDb included, will report the overall count until their follow-ups land, with one issue per family. If you'd rather have one of them in this PR, tell me which and I'll add it.

Running the two counts side by side touches most of the mechanism in ADR 0085-outbox-stale-message-telemetry. The new sections there are the ones on the smallest age, the age on Publication, the second gauge and when each setting is checked. Ready for another look when you have time.