API v3alpha: NotificationService

[siavashs]

Parent: #5450

Summary

Resolve the notification observability storage model, then define and implement the complete notification.v3alpha.NotificationService contract.

This issue owns the design decision, proto PR, and phased implementation. No public notification RPC should be published until its meaning, retention, persistence, and HA behavior are explicit.

Design gate

The current notification log (nflog) is not notification history. It stores the latest successful deduplication state for an exact group, receiver, and integration tuple. The event recorder has richer successful-delivery events, but it is feature-gated, best-effort, output-only, and has no query or replay API.

Choose one model:

1. Retained delivery history: add a dedicated append-only notification attempt or delivery-history store and make list/watch consume the same records.
2. Latest notification state: add an immutable query-all/filter/pagination API to nflog and name/document the RPC as current state rather than history.
3. Explicit deferral: do not publish NotificationService if neither storage model has acceptable semantics and operational cost.

The selected design must define:

• Which outcomes are recorded: pre-send, sent, failed, retrying, or final attempts only.
• Stable record identity.
• Receiver labels and group metadata captured at event time.
• Retention and persistence.
• HA replication, merge, failover, and duplicate behavior.
• Ordering, pagination, cursor, and replay guarantees.
• Backpressure and non-blocking publication.
• Sanitization of integration-specific data.

Do not expose internal eventrecorderpb messages or raw nflog.receiver_data as the public API contract.

Proto PR

Only after the design is approved:

• Define notification/state/history messages consistent with the chosen model.
• Define filters and shared pagination/error fields using common.v3alpha.
• Define ListNotifications and WatchNotifications if both are supported by the storage model.
• Document whether cursors are durable, node-local, or cluster-wide and what clients do after overflow, restart, or failover.

Implementation phases

• Add the selected queryable notification store or immutable nflog read model.
• Publish without blocking notification delivery or retry processing.
• Wire the dependency through app, reloader, and API options.
• Implement list/filter/order/pagination.
• Implement watch/snapshot/replay or resnapshot semantics supported by the chosen model.
• Register handler, health, reflection, and bounded instrumentation.
• Add unit, e2e, HA, retention, and backpressure coverage.

Acceptance criteria

• Notification semantics and storage are approved before proto publication.
• The public contract does not misrepresent nflog as delivery history.
• Internal event-recorder and receiver-specific payloads do not leak into the public schema.
• Retention, persistence, HA, ordering, pagination, and replay guarantees are documented.
• Notification publication cannot block the notification pipeline.
• List and watch read from semantics-compatible storage.
• Ginkgo/Gomega coverage includes filters, pagination, HA/failover behavior, backpressure, cancellation, and supported transports.
• If the decision is deferral, close this issue with the rationale and do not publish placeholder RPCs.

Dependencies

• Shared contracts and error/pagination infrastructure: API v3alpha: add shared contracts and reusable infrastructure #5479.
• Connect request, stream, and observability hardening: API v3alpha: harden the ConnectRPC foundation #5478.

[siavashs]

The optional notification-observability UI is tracked in #5494 under #5451. It is explicitly blocked on this design gate and excluded from the new UI's legacy-parity/completion criteria. If NotificationService is deferred, the UI child will close as not planned; if published, its terminology and list/watch behavior will follow the selected history-versus-latest-state semantics exactly.