Implement observability metrics for idempotent request outcomes

[raghavgopal]

Overview

Implement observability metrics to capture and track the outcomes of idempotent request handling in AvoOnce. This will provide visibility into request processing behavior and help monitor system health.

Metrics to Capture

Request Outcome Counters

The following metrics should be tracked for each idempotent request outcome:

• REPLAY: Requests that were replayed (idempotent request with cached result)
• EXECUTED: Requests that were executed normally (first-time execution)
• CONFLICT: Requests that encountered conflicts during processing
• MISMATCH: Requests with mismatched expected vs. actual results
• FAILED: Requests that failed during execution

Performance Metrics

• Execution Time: Duration taken to execute requests (histogram/gauge)
  • Track separately for REPLAY vs. EXECUTED outcomes
  • Measure from request start to response completion
• Cached Response Size: Size of cached responses in bytes
  • Track distribution of cached payload sizes
  • Help identify memory usage patterns

Additional Metrics

• Request throughput (requests per second)
• Cache hit/miss ratios
• Error rates by outcome type

Requirements

1. Counter Metrics: Implement counters to track the total number of requests for each outcome type
2. Histogram/Gauge Metrics: Use histograms or gauges to track execution time and response sizes with appropriate buckets
3. Labeling: Use appropriate labels/tags to distinguish between:
   • Outcome state (REPLAY, EXECUTED, CONFLICT, MISMATCH, FAILED)
   • Request type or endpoint (if applicable)
   • Any other relevant dimensions
4. Exportability: Ensure metrics are exposed in a standard format (e.g., Prometheus, OpenTelemetry) for easy integration with monitoring systems
5. Documentation: Document the available metrics, their meanings, units, and how to interpret them for operators

Expected Outcomes

• Clear observability into idempotent request behavior and performance
• Ability to monitor execution performance between cached vs. fresh executions
• Understanding of memory footprint through cached response size tracking
• Support for debugging, performance analysis, and capacity planning