Skip to content

backend/feat: add forks_started_total and forks_finished_total counters - #1514

Merged
piyushKumar-1 merged 1 commit into
mainfrom
backend/feat/fork-metrics
Sep 13, 2026
Merged

piyushKumar-1 merged 1 commit into
mainfrom
backend/feat/fork-metrics

Conversation

@piyushKumar-1

@piyushKumar-1 piyushKumar-1 commented Sep 11, 2026

Copy link
Copy Markdown
Member

What

Adds two Prometheus counters that track forked threads created through Forkable (FlowR r) (fork, awaitableFork, forkMultiple):

Metric Type Labels
forks_started_total Counter tag, fork_type, version
forks_finished_total Counter tag, fork_type, version

fork_typefork / awaitable_fork / fork_multiple. The pod label is added by vmagent at scrape time.

Active forks per pod at any instant:

sum by (pod) (forks_started_total{service="beckn-driver-offer-bpp-production"})
  - sum by (pod) (forks_finished_total{service="beckn-driver-offer-bpp-production"})

Which fork is piling up:

topk(10, sum by (tag) (forks_started_total - forks_finished_total))

How

  • New CoreMetrics method withForkCounters :: Text -> Text -> m a -> m a. The FlowR implementation wraps the forked body in bracket_, so forks_finished_total increments on success, on error, and when the thread is killed by an async exception. euler-hs's GeneralBracket interprets to the real IO generalBracket.
  • Counting happens inside the forked thread, so it reflects threads that are actually running, and the tag sanitisation cost stays off the caller's thread.
  • MockM and the test IO instance are no-ops. There are no CoreMetrics instances in nammayatri, so consumers only need a flake bump.

Tag label cardinality

47 call sites append ids to the fork tag, e.g. "cancelBooking:" <> bookingId. Prod logs confirm values like cancelBooking:0baf1774-afb8-…. To keep the label bounded, the tag is sanitised before use:

  1. UUIDs are replaced with :id (for tags of 36+ chars; the regex is compiled once).
  2. The result is truncated to 32 chars.
  3. At most 300 distinct tag labels per process; anything beyond is recorded as other.

Worst case this adds 600 series per pod (~1% on driver-bpp, ~3% on rider-app at today's per-pod series counts).

The forks-started and forks-finished counters for the same raw tag always resolve to the same label, because sanitisation is deterministic and the known-label set only grows.

Testing

  • cabal build mobility-core:lib:mobility-core passes; cabal test mobility-core:tests passes.
  • Formatted with the repo's treefmt ormolu settings (check mode clean). hlint reports no new hints.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WyZ8n4mSmpcmEnPfX7GVb5

Summary by CodeRabbit

  • New Features
    • Added lifecycle metrics for background fork operations, including started and completed counts.
    • Fork metrics now distinguish standard, multiple, and awaitable fork operations.
    • Added fork labels to help identify activity by operation and deployment version.
    • Labels are normalized to remove volatile identifiers, limited in length, and grouped under “other” when too many unique values are encountered.

Count forked threads created through Forkable (FlowR r) with two counters
labelled by sanitized fork tag, fork type and version. Active forks per pod
is started minus finished. The finished counter is bumped via bracket_ so it
also fires when the thread errors or is killed.

The tag label replaces UUIDs with ":id", is truncated to 32 chars and is
capped at 300 distinct values per process (overflow recorded as "other").

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WyZ8n4mSmpcmEnPfX7GVb5
@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: be49983f-d8f2-45e5-bd29-17ce67acc9d0

📥 Commits

Reviewing files that changed from the base of the PR and between 2a2485a and 495fe9a.

📒 Files selected for processing (5)
  • lib/mobility-core/src/Kernel/Mock/App.hs
  • lib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics.hs
  • lib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics/Types.hs
  • lib/mobility-core/src/Kernel/Types/Flow.hs
  • lib/mobility-core/test/src/APIExceptions.hs

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


Walkthrough

The change adds fork lifecycle counters with bounded, sanitized labels. FlowR instruments fork, forkMultiple, and awaitableFork. Mock and IO metrics implementations execute actions without counter handling.

Changes

Fork lifecycle metrics

Layer / File(s) Summary
Fork counter contracts and registration
lib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics/Types.hs
The metrics contract and container now include started and finished fork counters plus bounded label storage. Prometheus registration creates counters labelled by tag, fork type, and version.
Fork counter implementation
lib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics.hs
Fork tags are truncated, UUIDs are replaced, and known labels are bounded. Actions increment started and finished counters with bracketed cleanup.
Flow fork instrumentation
lib/mobility-core/src/Kernel/Types/Flow.hs, lib/mobility-core/src/Kernel/Mock/App.hs, lib/mobility-core/test/src/APIExceptions.hs
FlowR delegates to the implementation and instruments three fork operations. Mock and IO instances pass actions through unchanged.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant FlowR
  participant Metrics
  participant Counters
  FlowR->>Metrics: start fork with tag and fork type
  Metrics->>Counters: increment forks_started_total
  Metrics->>FlowR: execute fork action
  Metrics->>Counters: increment forks_finished_total
Loading

Suggested reviewers: harshit12c

Merge Risk: ⚪ Minimal · up to 495fe

The fork metrics implementation registers bounded labels and instruments all described FlowR fork paths without an identified merge-blocking risk.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: adding the forks_started_total and forks_finished_total counters.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch backend/feat/fork-metrics

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit reads each line,
The patch grows clear beneath the moon,
Small changes hop in place,
Tests guard the garden path,
Reviews bloom before the dawn.

Comment @coderabbitai help to get the list of available commands.

@piyushKumar-1
piyushKumar-1 merged commit 0a8170e into main Sep 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant