Add Prometheus alerting rules for API latency and error-rate SLA thresholds

[RUKAYAT-CODER]

Overview

src/monitoring/metrics/ collects Prometheus metrics but no alerting rules are defined. SLA breaches (P99 latency > 1s, error rate > 1%) are only detectable by manually watching dashboards.

Specifications

Features:

• Define Prometheus alerting rules for: request error rate, P99 latency, queue depth, and DLQ depth.

Tasks:

• Create charts/teachlink-backend/templates/prometheus-rules.yaml with PrometheusRule CR.
• Define alerts: HighErrorRate (>1% 5xx for 5m), HighP99Latency (>1s for 10m), QueueDepthHigh (>1000 jobs for 10m).
• Configure alertmanager webhook or Slack route in chart values.
• Add alert documentation in docs/RUNBOOKS.md.

Impacted Files:

• charts/teachlink-backend/
• docs/RUNBOOKS.md

Acceptance Criteria

• HighErrorRate alert fires in a test with injected errors.
• Alerts include runbook links.
• Helm chart renders without errors after adding the template.

[Darkvader-ship-it]

@Darkvader-ship-it has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

can i work on this

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Darkvader-ship-it to this issue.

[KingDavid9999]

@KingDavid9999 has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

Hi maintainer, I'd very much appreciate the opportunity to work on this issue.
Having reviewed the acceptance criteria, I'm confident that I'd be able to have a PR up shortly after being assigned.
Kindly assign this issue to me.
Thank you!

ℹ️ Repo Maintainers: To accept this application, review their application or assign @KingDavid9999 to this issue.

[drips-wave]

Congratulations, @KingDavid9999! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on June 30, 2026.

🧑‍💻 @KingDavid9999: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊