[agbochenuv-bit]

Summary

Harden backend: audit integrity, readiness probes, backup verification, and distributed tracing

What was solved

#975 — Make admin audit records tamper-evident

Add tamper-evident integrity protection for critical administrative audit records in the backend, defining a verification mechanism (e.g., hash-chained records) and covering audit writes, failure cases, and verification with tests.

Addressed:

• Changed: backend/src/auth.rs, backend/src/handlers.rs, backend/src/lib.rs
• Define an integrity verification mechanism for administrative audit records so silent modification or deletion is detectable
• Apply integrity protection to audit record writes
• Handle and surface integrity verification failures

#976 — Add dependency-specific readiness checks

Add dependency-specific readiness checks to the backend so MongoDB, Redis, and Soroban RPC are probed independently, with the readiness endpoint reporting each dependency's status and returning an unhealthy status when any required service is unavailable.

Addressed:

• Changed: backend/src/handlers_readiness.rs, backend/src/lib.rs, backend/src/main.rs
• Implement independent readiness probes for MongoDB, Redis, and Soroban RPC
• Readiness response must identify which dependencies are unavailable
• Return an unhealthy status when required services are down

#977 — Automate backup and restore verification

Add an operator-run/scheduled backup restore verification for MongoDB and Redis that actually restores backups into a scratch environment, checks recovered data, and reports pass/fail. This is a new operational script plus a CI workflow (and possibly a Makefile target) rather than changes to application code.

Addressed:

• Changed: .github/workflows/ci-stack-smoke.yml, Makefile, backend/.env.example, .github/workflows/ci-post-deploy-verify.yml
• Provide a repeatable operator-run and/or scheduled check that restores MongoDB and Redis backups (not just documentation).
• Verify recovered data after restore (e.g., expected collections/documents and Redis keys/values present).
• Report failures clearly with non-zero exit status so CI/schedulers surface them.

#978 — Add distributed tracing across backend operations

Add distributed tracing to the backend so API requests are correlated with Soroban RPC calls, indexing, and webhook delivery via propagated trace context, while redacting secrets and sensitive payload data from traces.

Addressed:

• Changed: backend/src/main.rs, backend/src/lib.rs, backend/src/handlers.rs, backend/Cargo.toml
• Introduce/propagate trace context (trace/span IDs) across backend operations so API requests correlate with Soroban RPC calls, indexing, and webhook delivery
• Preserve request correlation end-to-end (inbound request context flows into downstream RPC/indexing/webhook spans)
• Redact secrets and sensitive payload data from trace output (e.g., auth tokens, keys, request/response bodies)

Changes

• backend/src/lib.rs (modify)
• backend/src/auth.rs (modify)
• backend/src/handlers.rs (modify)
• backend/src/main.rs (modify)
• backend/.env.example (modify)
• backend/src/handlers_readiness.rs (modify)
• Makefile (modify)
• backend/Cargo.toml (modify)
• .github/workflows/ci-stack-smoke.yml (modify)
• .github/workflows/ci-post-deploy-verify.yml (modify)

Approach

1. Make admin audit records tamper-evident #975 — Make admin audit records tamper-evident (Changed: backend/src/auth.rs, backend/src/handlers.rs, backend/src/lib.rs)
2. Add dependency-specific readiness checks #976 — Add dependency-specific readiness checks (Changed: backend/src/handlers_readiness.rs, backend/src/lib.rs, backend/src/main.rs)
3. Automate backup and restore verification #977 — Automate backup and restore verification (Changed: .github/workflows/ci-stack-smoke.yml, Makefile, backend/.env.example, .github/workflows/ci-post-deploy-verify.yml)
4. Add distributed tracing across backend operations #978 — Add distributed tracing across backend operations (Changed: backend/src/main.rs, backend/src/lib.rs, backend/src/handlers.rs, backend/Cargo.toml)

Issues

Closes #975
Closes #976
Closes #977
Closes #978

[drips-wave]

@agbochenuv-bit Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits