fix(budgets): make LiteLLM spend reporting resilient

[ak684]

Context

This is an alternative follow-up to #207. It includes the useful #207 changes on top of current main. LiteLLM counters drive budget reporting and sync writes are verified, but this PR hardens the boundary before those counters can affect durable OpenHands cycle state.

The intended authority split is:

• OpenHands owns budget policy and cycle configuration.
• LiteLLM owns usage metering and final request enforcement.
• The OpenHands dashboard is an explicitly time-stamped, eventually consistent projection of LiteLLM spend.

If this direction makes sense, this PR can supersede #207.

What changes

• Validates the LiteLLM financial response in one typed boundary. Missing, malformed, negative, or non-finite counters are unavailable rather than $0.
• Tightens the existing LiteLlmManager adapter so upstream contract drift cannot be normalized to zero before service validation sees it.
• Retries transient financial read failures with a small bounded retry policy. Timeouts, transport errors, HTTP 429, and HTTP 5xx responses are retried, while malformed responses and non-transient errors fail immediately into the safe stale or unavailable path.
• Persists the last successfully validated team and member spend observation via migration 153.
• Reports spend as live, stale, or unavailable, including the observation timestamp. Unknown spend remains null through the API and UI.
• Uses a single initial snapshot per read or maintenance phase instead of implicitly refetching from multiple helpers. Sync still performs one separate fresh readback to verify writes.
• Requires a fresh, complete membership snapshot before enable, reset-day changes, or scheduled cycle rollover. Team and user baselines are captured atomically from that same observation.
• Leaves the existing cycle and baselines intact when fresh data is unavailable. Baseline-changing settings requests return 503 and maintenance records an actionable sync error.
• Compares OpenHands membership with LiteLLM membership. Missing governed users are a sync error, not zero-spend users. LiteLLM-only service accounts are not automatically assigned the default human-user cap.
• Makes the dashboard explain snapshot freshness and that LiteLLM performs final request admission.

Required before declaring the budget work 100% shipped

• Choose and implement the recovery policy for an OpenHands member who is missing from the LiteLLM team. This PR detects the drift, preserves any existing baseline, records a sync error, and does not fabricate zero spend. It intentionally does not recreate the membership because a recreated LiteLLM membership may start with a zero cumulative counter. Applying the old absolute cap could grant extra allowance, while applying a zero or reduced cap could block the user. A safe default would be to avoid automatic mid-cycle repair and recreate the membership at the next verified cycle boundary, when granting a fresh allowance is expected. If immediate repair is required, it needs an explicit rule for calculating remaining allowance from trusted preserved state.
• Choose and implement the rollout policy for enabled organizations whose user_cycle_start_spend map has no baseline for an existing member. Migration 149 introduced this map with an empty default. The current compatibility behavior initializes a missing baseline to the member's current cumulative spend, which gives that member a full allowance from deployment time. The alternatives are a conservative no-additional-allowance policy until the next cycle or a historical backfill from a trustworthy per-user spend source. This cannot be inferred safely from the current schema alone.
• Run the real OHE Replicated end-to-end matrix before promotion. It should cover UI conversations and direct LiteLLM SDK traffic, an enabled budget with team and per-user limits, threshold alerts based on direct SDK spend, cycle rollover, reset-day changes, dropped LiteLLM reads, malformed responses, write/readback mismatch, missing membership drift, service accounts that exist only in LiteLLM, container restart, and the migration from the currently deployed OHE version.
• Define the rollout and rollback procedure. At minimum, capture pre-release LiteLLM team and member caps, deploy first to a customer-specific or pre-release channel, verify counters and absolute caps after one maintenance run, and make sure rollback does not restore the old climbing-cap behavior or erase the new baseline state.

The first two items are policy-dependent code work. They do not require a change to the LiteLLM repository, but they require an explicit allowance decision because the historical per-user counter needed to distinguish prior-cycle spend from current-cycle spend may no longer exist. The third and fourth items are release validation and operations work rather than missing implementation in this PR.

Known boundary, not a ship blocker

This solution does not require an upstream LiteLLM change. A future endpoint exposing the same counter used synchronously by enforcement could reduce observation lag, but the dashboard still could not atomically predict whether a later inference request will be admitted. The UI therefore presents the observation time and treats LiteLLM as the final request-admission authority.

Validation

• PYTHONPATH=enterprise poetry run --project=enterprise pytest enterprise/tests/unit/test_org_budget_service.py enterprise/tests/unit/test_org_budget_maintenance_processor.py enterprise/tests/unit/test_lite_llm_manager.py enterprise/tests/unit/test_migration_graph.py -q passed locally.
• Targeted Python formatting, import checks, and mypy passed.
• npm run lint:fix passed with five existing warnings.
• npm run build passed.
• npm run typecheck passed.
• npm run test -- --run __tests__/components/features/budgets/budgets.test.tsx passed.

The new failure-path coverage includes transient read retry, malformed payloads, last-known-good fallback, no-cache unavailability, rollback on enable without fresh data, rollover preservation during an outage, readback mismatch, and membership drift.

---

Enterprise server image for this PR:

ghcr.io/openhands/enterprise-server:sha-b92e861

[github-actions]

⚠️ This PR contains migrations. Please synchronize before merging to prevent conflicts.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
enterprise/server/routes
org_models.py
orgs.py					1175-1184
enterprise/server/services
org_budget_service.py					77-78, 97, 101-108, 112-115, 119-123, 130-160, 176-178, 186-191, 202, 206-208, 216-246, 250, 287-294, 333-377, 398-404, 425-472, 571-588, 593-628, 635-640, 650-687, 694-715, 719, 722-749, 761-812, 815-826, 841-845, 866-867, 870-873, 883-933, 946-974, 991-1023
enterprise/storage
lite_llm_manager.py					1866-1906
org_budget_settings.py
Project Total

This report was generated by python-coverage-comment-action