Skip to content

feat: enforce daily quota and surface structured 429 with settings link - #201

Open
neubig wants to merge 28 commits into
mainfrom
feat/quota-enforcement
Open

feat: enforce daily quota and surface structured 429 with settings link#201
neubig wants to merge 28 commits into
mainfrom
feat/quota-enforcement

Conversation

@neubig

@neubig neubig commented Aug 19, 2026

Copy link
Copy Markdown
Member

Enforces the daily conversation quota at conversation creation time and ensures the blocked error is properly displayed to users.

Backend

  • Atomic reserve(): PostgreSQL upsert with conversation_count < limit guard; raises HTTP 429 with structured detail when limit is reached
  • release(): Decrements usage when conversation start fails
  • Enforcement wired into app_conversation_router.py: Reserves before starting, releases on failure
  • Structured 429 detail includes message field pointing to /settings/quota

OpenHands Frontend (error display)

  • retrieveAxiosErrorMessage extended to extract detail.message from structured FastAPI error responses
  • type-guards.ts: New isAxiosErrorWithStructuredDetail guard
  • Regression test for structured detail extraction

Stacking

PR Scope
#180 Schema foundation: limit column + usage table
#199 Read-only usage API + settings page with countdown
#212 Org-level quota exemptions and admin API
#200 Work-email quota requests, verification, admin APIs, PostHog
This PR Conversation enforcement, structured 429, error display

This pull request was created by an AI agent (OpenHands) on behalf of the user.


Enterprise server image for this PR:

ghcr.io/openhands/enterprise-server:sha-e0095e2

Add migration 148 with a nullable per-user daily_conversation_limit
override column on the user table and a daily_conversation_usage table
for atomic per-user, per-UTC-day conversation accounting with a unique
(user_id, usage_date) constraint. Register the model and add focused
storage tests for the nullable override and uniqueness invariant.

No enforcement or API behavior is introduced in this foundation; that
lands in subsequent stacked PRs.

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions github-actions Bot added the type: feat A new feature label Aug 19, 2026
neubig and others added 2 commits August 19, 2026 04:57
Co-authored-by: openhands <openhands@all-hands.dev>
Add GET /api/quota/status returning the authenticated user's effective
daily limit, used count, remaining, and next UTC-midnight reset_at.
Add a SaaS-only settings page at /settings/quota showing a progress bar
and a live HH:MM:SS countdown to the next reset. Includes focused tests
for the quota status service covering unlimited, partial, exhausted,
and no-usage-today cases.

Stacked on feat/daily-conversation-limit (PR #180).

Co-authored-by: openhands <openhands@all-hands.dev>
@neubig
neubig force-pushed the feat/quota-enforcement branch from ddaf29c to de1b36b Compare August 19, 2026 04:58
Co-authored-by: openhands <openhands@all-hands.dev>
@neubig
neubig force-pushed the feat/quota-enforcement branch from de1b36b to abe42cf Compare August 19, 2026 05:02
Add migration 149_org_quota with a nullable daily_conversation_limit
column on the org table. Update the quota service to resolve limits
with the precedence: user override → org override → env default → None.

Org-level exemptions use -1 to mean unlimited (NULL inherits the
deployment default). This allows paying SaaS orgs to be exempted from
daily conversation limits while still enforcing limits for other orgs.

Add PUT /api/admin/quota/orgs/{org_id}/quota admin endpoint for
setting or clearing org-level limits. 6 focused tests covering
user precedence, org override, org exemption, env fallback, and
unset/unlimited resolution.

Stacked on feat/quota-usage-page (PR #199).

Co-authored-by: openhands <openhands@all-hands.dev>
…cation

Add migration 149 with quota_increase_request table and work_email /
work_email_verified_at columns on the user model. Implement:
- Free-email domain rejection (exact set from research PR #88)
- QuotaIncreaseRequestService: create, approve (idempotent), list pending
- POST /api/quota/increase-request: validate work email, persist request,
  send signed JWS verification email (1-hour expiry), capture PostHog event
  and set work_email person property
- GET /api/quota/verify?token=...: unauthenticated self-service approval
  via signed token — applies requested limit immediately
- GET /api/admin/quota/requests + POST /api/admin/quota/requests/{id}/approve:
  admin list and approve fallback (superadmin-gated)
- Frontend: increase request form on /settings/quota with work email,
  requested limit (capped at 10x), optional reason, pending/approved states
- 14 focused tests covering free-email rejection, limit bounds, duplicates,
  idempotent approval, and self-service verification

Stacked on feat/quota-usage-page (PR #199).

Co-authored-by: openhands <openhands@all-hands.dev>
@neubig
neubig force-pushed the feat/quota-enforcement branch from abe42cf to 9f2fe46 Compare August 20, 2026 13:39
@linear

linear Bot commented Aug 20, 2026

Copy link
Copy Markdown
OSS-9996 Limit daily conversations to 20 with support for per-user increases

Context

The linked Slack discussion converged on:

  • Set a default daily conversation limit of 20.
  • If a user needs more, manually increase their limit to 200 without charging them.
  • If this happens frequently, automate the request/limit-increase workflow.
  • The initial implementation should be pragmatic and allow manual operations before building a larger self-service system.

Slack discussion: https://allhandsai.slack.com/archives/C0A4YK70L3X/p1786657867367199

Proposed design

Implement a per-user, per-UTC-day conversation-start quota enforced server-side at the canonical V1 app-conversation creation endpoint (POST /api/conversations). The limit should be independent of the number of active sandboxes and should count a conversation once when its start request is accepted. It should apply consistently to UI, SDK/API, and integration-triggered starts that run on behalf of a user.

Policy and data model

  • Add a configurable default limit in the enterprise server configuration, with the default set to None (unlimited). The OpenHands/OpenHands-Cloud Helm chart should be able to override this setting; OpenHands/saas-deploy should set the deployed SaaS default to 20. This keeps the enterprise application reusable without imposing a quota on every deployment.
  • Add a nullable per-user override (for example daily_conversation_limit) to the enterprise user/org-member settings model. NULL means use the configured deployment default; any explicit integer value is valid, including values below 20, above 200, 0, or negative values if operators intentionally set them. Overrides are operational data set directly in the database for now, not a user-facing/admin API. Keep the override separate from LLM budget settings because this is a count quota, not a spend quota.
  • Add a small daily usage table keyed by (user_id, usage_date) with conversation_count and timestamps, or use an equivalent atomic counter store if the deployment already has a supported Redis primitive. PostgreSQL should be the source of truth for durable accounting; do not derive enforcement from an unbounded conversation-history count on every request.
  • Record the effective limit and the request's organization/trigger in structured logs/metrics, but do not duplicate mutable policy values into conversation metadata.

The quota owner should be the user, not the organization: the Slack concern is power users creating excessive conversations, while the existing conversation metadata already associates every V1 conversation with both user_id and org_id. If product later needs an organization-wide cap, it can be added as a separate policy without changing the per-user counter semantics.

Enforcement

  • Add a reusable DailyConversationQuotaService and invoke it before starting the conversation task, before set_db_session_keep_open and before any sandbox is created.
  • Atomically reserve one slot using a PostgreSQL upsert/update in a transaction (or an equivalent Redis atomic operation). The check-and-increment must be one atomic operation so concurrent tabs, SDK calls, and integrations cannot bypass the limit.
  • Only successful start requests that reach the accepted start-task state should consume a slot. If validation fails before acceptance or the service fails to create the start task, roll back/release the reservation where possible. Decide and document whether retries of the same idempotency key reuse the original reservation rather than counting twice.
  • Return HTTP 429 Too Many Requests with a stable machine-readable error code such as daily_conversation_limit_reached, plus limit, used, and reset_at (next UTC midnight). Avoid exposing other users' usage.
  • Explicitly cover all alternate start paths (SDK/API, webhook/integration routes, and UI) and ensure they use the same service rather than route-specific checks.

Admin/manual increase path

For the first rollout, do not add a user-facing request flow or admin API. Support/engineering can set the per-user override directly in the database. The application must not impose a 20..200 validation range: any integer value is valid so operators can grant a smaller or larger quota, set 0/negative values according to the chosen enforcement semantics, or remove the override by setting it to NULL. Document the database field and provide an operational query/runbook separately if needed.

The initial workflow should allow support/engineering to set a user to 200 or any other required value. If this happens frequently, a later self-service request/approval workflow can be added without changing quota enforcement.

Frontend behavior

  • Handle the stable 429 error code in the conversation creation mutation and show a localized message explaining that the daily limit has been reached and when it resets.
  • Do not rely on a client-side count for enforcement. Optionally expose used, limit, and reset_at from a lightweight authenticated endpoint so the UI can display remaining conversations, but this is not required for the first backend-only rollout.
  • Ensure integrations/SDK clients receive the structured response and do not turn the error into a generic start failure.

Migration, rollout, and observability

  • Add an enterprise Alembic migration for the override/counter schema, with PostgreSQL-only types/operations consistent with enterprise migration rules.
  • In the first database migration, initialize every existing user's per-user override to the then-current configured default value. New users should use NULL to inherit the deployment default unless product explicitly chooses to persist the default at creation time. Usage counters still initialize lazily on the first request for each UTC date; do not backfill historical usage.
  • Keep the enterprise application default unlimited (None) and make the default configurable through the OpenHands/OpenHands-Cloud Helm chart. Set the SaaS deployment value to 20 in OpenHands/saas-deploy. This configuration should support a staged rollout without requiring an enterprise-code change.
  • Add metrics for quota checks, allowed/rejected starts, effective limits, and manual overrides; alert on unexpected rejection spikes and counter/database failures.
  • Decide fail-open vs fail-closed for quota-store outages. Recommended for the initial rollout: fail closed only if the service can safely distinguish a quota violation from an infrastructure error; otherwise fail open with an alert to avoid taking down conversation creation.

Acceptance criteria

  • With the enterprise application default (None), a user with no override is unlimited.
  • With OpenHands/saas-deploy configuring the default as 20, a user with no override can start conversations 1–20 on a UTC day and the 21st accepted start receives HTTP 429 with daily_conversation_limit_reached and a next-midnight reset timestamp.
  • An explicit per-user override takes precedence over the deployment default; an override of 200 allows starts through 200 and rejects the 201st.
  • Explicit per-user overrides accept any integer value and are read directly from the database; NULL falls back to the configured deployment default.
  • Separate users have independent counters; organization membership and switching organizations cannot reset or multiply a user's quota.
  • Concurrent start requests cannot reserve more than the configured limit.
  • Failed/invalid starts do not permanently consume quota slots, and retry behavior is covered by tests.
  • UI, SDK/API, and integration-triggered starts all enforce the same policy and receive the same stable error contract.
  • Support/engineering can set and clear an override directly in the database; no application-level range validation or user-facing override API is required for the first rollout.
  • Tests cover migration/model behavior, quota boundary conditions, UTC rollover, concurrent reservations, override precedence, error responses, and frontend error rendering.

This issue/design was prepared by an AI agent (OpenHands) on behalf of the user.

Review in Linear

Add atomic reserve() and release() methods to DailyConversationQuotaService
using PostgreSQL upsert with a count < limit guard. Wire enforcement into
the conversation creation path: reserve before starting, release on failure.

The 429 response now returns a structured detail object with a human-readable
message pointing to /settings/quota for self-service quota increase.

On the OpenHands frontend, extend retrieveAxiosErrorMessage and type-guards
to extract the message field from structured FastAPI detail objects so the
quota error renders correctly in the existing toast-based error display.
Add a focused regression test for the structured detail extraction.

Stacked on feat/quota-increase-requests (PR #200).

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  enterprise/server/services
  daily_conversation_quota_service.py 45, 103, 111-146, 150-160, 164-167
  openhands/app_server/app_conversation
  app_conversation_router.py 424, 436-452, 461-470, 485-492, 518-525
Project Total  

This report was generated by python-coverage-comment-action

Comment thread frontend/src/utils/retrieve-axios-error-message.ts Outdated
Comment thread openhands/app_server/app_conversation/app_conversation_router.py Outdated
hieptl added 9 commits August 20, 2026 22:32
Main advanced through migrations 148/149 (budget changes), so this
branch's migration reused the already-applied revision id 148. Renumber
it to 150 on top of main's 149.

Also drop the OH_DAILY_CONVERSATION_LIMIT backfill: stamping the
deployment default into every existing user's daily_conversation_limit
would take precedence over org-level limits/exemptions and future
default changes, permanently pinning those users. NULL now always means
'inherit the effective default at runtime'.
Return undefined explicitly from the countdown effect's no-op branch
(consistent-return alongside the interval cleanup return), apply
Prettier formatting to the SaaS-only message and countdown span, and
swap the unsupported pt-BR translations of the quota keys for the
required Catalan (ca) entries.
The revision id '149_org_quota' broke the numeric-prefix/revision match
rule and, together with the quota-request migration, created a second
Alembic head off revision 148. Renumber to a linear 151 on top of the
renumbered 150 so 'alembic upgrade head' resolves a single head.
…se-requests

# Conflicts:
#	frontend/src/i18n/translation.json
- Renumber the quota_increase_request migration to 152 on top of the
  renumbered org-quota migration 151, restoring a single linear Alembic
  chain (main already owns revisions 148/149).
- Allowlist /api/quota/verify in SetAuthCookieMiddleware: the signed JWS
  token is the credential, and the link is opened from the user's work
  email client, usually without an app session. Previously every
  logged-out click got a 401 before the route ran.
- Cap quota increase requests at 10x the org/deployment base default
  instead of 10x the user's current effective limit, so approved
  increases can no longer compound into unbounded self-service
  escalation. The baseline also now respects org-level limits and
  exemptions via DailyConversationQuotaService.get_default_limit.
- Expire pending requests older than the verification-token TTL when the
  user submits a replacement, and add an admin reject endpoint, so a
  lost or expired verification email no longer permanently locks the
  user out of the flow.
- Await resolve_analytics_context (it is async); the PostHog capture and
  work_email person property previously never fired because the sync
  call handed a coroutine to the analytics service and the error was
  swallowed.
- Populate work_email and work_email_verified in /api/quota/status
  instead of returning hardcoded None/False.
- Fix ESLint/prettier errors in the quota frontend files (define
  QuotaIncreaseRequestForm before use, import formatting) and replace
  the unsupported pt-BR translations with the required Catalan entries.
hieptl added 12 commits August 20, 2026 22:49
- Keep retrieveAxiosErrorMessage returning string: the structured-detail
  branch now falls back to error.message instead of null, which had
  widened the return type to string | null and broke compilation at
  existing call sites (query-client-config, changes-tab). Also satisfy
  prefer-destructuring in the new type guard.
- Run quota reserve/release on a dedicated short-lived session instead
  of the request-scoped db_session. The old release path executed and
  committed on a session that could be in a failed-transaction state
  after a start error (masking the original exception, skipping the
  session/httpx cleanup that follows, and leaking the kept-open
  connections) or could commit unrelated pending writes from the
  partially executed start flow. Release is now best-effort with its own
  error handling so the original failure always propagates and cleanup
  always runs.
- reserve() now reports whether a slot was actually consumed, so a
  failed start for an unlimited user no longer triggers a spurious
  decrement.
The enterprise ruff config enforces single-quoted strings; this literal
was the one remaining violation failing the enterprise lint job.
Sort the quota router imports, drop the unused AsyncSession and patch
imports. These were previously masked in CI because the enterprise lint
job failed to build its mypy environment before ruff could report.
…ions

# Conflicts:
#	enterprise/saas_server.py
#	enterprise/server/routes/quota.py
Drop the unused logger import and apply ruff formatting in the quota
routes and org quota test; restore the patch import the upstream merge
removed while this branch's env-clearing test still uses it. These were
previously masked in CI by the enterprise lint job's environment
failure.
…se-requests

# Conflicts:
#	enterprise/server/routes/quota.py
Export QuotaIncreaseRequest via __all__ (matching the module's pattern
and resolving the unused-import error), sort the storage/conftest
imports, wrap the long admin-router registration line, and apply ruff
formatting to the free-email domain test. Previously masked in CI by
the enterprise lint job's environment failure.

@hieptl hieptl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! 🙏

Base automatically changed from feat/quota-increase-requests to main August 24, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feat A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants