Improve ingestion & query service - #75
Open
tekrajchhetri wants to merge 63 commits into
Open
Conversation
Track all mutating actions (ingestion, named-graph registration, crash recovery) as W3C PROV-O triples in a dedicated provenance named graph (https://brainkb.org/provenance/), queryable via SPARQL. Postgres keeps job execution state; Oxigraph is the provenance source of truth. - core/provenance.py: PROV-O builders (IngestionActivity/RegistrationActivity/ RecoveryActivity) with typed user/system agents, plus Graph Store HTTP write and CONSTRUCT->JSON-LD retrieval helpers. Writes are best-effort. - insert.py: emit provenance from run_ingest_job (all terminal states), create_named_graph, and recover_stuck_jobs; stop embedding PROV into domain data (files upload unmodified); add GET /provenance/job and /provenance/named-graph (application/ld+json). - PROVENANCE_MODEL.md: design and reference.
- query_provenance_jsonld: Oxigraph returns HTTP 406 for Accept application/ld+json (it serializes Turtle/N-Triples/N-Quads/RDF-XML only), so request Turtle and convert to JSON-LD locally with rdflib. Keeps the JSON-LD API contract independent of the triplestore's output formats. - construct_for_job: also traverse inbound prov:used so a job's provenance bundle includes the recovery activity (and its system agent) that acted on the job, not just forward links.
Track which triples each ingestion job adds, not just activity-level metadata. Validated end-to-end against a live Oxigraph. - Each job stages triples in a per-job delta graph (https://brainkb.org/provenance/delta/{job_id}), then merges into the target via SPARQL ADD (idempotent set union). The delta graph is preserved as the exact change record; a brainkb:IngestionDelta PROV-O entity records the derivation, target, delta graph, and added triple count. - Gated by TRACK_TRIPLE_DELTAS (default on); disable to upload directly to the target (delta graphs persist, roughly doubling stored triples). - New endpoints: GET /provenance/delta (added triples as JSON-LD), /provenance/delta/history (change history for a graph), /provenance/delta/compare (diff two jobs' deltas: A-only/B-only/shared). - provenance.py: delta_graph_for, merge_delta_into_target, count_graph_triples, construct_delta_content, delta_history_for_graph, compare_deltas; job CONSTRUCT now includes the delta entity. - PROVENANCE_MODEL.md: document the delta model and endpoints.
Registration was recorded twice: once in the named-graph registry graph (metadata/named-graph, via named_graph_metadata) and again as a separate RegistrationActivity in the provenance graph. Both asserted the registration timestamp and made the graph IRI a subject. Keep the registry graph as the single home for registration facts and attribute it to the registering user there (prov:wasAttributedTo); drop the duplicate RegistrationActivity from the provenance graph. - shared.py: named_graph_metadata takes an optional agent_uri and adds prov:wasAttributedTo on the registry entry. - insert.py: create_named_graph passes the agent URI; remove the duplicate build_registration_provenance write (and its import). - provenance.py: remove build_registration_provenance (now unused). - query.py: /query/registered-named-graphs returns registered_by. - PROVENANCE_MODEL.md: document registration living in the registry graph.
Scopes (require_scopes) — GET reads require 'read', mutations require 'write':
- read added: GET /insert/jobs, /insert/user/jobs/detail,
/insert/jobs/check-recoverable, /provenance/job, /provenance/named-graph,
/provenance/delta, /provenance/delta/history, /provenance/delta/compare,
/query/registered-named-graphs
- write added: POST /insert/jobs/recover, POST /register-named-graph
- unchanged: /insert/{raw,files}/knowledge-graph-triples (write),
/query/taxonomy (read), /query/sparql/ (write+admin, arbitrary query),
/register + /token (public)
Docstrings: document the difference between /query/registered-named-graphs
(the registry/catalog of graphs) and /provenance/named-graph (the PROV-O
ingestion/activity history of a graph); expand descriptions on all provenance
and delta endpoints.
Arbitrary SPARQL is a powerful, unrestricted capability; require the 'admin' scope (dropping the redundant 'write'), and keep it off the default 'read' scope used by fixed-shape read endpoints.
Users/teams create owner-controlled spaces, keep them private, or publish them publicly for anyone (incl. unauthenticated clients) to read. Supports a decentralized, IRI-addressable model (https://brainkb.org/space/{slug}). Storage split (confirmed): Postgres holds identity/teams/enforcement (spaces, space_members, space_graphs); Oxigraph holds all KG data + provenance plus a best-effort RDF mirror of each space manifest (metadata/spaces graph). - core/spaces.py: CRUD, per-request authorization (public=anonymous read; private=members; write=owner/editor), and SPARQL-Update RDF mirror. Legacy unmapped graphs fall through to existing scope checks (backward compatible). - core/routers/spaces.py: POST/GET /spaces, GET /spaces/{slug}, PATCH visibility, POST/DELETE members, POST graphs, GET /spaces/{slug}/data (public = anonymous). - security.py: get_current_user_optional (token used if present, never 401) for anonymous public reads. - insert.py: ingestion now enforces space write-authorization on the target graph, in addition to the write scope and user-identity check. - main.py: create spaces tables on startup; mount spaces router. - SPACES_MODEL.md: design + endpoints. Validated end-to-end against the live stack (15/15): private owner ingest/read, outsider ingest/read denied (403), anonymous read denied while private, public flip enables anonymous read + listing, public grants read-not-write, RDF mirror.
Stop leaking private graph existence via the registry listing: graphs in a private space the caller is not a member of are omitted. Public-space and legacy (unmapped) graphs remain listed. The endpoint now loads the caller identity (get_current_user) and excludes hidden graphs via spaces.hidden_graphs_for(). Also clarify in SPACES_MODEL.md that job-scoped provenance is intentionally owner-only and ingestion is always restricted to activated JWT users with valid credentials + space owner/editor membership (never anonymous); only reads of public spaces are anonymous. Validated live: owner sees private+public graphs; non-member sees only public.
- query_service/README.md: document auth/scopes, ingestion + jobs, PROV-O provenance, triple-level delta endpoints, and private/public spaces; add architecture note (Postgres = identity/enforcement, Oxigraph = graph data + provenance). - readme.md: expand the Query Service bullet to mention provenance, deltas, and spaces, with pointers to the model docs.
Search over the knowledge graphs that respects space visibility. Hybrid design: Postgres holds a full-text locator index (graph_search_index: subject, text, named graph, owning space) populated at ingest; a query runs in Postgres (fast, filtered by space visibility/membership) to locate subjects, then the matched triples are fetched from Oxigraph (the source of truth for KG data). - core/search.py: index_graph_subjects (indexes a graph's/-delta's subject literals), reindex_graph_space, and search() with the access filter (anonymous -> public spaces only; authenticated -> public + member spaces + legacy). Data for the located subjects is fetched from Oxigraph as JSON-LD. - main.py: create graph_search_index (GIN full-text) on startup; mount router. - run_ingest_job: index the target graph's subjects after merge (best-effort). - spaces.attach_graph: point existing index rows at the space so search picks up the workspace immediately (inline SQL to avoid an import cycle). - routers/search.py: GET /search (optional auth; space-scoped or full). - READMEs updated. Validated live (9/9): anon finds public term + data from Oxigraph; anon/outsider cannot see private term; owner finds private; scoped search respects membership; anon scoped to a private space returns nothing.
Indexing a large graph inline was slow and delayed ingest jobs. Move it off the ingest path into an in-process async task queue with durable status. - core/indexing.py: asyncio queue + single background consumer, durable index_tasks table, atomic queued->running claim (safe across gunicorn workers), and startup recovery (re-queue tasks left over from a crash). Supports 'ingest' (one graph) and 'backfill' (reindex every user graph, with progress). - main.py: create index_tasks table; start the consumer on startup. - insert.py: ingest now ENQUEUES indexing (non-blocking) instead of awaiting it, so jobs finish without waiting on indexing. - routers/search.py: POST /search/reindex (admin, background backfill) and GET /search/index-tasks (status). - README updated. Validated live: ingest job completes in ~1s while indexing runs in background; background task indexes the subject and it becomes searchable; backfill reindexed 3/3 graphs in the background.
Ingestion stays submit-and-forget/background, but a burst of concurrent submissions could previously run unbounded and exhaust memory / the DB pool / Oxigraph and crash the worker. Add a per-worker asyncio.Semaphore limiter (MAX_CONCURRENT_INGEST_JOBS, default 3): run_ingest_job now acquires a slot before processing; excess jobs return immediately and wait as 'pending' until a slot frees (backpressure without a queue). Effective global cap ~= cap x workers. Validated live: 6 concurrent submissions all accepted in ~0.07s (submit-and- forget intact) and all completed 'done', throttled, no crash.
SPARQL (Oxigraph) + SQL (Postgres) queries to verify ingested graphs, provenance, per-job deltas, spaces manifest/membership, and the search index, with instructions to run via the API or directly against Oxigraph/Postgres.
…I access only Authorization now comes from the user's roles (joined by email to Web_user_profile -> Web_user_role), mapped to capabilities, layered with space membership. JWT scopes remain only an API-access gate. - core/rbac.py: roles->capabilities policy; delegated grants (user_capability_grants); SuperAdmin>=Admin>write>read>none hierarchy; admin-intrinsic caps (grant, sparql_admin) are NOT delegatable (no escalation); query_service never assigns roles (role assignment stays Django-owned). - Space types: 'individual' (any write-capable user) vs 'team' (Admin/SuperAdmin or granted create_team_space). - Enforcement: create space (by type), ingest, recover, arbitrary SPARQL, space management, and reads (no-role -> public content only). - Admin endpoints: GET/POST /admin/capabilities[/grant|/revoke]. - main.py: space_type column + user_capability_grants table. - RBAC_MODEL.md: full model. Validated live (14/14): no-role denied (public read only); Lab Member creates private + ingests but not team; Admin creates team + SPARQL; delegated grant upgrades Lab Member to create team spaces; non-admins can't grant; admin-intrinsic caps rejected for delegation.
Within a space, restrict an action to a global role, a space role, or specific
members — e.g. 'only Admins may write here', 'only these Lab Members may read',
'let this member manage'. Layers on top of capabilities + owner/editor/viewer
membership; owner and global Admin/SuperAdmin always bypass (no lockout).
- space_access_rules table (action, subject_type[global_role|member|space_role],
subject_value).
- spaces.py: rule CRUD, matches_access_rule (pure match) and
space_action_permitted (owner/admin bypass; no-rules -> allow for read/write).
- Enforced on: reads (get space / space data), ingest (insert raw+files), and
manage (members/visibility/graphs, via _can_manage grant).
- Endpoints: GET/POST/DELETE /spaces/{slug}/access-rules (manager only; GET
member/manager).
- RBAC_MODEL.md updated.
Validated live (9/9): write Admin-only rule blocks a Lab Member editor but owner
bypasses; member rule then allows the Lab Member; read Admin-only rule blocks a
member read (owner bypass); manage rule grants a member management.
POST /api/admin/users/activate and /deactivate (by email), Admin-gated, using the existing jwt_user_repo.activate_user/deactivate_user. Enables admins to activate accounts via API (e.g. after password self-registration) instead of only the UI/DB.
…ntial Web_user_profile becomes the single user of record; Web_jwtuser is demoted to a 1:1 credential linked via a new profile_id FK (email backfill for existing rows). Per-service token isolation is preserved (each service keeps its own secret) — this unifies identity, not tokens. - schema: add Web_jwtuser.profile_id (FK -> Web_user_profile, SET NULL, indexed) in the ORM + an idempotent inline migration with case-insensitive email backfill (usermanagement bootstrap). - usermanagement: new provision_identity() as the single path that ensures profile + linked credential + default role (Curator) + bootstrap-superadmin; OAuth callback refactored onto it (drops _ensure_jwt_user_shell + duplicated default-role/bootstrap blocks) and now sets profile_id. - query_service: /api/register now provisions a canonical profile, assigns the default role, and links the credential (best-effort so it never blocks signup) — fixes password users having no roles. - query_service tokens now carry sub/scopes/user_id/profile_id/roles/auth_source to match usermanagement's v2 token shape, still signed with query_service's own secret. Roles stay informational; rbac re-reads them from the DB. - docs: AUTH_UNIFICATION.md design + Phase 1 implementation status. Verified live: migration + backfill, fresh register -> profile+link+Curator, both services' /api/token return the same claim shape, protected endpoints OK.
usermanagement becomes the sole token issuer; a single login mints a short-lived refresh token, exchanged for narrow per-service access tokens (aud=<service>). Services verify via the published JWKS and require their own audience, so a token minted for one service can't be replayed against another (containment enforced by aud, not shared secrets). Additive: legacy HS256 tokens still validate, so this is a safe migration rather than a cutover. usermanagement: - tokens_rs256.py: RS256 key load from env PEM/FILE, else a process-shared ephemeral key persisted to a file (all uvicorn workers agree — per-worker ephemeral keys break cross-worker verification). JWKS builder, refresh/access minting, refresh verification. - routers/sso.py: GET /.well-known/jwks.json, POST /api/auth/login (refresh), POST /api/auth/exchange (per-audience access; roles/scopes re-read fresh from the DB, active + ban checks enforced here). - configuration.py: issuer, private key, TTLs, allowed audiences. query_service: - jwks.py: sync JWKS fetch/cache + RS256 verification requiring iss + aud. - security.py: decode_token_any() tries RS256 (SSO, aud-checked) then legacy HS256; wired into get_current_user(_optional), verify_scopes/require_scopes, and websocket auth. - configuration.py: SSO JWKS URL, issuer, audience. docs: AUTH_UNIFICATION.md Phase 2 status + deployment env + remaining rollout (ml_service/chat_service/MCP, then retire legacy HS256).
Make the RS256 SSO key zero-touch for deployment: the unified container's start.sh generates a persistent key at /app/secrets/um_jwt_private.pem on first boot (only if no key is configured), so the JWKS kid stays stable across the 4 usermanagement gunicorn workers and across redeploys. An explicit USERMANAGEMENT_JWT_PRIVATE_KEY_PEM/_FILE still takes precedence. - Dockerfile.unified: openssl key-gen block in start.sh; exports USERMANAGEMENT_JWT_PRIVATE_KEY_FILE (inherited by supervised processes). - docker-compose.unified.yml: mount ./secrets:/app/secrets so the key persists. - .gitignore: ignore secrets/. - env.template: document that the key is auto-provisioned; override is optional. - AUTH_UNIFICATION.md: update deploy notes.
genpkey already emits PKCS#8; the -pkcs8 flag is invalid and made key generation fail, so the container silently fell back to the /tmp ephemeral key (still shared across workers, but not on the persistent ./secrets volume). Drop -pkcs8 so the key lands at /app/secrets/um_jwt_private.pem and survives redeploys with a stable JWKS kid. Same fix in the env.template hint.
…oped) Extend single-issuer SSO verification beyond query_service so per-audience tokens work across services. Additive — legacy HS256 tokens still validate. usermanagement (now accepts its own SSO tokens): - verify_token() tries an RS256 access token minted for aud=usermanagement (verified with our OWN public key — we are the issuer, no network) before the legacy HS256 v2 token. Flows through get_current_user / require_admin / scopes / ban-check unchanged. - tokens_rs256: add verify_access_token(token, audience) + user_id claim in access tokens; exchange now stamps jwt_user_id. - add "usermanagement" to the exchangeable audiences (config + env.template). ml_service: - new core/jwks.py (httpx, sync) verifies RS256 via the issuer's JWKS and requires aud=ml_service. - decode_token_any() tries RS256 then legacy HS256; wired into get_current_user, verify_scopes/require_scopes, decode_jwt (covers SSE), and the websocket path. - SSO config (JWKS URL, issuer, audience) in configuration.py. Verified live (hot-swap): usermanagement-aud and ml-aud tokens accepted (200); a query_service-aud token is rejected at each (401, containment holds); legacy HS256 tokens still work. chat_service deferred (not in use).
- query_service/README.md: Auth section now documents dual verification (RS256/JWKS SSO with aud=query_service + legacy HS256), and that /register provisions a canonical profile + default role. - usermanagement_service/README.md: document the SSO endpoints (/.well-known/jwks.json, /api/auth/login, /api/auth/exchange), auto-provisioned signing key, and that it accepts aud=usermanagement SSO tokens on its routes. - top-level readme.md / README.md: describe usermanagement as the identity + SSO issuer and add an Authentication section pointing to AUTH_UNIFICATION.md.
… a group)
Previously per-space access rules could only *restrict*, and ingest required
owner/editor membership — so there was no way to let a whole group ingest into a
team space without adding each user individually. Now a write access rule GRANTS
write:
- spaces.can_write_space(space, email): write allowed if global Admin, owner/
editor membership, OR a matching write access rule (global_role / member /
space_role). Returns a reason for clear 403s.
- insert.py: both ingest endpoints use can_write_space instead of the old
membership-only authorize() + restrict-only space_action_permitted() combo
(drops the now-unused authorize import). The INGEST capability (write-capable
role) is still required separately, so a read-only group can't ingest.
So an admin/space-manager can add {action=write, subject_type=global_role,
subject_value="Lab Member"} and every Lab Member can ingest into that space; remove
the rule to revoke.
Docs: query_service/README.md gains a "Capabilities & roles (RBAC)" section
(capability meanings, role→capability mapping, delegation, SuperAdmin vs Admin,
and giving a group ingest access to a team space).
Verified live: rule present → Lab Member ingest 200, non-group 403; rule removed
→ 403.
Adds role/group-level capability grants so an admin can give a custom group (e.g. "uk_collaborator") a global KG capability without per-user grants — the missing piece next to per-user grants and per-space access rules. - new role_capability_grants table (role, capability), created at startup. - rbac: role_granted_capabilities(roles); capabilities(email) now = role-derived caps ∪ role/group grants ∪ per-user grants. grant/revoke/list_role_capability. - spaces admin router: GET /admin/capabilities/available (catalog + which are delegatable), GET /admin/capabilities/role, POST grant-role / revoke-role. Admin+SuperAdmin only; only GRANTABLE_CAPS delegatable (grant/sparql_admin stay admin-intrinsic — no escalation). Verified live: uk_collaborator [read_private] -> grant ingest -> [ingest, read_private]; sparql_admin refused (400).
…ban only)
Enforce SuperAdmin > Admin and make ban (not delete) the removal mechanism.
- Only a SuperAdmin may assign/remove the Admin (or SuperAdmin) role and ban an
Admin account; regular Admins manage non-admin users only. SuperAdmin role
stays fully protected (no strip/ban). Added _is_superadmin/_require_superadmin
(honors the bootstrap-superadmin allowlist).
- User deletion is DISABLED (DELETE /users/{id} -> 405): we don't delete
accounts — ban instead (reversible, preserves provenance/audit history).
- Fix a latent MissingGreenlet in ban_user: build the response from locals
captured before commit instead of touching expired ORM attributes.
Verified live: Admin assign/remove Admin + ban Admin -> 403; SuperAdmin -> 200;
delete -> 405.
…, no-delete (ban) policy
Lets the MCP/skill complete an OAuth login without the web UI. The browser
sign-in (user consent) is unavoidable, but the result is picked up out-of-band
via a short paste-code instead of a frontend redirect.
- Web_oauth_state gains a `mode` ('web'|'cli'); new Web_oauth_cli_result table
(code -> SSO refresh token, single-use, short-lived) + repo.
- POST /api/auth/cli/start {provider} -> authorize URL (state marked cli).
- OAuth callback branches on mode: for cli it provisions as usual, mints an SSO
refresh token, stores it behind a short code, and renders a minimal
"copy this code" page (no SPA).
- POST /api/auth/cli/exchange {code} -> refresh token (reads it inside the
session to avoid MissingGreenlet), single-use.
Verified: cli/start (globus) 200 with authorize URL; exchange of a seeded code
returns the token and reuse is refused (400); success page renders the code.
The real Globus click-through is verified on deploy.
…s) + status update
…al vs per-space, full admin action catalog
…; Admin/SuperAdmin still manage all Previously any manage_team_space holder could manage EVERY team space. Now a non-admin manages a team space only if they own it (created), are matched by a per-space 'manage' rule, or hold manage_team_space AND are a member of that space. Admin/SuperAdmin still manage all. Verified: non-member holder 403; owner 200; admin 200; member holder 200.
… scoped (owned/assigned, not all)
require_admin no longer trusts the token 'roles' claim — it re-reads active roles from the DB (by profile_id/email), so a revoked/demoted admin loses access immediately without waiting for token expiry. Same for the SuperAdmin gate on admin-tier actions (_is_superadmin). Bootstrap-superadmin allowlist still honored for first sign-in. Verified: old token claiming roles=[Admin] -> 200 while Admin in DB; after the Admin role is removed in the DB, the same token -> 403.
…rement is gated on all-clients-on-SSO
…ice, usermanagement
…ess token, role-derived scopes) Lets the web UI swap its usermanagement session JWT (v2 or SSO) for a short-lived aud-scoped access token for query_service/ml_service, with scopes derived from the user's roles (RBAC authoritative). Removes the UI's need for a shared service-account password on those services. Verified: Curator session token -> aud=query_service token (scopes read,write) -> accepted at query_service (200).
…now unblocked (do after deploy test)
Mint an opaque, revocable, time-bounded token once while logged in, set it as BRAINKB_TOKEN in the MCP/skill config, and authenticate with it thereafter with no browser or password. The PAT is stored hashed and validated at usermanagement, then exchanged for the same short-lived per-service access token the login flow issues, so downstream services are unchanged and aud containment is preserved. Roles are re-read live on exchange (instant ban/demotion/revoke). - Web_personal_access_token model + repository - POST/GET/DELETE /api/auth/tokens (session-auth) + POST /api/auth/pat/exchange - env: USERMANAGEMENT_PAT_DEFAULT_DAYS/_MAX_DAYS/_MAX_PER_USER - AUTH_UNIFICATION.md: PAT decision (9.11) + RS256-vs-shared-secret rationale (9.12)
Shorter default PAT lifetime; users may still request up to PAT_MAX_DAYS. Updated code default, env.template, and .env (untracked).
Return your_role / is_owner / access (owner|member|public) / can_write for each visible space so callers can see what they may do in each, not just its existence. (Slug + named-graph IRI uniqueness and no-hard-delete were already enforced.)
…alid user get_user returns False (not None) when there is no active user row, but the callers checked 'is None' — so False slipped through as the authenticated user, breaking _agent/role lookup and yielding a misleading 403 (and, for the optional path, a bogus False instead of anonymous None). Check falsiness / normalize to None in get_current_user, get_current_user_optional, and verify_and_get_user.
Each successful PAT exchange pushes expires_at to now + PAT_DEFAULT_DAYS (the idle window), capped at created_at + PAT_MAX_DAYS, and only ever extends. So an actively-used token never re-prompts, while an unused one lapses after the window. Controlled by USERMANAGEMENT_PAT_SLIDING (default on). Verified: a 1-day token rolled to the 3-day window after one use.
The one-time login paste-code was 8 chars (~39 bits). Raise it to 20 chars over the 30-symbol unambiguous alphabet (~98 bits), grouped in 4s, clamped to fit the String(32) code column, env-configurable via USERMANAGEMENT_CLI_CODE_LEN. It stays short-lived (~10 min) + single-use; the extra entropy is defense-in-depth against brute force in the window.
…d dummy example) Show that USERMANAGEMENT_BOOTSTRAP_SUPERADMIN_EMAILS supports multiple emails and note the runtime path (an existing SuperAdmin can grant the role). Use dummy placeholder emails in the template.
…session The OAuth callback handed the UI a 30-min HS256 token that NextAuth stores and never refreshes, so after 30 min /api/users/me (and the profile/activity routes that reuse the session token) returned 401/404. Make create_access_token_v2 take an expires_minutes override and have the OAuth callback mint the web-session token with USERMANAGEMENT_WEB_SESSION_TTL_MIN (default 720 = 12h). Password-login and per-service tokens keep their short defaults.
…enew The OAuth callback now also mints a longer-lived SSO refresh token (aud=brainkb-auth, USERMANAGEMENT_WEB_REFRESH_TTL_MIN, default 7d) and returns it to the UI as ?refresh=. The UI exchanges it at /api/auth/exchange (audience=usermanagement) to renew its short access token without re-login. create_refresh_token gains an expires_minutes override. Verified: refresh -> exchange -> /api/users/me 200.
Auth unification
Add longer web-session and issue a web refresh token
/api/auth/exchange looked the credential row up with the active-only
jwt_user_repo.get_by_email, so it 401'd "Account inactive" for every
Globus/ORCID/GitHub user. OAuth onboarding creates that row as a SHELL with
is_active=False on purpose — an OAuth user has no usable password, and the shell
exists only to supply a stable user_id claim (see the get_by_email_any_status
docstring, which already says OAuth flows must use it). The result: an OAuth user
could log in and mint a refresh token that nothing would ever accept.
That broke two flows on the same line. The MCP/skill paste-code login
(cli/start -> cli/exchange -> exchange) dead-ended at the last hop, so
brainkb_whoami read authenticated:false right after a successful login and a PAT
could never be minted — minting needs a session token, and the only way to one
from a refresh token is this endpoint. The UI's silent renew goes through the
same call (oauth.py mints web_refresh precisely "exchanged by the UI at
/api/auth/exchange"), so web sessions died at TTL instead of renewing.
Not a bare swap to get_by_email_any_status, because is_active is overloaded:
POST /api/admin/users/deactivate flips the same column, so dropping the check
would make deactivation a no-op here. The refresh token records how it was
issued — auth_source="password" from /auth/login, the provider name from OAuth —
so the check now applies only to password credentials, where is_active really is
the deactivation switch. OAuth accounts are removed by banning, which the
is_banned -> 403 check below already enforces.
Also distinguishes a missing row ("Unknown account") from a switched-off one
("Account inactive"), which were previously the same message.
usermanagement: let OAuth accounts exchange a refresh token
get_current_user verified the token, then looked the caller up with
get_user(email), whose SQL filters `AND is_active = True`. An OAuth caller's
credential row is the SHELL usermanagement provisions with is_active=False — they
have no usable password, the row exists only to carry a stable user_id — so the
lookup found nothing and raised 401 "Could not validate credentials" for every
Globus/ORCID/GitHub user, with a valid, correctly-audienced, correctly-signed
token. Same root cause as the /api/auth/exchange fix in the previous commit, one
service further down.
Worse than a plain error on the optional path: get_current_user_optional swallows
the failure and returns None, so a signed-in OAuth user read those endpoints as
ANONYMOUS. list_spaces answered {"spaces": []} — indistinguishable from "you own
no spaces", and it was reported to a user as exactly that, while authenticated
endpoints 401'd alongside. That combination reads like a token/issuer mismatch and
sent debugging after the wrong thing entirely.
get_user takes include_inactive (default False, so nothing else changes) and the
three token-verification call sites pass it based on the token's own auth_source
claim: relaxed for OAuth, strict for password credentials where is_active is the
switch POST /api/admin/users/deactivate flips. authenticate_user, the actual
password path, is untouched and still refuses inactive rows. Banned accounts are
unaffected — that is enforced separately on the profile.
Readme update
…cope table
A Globus SuperAdmin got 403 "Insufficient scopes" from every query_service admin
route, including /api/admin/capabilities, while usermanagement's own admin routes
worked. Setting USERMANAGEMENT_BOOTSTRAP_SUPERADMIN_EMAILS did not help, and it was
never going to: promote_bootstrap_superadmins assigns the Admin and SuperAdmin ROLES
to a UserProfile and writes nothing else.
Four paths mint tokens, and until now they disagreed about where scopes come from:
/api/auth/session-exchange scopes from roles ("RBAC is authoritative")
/api/pat/exchange scopes from roles
/api/auth/login scopes from Web_jwtuser_scopes
/api/auth/exchange scopes from Web_jwtuser_scopes <- the MCP path
Web_jwtuser_scopes is the legacy Django table, populated only for accounts created
through the old admin. An OAuth account has no rows in it: its Web_jwtuser row is
the shell created to supply a stable user_id claim. So the refresh-token exchange
minted `roles: ["Admin", "SuperAdmin"]` alongside `scopes: ["read"]`.
query_service gates its admin routes on the scope claim — require_scopes(["admin"])
runs as a dependency, before the rbac.is_admin() check inside the handler that would
have passed — and it has no bootstrap-email allowlist of its own
(config.bootstrap_superadmin_emails appears nowhere in query_service). That
combination is why the symptom looked like a missing audience: the same identity
could list users through usermanagement, which honours the allowlist, and could not
read capabilities through query_service, which trusts the token.
Both refresh-token paths now union the stored scopes with the ones the user's roles
imply. Union rather than replacement, because a legacy account may hold an
explicitly granted scope that no role implies and dropping it would be a silent
downgrade. This also removes the need for a PAT as a workaround — the PAT exchange
only worked because it already derived scopes from roles.
Verified on the pure functions: a Globus SuperAdmin with no scope rows now yields
["admin", "read", "write"], a Curator ["read", "write"], a user with no roles
["read"], and a legacy account keeps a scope no role implies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
usermanagement: derive token scopes from roles, not just the legacy scope table
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes several improvements to existing implementations, such as introducing private, public graph/space. Some major changes are: