Permission-aware RAG. Access control is enforced inside the retrieval query, not after it.
$ vaultrag ask "what is the quarterly bonus payout policy" --user alice
asking as alice, principals: ['user:alice', 'group:engineering']
retrieved (only what this user may see)
doc title score
eng-onboarding Engineering Onboarding 0.0328
all-hands All Hands Notes 0.0161
conflict eng-onboarding vs all-hands: an official and an informal document both answer this
stale old-handbook last updated 2431d ago (owner: None)
The quarterly bonus for engineering is 10% of base salary, paid the month after quarter end.
sources: eng-onboarding
$ vaultrag ask "what is the quarterly bonus payout policy" --user bob
asking as bob, principals: ['user:bob', 'group:sales']
# ...same question, different corpus. bob never sees engineering's answer, and the CEO's
# private note is invisible to both of them.The usual RAG pipeline is: embed the question, retrieve top-k, filter out what the user is not allowed to see, generate an answer. That is a leak waiting to happen.
By the time you filter, the unauthorized chunks are already in your process. They can land in a log line, a trace span, an error report, or a prompt you assembled one step too early. And a top-k of 5 that filters down to 1 silently degrades the answer with no signal that it happened.
VaultRAG puts the ACL predicate in the same SQL query as the vector search and the keyword search. A chunk the user cannot see is never selected, never scored, never ranked, never logged. It cannot leak, because it was never fetched.
"Permissions are enforced at retrieval" is a slogan until there is a number attached. vaultrag eval runs a gold set of (user, question, what-they-should-and-should-not-see) against a real
corpus and reports two metrics that only mean something together:
- leak rate: did anything the user may not see surface. One leak is a failure, not a percentage point. Target: exactly zero.
- recall: of the documents they may see that answer the question, how many made it back.
The pairing is the whole design, because each is trivial to fake alone. Retrieve nothing and you score a perfect 0% leak rate. Retrieve everything and you score perfect recall.
$ vaultrag eval demo/gold.json --strict
cases 11
leak rate 0.0%
mean recall 100.0%
pass rate 100.0%
refusal correctness not measured (stub LLM)
Delete the ACL predicate and re-run:
$ vaultrag diff before.json after.json
LEAK
leak rate: 0.0% -> 81.8%
mean recall: 100.0% -> 100.0%
Read that second line. Recall did not move. The broken build answers every question correctly
and completely, while handing alice the CEO's private notes and HR's salary bands. A quality-only
eval scores that build perfect. This is why leak rate is never reported on its own, and why CI
fails on --strict rather than on a threshold.
Refusal correctness is deliberately reported as not measured under the stub LLM. Whether the model declines when it has no evidence is a property of the model, and scoring a stub on it would launder a fake into a metric. CI runs the two metrics it can actually prove.
The harness earned its keep before it was even committed:
-
The refusal threshold was unreachable.
weak_evidencerefused anything scoring under 0.02. With RRF at k=60, a chunk found by a single arm scores at most 1/61 = 0.0164, which is below the floor. Every document only one arm found was refused no matter how perfectly it ranked. The threshold is now derived from the RRF constant rather than picked by hand. -
The test suite deleted the demo corpus.
conftest.pyreset the schema of whateverDATABASE_URLpointed at, which is the database you have exported. The eval then reported 0% leaks and a 100% pass rate against an empty corpus. The most convincing wrong answer the tool can produce is the one that says everything is fine.test_empty_corpus_does_not_score_greennow pins it, and the tests choose their own database.
Every document in the test corpus contains the phrase "quarterly bonus payout policy". A retriever without access control would happily hand the CEO's private notes to anyone who asks about bonuses. The only thing preventing that is the ACL predicate.
tests/test_acl.py asserts it, 12 ways:
test_group_isolation_alice_cannot_see_sales_or_hr
test_group_isolation_bob_cannot_see_engineering_or_hr
test_multi_group_user_sees_union_not_everything
test_nobody_sees_the_ceo_document
test_empty_acl_is_deny_by_default
test_user_level_grant_works_without_any_group
test_shared_document_reaches_every_permitted_group
test_revoking_acl_takes_effect_immediately
test_removing_user_from_group_takes_effect_immediately
test_soft_deleted_document_is_unreachable
test_unknown_user_resolves_to_nothing
test_groups_are_not_taken_from_the_caller
These tests run against a real Postgres, not a mock. Mocking the database would mean mocking the thing under test: the tests would pass while the real query leaked.
They fail when the invariant breaks. Delete the ACL predicate from retrieval.py and 9 of the
12 fail immediately with LEAK: assertions. That is the difference between a test suite and
decoration.
Retrieval decides what the model may read. vaultrag/generate.py decides what it may say.
- Weak evidence is a refusal. Top-k always returns k things. If the best of them is barely related, answering from it produces a confident irrelevant answer.
- Citations are verified, not trusted. Asking a model to cite is not the same as it citing correctly. Claimed citations are checked against the chunks we actually handed it; invented ones are dropped and recorded. An answer whose only citation was invented is not served.
- Unparseable output refuses. If we cannot parse it, we cannot verify it.
- Conflicts are surfaced, not resolved. Two sources disagree, say so and cite both.
tests/test_generate.py covers all of it, including the hallucinated-citation case.
Two failures that no prompt can fix, because they are properties of the corpus rather than the model:
- Conflicts. Two documents the user can see answer the same question. Silently picking one and
sounding confident is the worst available behaviour, because nobody can tell it happened.
detect_conflictsflags it and names the owners, since "these disagree" is not actionable but "ask these two people" is. - Staleness. A document nobody has touched in two years is still retrievable and still answers with total confidence. Most "the AI gave me wrong information" incidents are really "the wiki was wrong and nobody noticed for a year".
$ vaultrag health
live documents 10
stale documents 1
stale pct 10.0
ownerless documents 2
orphaned no acl 1
orphaned_no_acl is the one to watch: a document with no ACL rows is retrievable by nobody. It is
technically the safest document in the corpus, and it is almost always someone who forgot the ACL
and is about to file a bug saying search is broken.
Deny by default. A document with no ACL rows is visible to nobody. Fail-open here means one ingestion bug silently publishes a document to the whole company.
Groups are read from the database, never from the request. If a caller could assert its own group membership, the ACL would be decorative.
Access is evaluated per query, not baked into the index. Revoking access takes effect on the next question. If revocation needed a reindex, every ACL change would race the pipeline.
Hybrid retrieval. Vector search is good at "what's our policy on remote work" and bad at
ERR_4021 or "Policy 7.3". Keyword search is the opposite. Real questions contain both, so both
run and the rankings are fused with RRF.
Chunking splits on structure, not a token count. Fixed-size chunks cut sentences in half and strand a subject from its predicate, which embeds badly and retrieves badly. Headings are a free signal the author already provided.
Everything retrieved is audited. queries and retrievals record who asked what and which
documents were actually surfaced, so "did we ever leak document X to user Y" is a query rather
than an archaeology project.
| Failure | Mitigation |
|---|---|
| Retrieve-then-filter leaks | ACL predicate is inside the retrieval CTE |
| Missing ACL publishes a doc to everyone | Deny by default; ingest warns loudly on an empty ACL |
| Caller forges group membership | Groups resolved from the DB, not the request |
| Revocation lags behind the index | Access evaluated per query |
| Deleted doc still answerable | Soft delete filtered in retrieval |
| Exact terms (error codes) not found | Hybrid keyword arm |
| Two sources disagree | Official + most-recent wins the tie-break; detect_conflicts surfaces it |
| The corpus rots and answers stay confident | detect_stale / vaultrag health |
| An ACL regression ships quietly | Gold-set eval fails CI on any leak |
| Security tightened until nothing is retrievable | Recall reported next to leak rate, always |
FastAPI, Postgres + pgvector, sentence-transformers locally (free, no API key), Groq for
generation. Zero cost to run.
| Method | Path | |
|---|---|---|
| POST | /ask |
Answer a question using only documents this caller may see |
| POST | /documents |
Index a document + its ACL (idempotent) |
| DELETE | /documents/{id} |
Soft delete: unreachable immediately, audit trail kept |
| GET | /audit/document/{id} |
Who has this document ever been surfaced to? |
| GET | /health |
Note what /ask does not accept: a groups field. The caller says who it is; the server looks
up what that means. If a client could assert its own groups, the ACL would be decorative.
/audit/document/{id} exists because on the day someone asks "was this ever leaked", you want a
query, not an archaeology project.
pip install vaultrag-cliPublished at pypi.org/project/vaultrag-cli.
vaultrag seed # load the demo corpus (10 docs, 5 users, 6 trust boundaries)
vaultrag ask "..." --user alice # ask as a specific person
vaultrag eval demo/gold.json --strict # leak rate + recall; exits 1 on any leak
vaultrag diff before.json after.json # did my change leak or regress?
vaultrag health # corpus freshnessdocker compose up -d db # Postgres 16 + pgvector on :5433
python -m venv .venv && ./.venv/bin/pip install -e ".[dev]"
PYTHONPATH=. ./.venv/bin/python -m pytest -q # 56 tests, no secrets needed
vaultrag seed && vaultrag eval demo/gold.json --strict
# serve it
EMBEDDER=fake LLM=fake ./.venv/bin/python -m uvicorn vaultrag.main:app --reload
# -> http://localhost:8000/docsTests need no API keys: they use a deterministic offline embedder and a scripted LLM, because access control is not a semantic question and shouldn't need a 90MB model download to verify.
For real use, set EMBEDDER=local (sentence-transformers, free, no key) and LLM=groq with a
GROQ_API_KEY. Zero cost either way.
Done and green: schema, ACL model, hybrid retrieval, ingestion, generation guardrails, the HTTP API, the audit trail, the eval harness, corpus-health signals, and a CLI. 56 tests passing against real Postgres + pgvector, plus an 11-case gold set at 0% leak rate, both gating CI.
Not done: a reranker model (a score blend is used instead), a UI, and refusal correctness measured against a real LLM rather than reported as unmeasured.