Skip to content

The audit endpoint silently truncates at 100 exposures #12

Description

@royalpinto007

Problem

GET /audit/document/{doc_id} in app/main.py is described in its own docstring as the reason the audit tables exist:

On the day someone asks "was this leaked", you want a query, not an archaeology project.

Its query ends with a hardcoded LIMIT 100, and the response says nothing about it:

ORDER BY q.asked_at DESC
LIMIT 100
...
return {"doc_id": doc_id, "exposures": [...]}

So a document surfaced 500 times returns 100 rows and looks exactly like a document surfaced 100 times. There is no total, no truncated flag, no cursor, and no way to ask for the next page.

Why it matters

Every other refusal in this codebase is explicit and reasoned: generate returns a refusal_reason, retrieval fails closed on an unknown principal, ingest warns loudly on an empty ACL. This endpoint quietly gives an incomplete answer to the one question the feature exists to answer, and the caller is a compliance or incident responder who has no way to know the answer is partial. An incident response that concludes "only these 100 users saw it" from a silently truncated list is worse than having no endpoint.

There are two smaller problems in the same handler: it takes no time range, which is the first filter anyone actually wants, and it has no caller identity check (tracked separately in #5), so the exposure log itself is readable by anyone.

Suggested approach

  1. Return the true total alongside the page: a COUNT(*) over the same predicate, exposed as total_exposures.
  2. Add keyset pagination on (asked_at, id) with limit and before query parameters, defaulting to the current 100 and capped at something sane. Keyset rather than offset, because the audit table only grows.
  3. Add optional since and until filters on q.asked_at.
  4. Add an index supporting the access pattern. The query filters retrievals.doc_id and orders by queries.asked_at, so check app/schema.sql covers it; without one this is a scan of the whole retrievals table joined to queries.
  5. Mirror the same shape in the CLI if it exposes this view (app/cli.py).

Done when

  • The response states the true total and whether the page is partial.
  • A caller can page through the full history deterministically.
  • A time range can be requested.
  • Tests cover: fewer than one page, exactly one page, more than one page, and a since filter.

If you want to take this on, comment on the issue to claim it and it will be assigned. Please keep to a maximum of 2 open claims per person at a time so other contributors get a chance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions