Skip to content

feat: index posts, releases and changelog in search - #31

Open
mattdjenkinson wants to merge 1 commit into
feat/auditfrom
feat/search
Open

mattdjenkinson wants to merge 1 commit into
feat/auditfrom
feat/search

Conversation

@mattdjenkinson

Copy link
Copy Markdown
Contributor

Makes posts, releases and changelog entries searchable in Milo's search service. PostQuery's text search depends on it, and it also makes feedback findable through platform search. Part of #12. The issue stays open until the index lag is measured in staging, which needs the deploy in #13.

Stacked on #30 (audit), which is on #29 down to #16.

Index policies (config/milo/search)

A ResourceIndexPolicy for each of Post, Release and ChangelogEntry, included in config/milo:

Kind Searchable
Post title, summary, body, categories, labels, state
Release title, codename, version, description
ChangelogEntry title, body, categories
  • Not searchable: votes, trending score and delivery change as side effects and would go stale in the index, so they aren't searchable fields, and neither are the derived post lists on releases and changelog entries. The indexer still stores the whole object, but search only matches on these fields.
  • Hidden posts: the condition !has(spec.hidden) || !spec.hidden keeps hidden posts out. I checked the search indexer: an update that stops matching deletes the document, so hiding removes a post from the index. Re-indexing applies the same condition.
  • Never indexed: submissions, security reports, votes and subscriptions.

How it connects

  • Indexing: the indexer builds documents from the responseObject in audit events, which the apiserver now writes at RequestResponse for these kinds (feat: audit feedback writes from the apiserver #30).
  • Tenant: posts carry no scope annotations, so they're indexed in the platform tenant. The apiserver queries search with its own platform identity, which sees them.
  • Permissions: search does no per-result authorization (it scopes by tenant), so the apiserver's existing create resourcesearchqueries grant (feat: register feedback with Milo IAM #29) is enough.
  • Re-indexing: search lists posts as itself, through Milo, using the paged list our Postgres store supports.

Measuring lag

hack/search-lag.sh <post> takes a visible post and times two things:

  • hiding it until search stops finding it;
  • unhiding it until search finds it again.

It restores the post on exit. Hiding is briefly visible to users, so run it in staging after #13.

Testing

  • TestSearchPolicies resolves every field path against the Go types by JSON name. A misspelled field fails, as checked by misspelling codename. It also keeps derived fields and every kind but the three out.
  • TestHiddenPostCondition compiles the condition in the same CEL environment the indexer declares (metadata, spec and status as dynamic values), and checks that a visible post is indexed and a hidden one isn't.
  • Not run: a kind run would need Milo, NATS, Vector and Meilisearch, so the end-to-end check (a post write appears in search and PostQuery text returns it) is part of the staging work in Deploy to staging #13.

PostQuery's text search and platform search need feedback in Milo's search
service, which indexes resources from the audit events the apiserver now
writes.

- ResourceIndexPolicies for Post, Release and ChangelogEntry make titles,
  summaries and body text, categories, labels and post state searchable.
  Votes, trending, delivery and derived post lists stay unsearchable since
  they change as side effects.
- A CEL condition keeps hidden posts out; the indexer removes a post from
  the index when an update makes it hidden.
- Tests resolve every field path against the Go types, keep derived fields
  and private kinds out, and evaluate the hidden-post condition the way the
  indexer does.
- hack/search-lag.sh measures how long a change takes to reach the index,
  for staging.

Part of datum-cloud/enhancements#943 and #12; #12 stays open until the
index lag is measured in staging.
@mattdjenkinson mattdjenkinson self-assigned this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant