feat(context): Add priority_select for budget-aware section selection

[patelchaitany]

What this PR does / why we need it:

Stacked on #6824. GitHub cannot base a pull request on a branch that lives in a fork, so this targets master and carries #6824's commit along with it. The only new commit here is feat(context): Add priority_select for budget-aware section selection — review that one. Once #6824 merges, this diff collapses to priority_select.py, its tests, and two lines of exports.

Adds priority_select(), the second piece of feast.context: it decides which parts of a prompt survive a token budget. Prompt assembly is a packing problem — a user profile, a watch history and a catalogue of candidates rarely fit together in a context window — and this is the mechanism that chooses what gives.

selection = priority_select(
    [
        ("critical", system_instructions),
        ("high", f"User profile: {profile}"),
        ("medium", f"Recent history: {history}"),
        ("low", f"Device: {device}"),
    ],
    TokenBudget.of(4096),
)
prompt = selection.render()

Sections are considered most important first, and dropped least important first. Priority is an IntEnum (critical > high > medium > low), spaced by ten so a caller can slot a level in between. Ties keep declaration order. A section too large to fit is dropped and the scan continues, so a small low-priority section is still kept after a large high-priority one was refused. A critical section is never dropped: if it does not fit, RequiredSectionError is raised rather than quietly returning a prompt with its instructions missing. That error subclasses TokenBudgetExceededError, so callers already guarding assembly keep catching one type.

The separator is charged against the budget. Counting sections alone and then joining them with "\n\n" overshoots the allowance by one token per joint, which is exactly the sort of drift that turns into a 400 from a model server. priority_select charges one separator between each pair of kept sections, so Selection.render() fits what was budgeted. Tokenizers may merge text across a boundary, which only makes the charge an upper bound.

The result is a value, not a list of strings. Selection reports what was kept, what was dropped, and the budget left over — the last of which matters when a caller wants to spend the remainder elsewhere. It iterates over the kept content and stringifies to the rendered prompt, so it still drops straight into a join or a template:

selection.contents      # ("You are...", "User profile: ...")
selection.dropped       # (Section("Device: ...", Priority.LOW),)
selection.is_complete   # False — something was left out
selection.budget        # TokenBudget(41/4096 tokens used, ...)

Kept sections come back in declaration order by default, so the prompt reads as it was written; order=SectionOrder.PRIORITY returns them most important first instead. Sections given blank content are ignored rather than rendered as a doubled separator.

Determinism. Selection is a pure function of the sections and the budget — no clocks, no iteration over unordered collections, no mutation of the budget passed in. The same ODFV therefore drops the same sections during training as it does at serving time, which is what AC-6 of the epic asks for.

One change outside the new file: TokenBudgetExceededError gained a keyword-only message override so RequiredSectionError can name the offending section while keeping the same type and attributes. It is additive and backwards compatible.

Which issue(s) this PR fixes:

No GitHub issue. Tracked internally as RHOAIENG-80116.

Checks

• I've made sure the tests are passing.
• My commits are signed off (git commit -s)
• My PR title follows conventional commits format

Testing Strategy

• Unit tests
• Integration tests
• Manual tests
• Testing is not required for this change

sdk/python/tests/unit/context/test_priority_select.py — 48 tests with tiktoken installed, 47 passing and 1 skipped without it. Covers priority parsing and ordering, section normalization from tuples, dropping order, critical sections listed last, keeping a smaller section after refusing a larger one, separator accounting (including the assertion that the rendered prompt costs exactly what was charged), an already-consumed budget, blank and empty input, and the error's shortfall reporting. One exact-tokenizer test asserts a rendered prompt fits a 64-token cl100k_base budget.

Misc

Nothing in Feast imports feast.context yet, so the change remains additive and inert. ruff check, ruff format --check and mypy are clean.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.08%. Comparing base (81e1546) to head (9bca2fb).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6826   +/-   ##
=======================================
  Coverage   47.08%   47.08%           
=======================================
  Files         419      419           
  Lines       51877    51877           
  Branches     7525     7525           
=======================================
  Hits        24428    24428           
  Misses      25700    25700           
  Partials     1749     1749           

Flag	Coverage Δ		*Carryforward flag
go-feature-server	30.58% <ø> (ø)
python-unit	48.39% <ø> (ø)		Carriedforward from dd7b805

*This pull request uses carry forward flags. Click here to find out more.

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 81e1546...9bca2fb. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[patelchaitany]

Closing this one: it targets master and therefore carries #6824's commit along with it, which is not a genuine stacked review. I'll reopen the priority_select change against master as a single self-contained commit once #6824 has merged. The branch is retained, so no work is lost.