Skip to content

Keep feature category alignment in benchmarks - #986

Closed
cedric-lorenz wants to merge 1 commit into
mainfrom
fix/beyondarena-category-alignment
Closed

cedric-lorenz wants to merge 1 commit into
mainfrom
fix/beyondarena-category-alignment

Conversation

@cedric-lorenz

@cedric-lorenz cedric-lorenz commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

BeyondArena failed with Expected the category vocabulary ... to match because train and test features used independent categorical vocabularies.
Keep AlignCategories in the feature pipeline to map query categories onto the fitted training vocabulary before categorical processing.

@copy-pr-bot

copy-pr-bot Bot commented Sep 25, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@cedric-lorenz
cedric-lorenz force-pushed the fix/beyondarena-category-alignment branch from ae30587 to 335ab3c Compare September 25, 2026 21:16
@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: NVIDIA/structured-data-models/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: ddb41a9a-57c6-450d-9175-e5bb60ecf695

📥 Commits

Reviewing files that changed from the base of the PR and between 5e0a22e and 335ab3c.

📒 Files selected for processing (1)
  • benchmark/tabular/model.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Summary

Summary by CodeRabbit

  • Bug Fixes
    • Corrected categorical preprocessing in tabular models so category alignment changes apply only to target data; feature processing remains unchanged.

Walkthrough

_create_recipe now removes the leading AlignCategories processor only from the target pipeline. The feature pipeline is no longer modified by this method.

Changes

Tabular recipe

Layer / File(s) Summary
Categorical processor adjustment
benchmark/tabular/model.py
_create_recipe removes AlignCategories from the target pipeline while retaining it in the feature pipeline.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 335ab

The change preserves feature category alignment for benchmark inputs while avoiding unnecessary target alignment. No actionable merge-blocking risk was found.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ❓ Inconclusive No pull request description was provided, so the change context and validation details are not documented. Add a brief description that explains the _create_recipe change, its effect on feature and target categorical processing, and the relevant validation performed.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main change: preserving feature category alignment in benchmarks.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant