Repository navigation
feat(rl): compose IcePop with score centering - #4012
Merged
Merged
Conversation
hallerite
force-pushed
the
exp/icepop-sc-main-5k
branch
from
October 10, 2026 23:13
42017ed to
0d77d4c
Compare
hallerite
force-pushed
the
exp/icepop-sc-main-5k
branch
from
October 10, 2026 23:18
0d77d4c to
da79604
Compare
hallerite
marked this pull request as ready for review
October 10, 2026 23:38
Preserve main packed sampled-logprob and sampling-mask transport alongside SC top-k evidence. Refresh renderer and verifier pins, retaining the tested loss math.
hallerite
added a commit
to PrimeIntellect-ai/verifiers
that referenced
this pull request
Oct 19, 2026
) Carry sampler probability heads from renderer responses onto assistant nodes and training branches for [prime-rl #4012](PrimeIntellect-ai/prime-rl#4012), paired with [renderers #180](PrimeIntellect-ai/renderers#180). `TopLogprobs` is a flat IDs/logprobs/counts carrier. It shares the existing raw-array serializer with sampling masks, checks sampled-token alignment when attributed, and gives context or uncaptured positions zero counts when flattening a branch. Public trace records exclude the arrays. Empty completions preserve an empty carrier. Validation: 179 unit tests passed; a temporary probe covered multi-turn alignment, mixed capture, empty completions, malformed rows/counts, MessagePack round trips, and record exclusion. All contributor hooks and the configured type gate passed using uv 0.13.1. The full suite was attempted: four Docker Compose integration cases failed because the local Docker daemon is unavailable; provider/runtime cases were skipped where prerequisites were absent. Updated onto current main without rewriting the existing PR history. Co-authored-by: hallerite <git@hallerite.com>
samsja
reviewed
Oct 20, 2026
hallerite
force-pushed
the
exp/icepop-sc-main-5k
branch
from
October 20, 2026 12:44
cb33b75 to
f186048
Compare
mikasenghaas
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds opt-in score centering to IcePop (
trainer.loss.score_centering = true), using captured top-k sampler probabilities and a proportional tail approximation.512b8f7for sampler probability transport through traces and training branches.70c8204for top-logprob capture and compact response parsing.Note
High Risk
Touches RL loss gradients, inference logprob transport, and training data validation on the critical policy–inference mismatch path; misconfiguration or missing top-k evidence fails hard but wrong rollout data could skew updates.
Overview
Adds opt-in score centering for IcePop (
trainer.loss.type = "icepop"+score_centering, default top-k 128). The launcher enforces untruncated stochastic policy sampling, bumpssampling.logprobsand vLLMprocessed_logprobs/max_logprobs, and documents the approximate head+tail centering math and metrics.End-to-end sampler top-k evidence is new: compact binary logprob payloads from vLLM,
TopLogprobson traces/micro-batches, validation in the trainer, and fused LM-head top-k logprobs (with gradients) feeding an IcePop correction that still applies when the sampled token is ratio-rejected.The Qwen tool example switches agent runtime from Modal to Prime sandboxes (README +
tool.toml). Lockfile bumpsprime-sandboxesand drops unused Modal/e2b extras; a GLM FLOP test fixture setsdsa_backend="tilelang"for CI.Reviewed by Cursor Bugbot for commit b3d985a. Bugbot is set up for automated code reviews on this repo. Configure here.