Skip to content

feat(rl): compose IcePop with score centering - #4012

Merged
hallerite merged 19 commits into
mainfrom
exp/icepop-sc-main-5k
Oct 20, 2026
Merged

hallerite merged 19 commits into
mainfrom
exp/icepop-sc-main-5k

Conversation

@hallerite

@hallerite hallerite commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Adds opt-in score centering to IcePop (trainer.loss.score_centering = true), using captured top-k sampler probabilities and a proportional tail approximation.

  • Bumps verifiers to 512b8f7 for sampler probability transport through traces and training branches.
  • Bumps renderers to 70c8204 for top-logprob capture and compact response parsing.

Note

High Risk
Touches RL loss gradients, inference logprob transport, and training data validation on the critical policy–inference mismatch path; misconfiguration or missing top-k evidence fails hard but wrong rollout data could skew updates.

Overview
Adds opt-in score centering for IcePop (trainer.loss.type = "icepop" + score_centering, default top-k 128). The launcher enforces untruncated stochastic policy sampling, bumps sampling.logprobs and vLLM processed_logprobs / max_logprobs, and documents the approximate head+tail centering math and metrics.

End-to-end sampler top-k evidence is new: compact binary logprob payloads from vLLM, TopLogprobs on traces/micro-batches, validation in the trainer, and fused LM-head top-k logprobs (with gradients) feeding an IcePop correction that still applies when the sampled token is ratio-rejected.

The Qwen tool example switches agent runtime from Modal to Prime sandboxes (README + tool.toml). Lockfile bumps prime-sandboxes and drops unused Modal/e2b extras; a GLM FLOP test fixture sets dsa_backend="tilelang" for CI.

Reviewed by Cursor Bugbot for commit b3d985a. Bugbot is set up for automated code reviews on this repo. Configure here.

@hallerite
hallerite force-pushed the exp/icepop-sc-main-5k branch from 42017ed to 0d77d4c Compare October 10, 2026 23:13
@hallerite hallerite changed the title exp: IcePop + score centering on latest main for 5k ScaleSWE run feat(rl): compose IcePop with score centering Oct 10, 2026
@hallerite
hallerite force-pushed the exp/icepop-sc-main-5k branch from 0d77d4c to da79604 Compare October 10, 2026 23:18
@hallerite
hallerite marked this pull request as ready for review October 10, 2026 23:38
hallerite added a commit to PrimeIntellect-ai/verifiers that referenced this pull request Oct 19, 2026
)

Carry sampler probability heads from renderer responses onto assistant
nodes and training branches for [prime-rl
#4012](PrimeIntellect-ai/prime-rl#4012), paired
with [renderers
#180](PrimeIntellect-ai/renderers#180).

`TopLogprobs` is a flat IDs/logprobs/counts carrier. It shares the
existing raw-array serializer with sampling masks, checks sampled-token
alignment when attributed, and gives context or uncaptured positions
zero counts when flattening a branch. Public trace records exclude the
arrays. Empty completions preserve an empty carrier.

Validation: 179 unit tests passed; a temporary probe covered multi-turn
alignment, mixed capture, empty completions, malformed rows/counts,
MessagePack round trips, and record exclusion. All contributor hooks and
the configured type gate passed using uv 0.13.1. The full suite was
attempted: four Docker Compose integration cases failed because the
local Docker daemon is unavailable; provider/runtime cases were skipped
where prerequisites were absent.

Updated onto current main without rewriting the existing PR history.

Co-authored-by: hallerite <git@hallerite.com>
Comment thread src/prime_rl/entrypoints/rl.py Outdated
@hallerite
hallerite force-pushed the exp/icepop-sc-main-5k branch from cb33b75 to f186048 Compare October 20, 2026 12:44
Comment thread src/prime_rl/inference/vllm/serving_tokens.py
@hallerite
hallerite merged commit 90fbb8d into main Oct 20, 2026
18 of 20 checks passed
@hallerite
hallerite deleted the exp/icepop-sc-main-5k branch October 20, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants