English | 简体中文
See the example README for usage. This document maintains implementation boundaries, grading contracts, and acceptance requirements.
Tasks integrate with the shared SyncTrainer through program_callable; task data, repository orchestration, and business scoring remain in examples.
| Location | Responsibility |
|---|---|
examples/swe_smith/agent.py |
Repository preparation, Codex lifecycle, call budgets, frozen submissions, grading, and per-call trajectories |
workspace.py |
Pin issue branches and restore trusted tests |
rl/agentic/codex/container_execution.py |
In-container session relay, CLI execution, and output recording shared with Search-R1 |
artifacts.py, reward.py |
Source patch projection, protected tests/configuration, independent pytest grading, and failure attribution |
prepare_data.py, prepare_images.py |
Subset/full-data conversion, task image mapping, architecture and identity checks |
infer.py, configs/ |
Inference without training, model recipes, and fixed nine-case image/task identities |
launch.py, run_config.py |
Local/SSH sandbox selection, paths and data/image identity checks, trusted controller lifecycle |
rl/agentic/codex/, rl/agentic/envs/docker_workspace.py |
Generic Codex protocol, original sampling evidence, local/remote workspaces |
rl/workers/adapters/log_probs.py |
RL-side action-position probabilities, chunked vocabulary projection, and recomputation |
rl/workers/, rl/weight_sync/, rl/trainer/checkpoint.py |
Training construction, streaming publication, graph refresh, and complete recovery |
Candidates check out the instance's problem branch, validate the “Bug Patch → Remove F2P Tests” structure, and resolve pinned commits. Git history is then removed, while the controller retains the captured trusted problem baseline. Containers have no network, Docker socket, or NPU. Management credentials never enter candidates, and the relay is bound only to its session. Candidate, relay, and grader use the same explicit Docker endpoint; remote connection failure never switches to local execution.
Explanatory text and function calls in one Responses reply are combined into one assistant Chat message. Source validation permits decoded text and engine text to differ only by one exact trailing EOS; original sampled tokens remain unchanged. Unclosed XML is a nonexecutable model-format error; other source inconsistencies still reject the session. Once delivery of a budget-termination response is acknowledged, the relay stops accepting new requests and the controller shuts it down. CLI retries cannot reopen the session.
After normal Codex completion or gateway-confirmed budget exhaustion, stop the candidate and export its actual code.
artifacts.py generates the patch on the controller, allowing ordinary Python/Python-stub source changes
while protecting existing tests, project configuration, special files, and execute permissions; candidate Git metadata is untrusted.
New non-Python reproduction inputs, backups, and other intermediate files are excluded. New Python tests/hooks are still checked against protected paths.
The grading version is swe-smith-pytest-v3; frozen artifacts can be regraded independently without regenerating model trajectories.
Task configuration controls artifact byte/file limits.
The grader starts in a fresh container, verifies the same problem/test commits, applies the frozen patch, and restores trusted tests.
Default f2p_only=true selects all F2P and P2P in the same test files; false checks the complete P2P set.
Reward 1 requires trusted statuses for all selected tests, all PASSED/XFAIL. Complete grading failure yields 0.
Independently confirmed submission or syntax errors may yield zero reward. If candidate source prevents collection,
pytest hooks must attribute the exception to modified source, and the original problem baseline must collect every selected test, before assigning zero reward.
Retain actual exceptions and reported statuses; do not fabricate FAILED for unexecuted nodes.
Sampling, container, transfer, unattributable collection, and other incomplete grading failures reject the update.
Supported models are Qwen3.5-9B / Qwen3.8-27B for text GRPO with training TP=CP=1,
colocated native vLLM and streamed full_gather, without the removed consistency field. No BF16 bit-exact claim is made.
See the example README for training and new-process resume commands.
Recovery requires identical model, recipe, and data/image manifest identities. Publish the restored version before resuming sampling and updates.
Full training checkpoint recovery is supported; ongoing Codex sessions and incomplete optimizer steps are not recoverable.
Short-budget recipes use one prompt per rank, two sessions per prompt, at most two calls per session, and 512 output tokens per call.
train.response_mini_batch_size counts per-call training rows. Larger call budgets increase recomputation context,
so check training batch size, device memory, and runtime together. Identical real rewards may produce zero gradients; learning acceptance requires real mixed rewards.
Enable train.learning_gate to check nonzero updates. Default scoring does not inject synthetic rewards.
The unified training entry defaults to HCCL_IF_BASE_PORT=63200 and HCCL_NPU_SOCKET_PORT_RANGE=63200-63299;
rollout uses 63400-63499. Override both training port variables together; the range must lie within 1024..65520,
include the base port, and avoid overlap with rollout. The short 9B recipe explicitly reserves an 8 GiB KV cache to leave room for repeated publication.
The shared Trainer assigns prompts by training DP rank; rollout.num_return_sequences is the session count per prompt.
Within-group GRPO advantages are computed on that rank. Distributing sessions from one prompt group across training DP ranks is currently unsupported.
Inference TP is independent of training sampling groups: vLLM DP1 / TP2 is one two-device inference replica and does not reassign sampling groups automatically.
Two-device FSDP2 needs two assignable prompts. After creating duplicate-question records with prepare_data --repeats 2,
four sessions per rank form two independent four-sample groups, eight sessions total; eight per prompt yield sixteen globally.
Redistribution across training DP ranks belongs in shared RL, preserving unique episode identities, actual rewards,
policy versions, and zero-loss padding.
qwen3_5_9b_infer.yaml
provides a longer inference budget: 100 calls, at most 12,288 output tokens per call, 81,920 context tokens,
5,400 seconds, temperature 1.0, and thinking disabled. The tool protocol is Codex/Qwen3Coder.
max_observation_chars=6000 retains the head and tail of observations delivered to the model and truncates by characters; raw tool evidence is unchanged.
The automatic compaction threshold is 65,536; compaction requests also consume call budget. CLI output limits remain effective.
This recipe does not claim identical prompts, observation wrappers, compaction strategies, or random trajectories to other agents.
The recipe controls independent grader resources and timeouts; external termination is not a solving failure.
Pinned vLLM 0.23.0 / vLLM-Ascend 0.23.0.post1 with rollout DP1 supports explicit FULL_DECODE_ONLY:
Set enforce_eager=false, add the following under rollout.vllm, and retain max_num_seqs=4.
compilation_config:
mode: 3
cudagraph_mode: FULL_DECODE_ONLY
cudagraph_capture_sizes: [1, 2, 4]
max_cudagraph_capture_size: 4This mode includes torch.compile and graph replay. After complete streamed full-gather publication and KV wake,
recapture decode graphs before admitting sampling; a refresh failure must not commit the version.
9B short-budget eager validation covers ARM64 Ascend controllers with local ARM64 or remote amd64 sandboxes, including inference, training, complete saving, and new-process recovery. Task grading covers nine schema, boltons, and oauthlib cases, checking the independent grader against failing problem baselines and successful trusted fixes. Zero-reward, zero-gradient loops prove execution only, not effective learning.
9B graph inference has run. Repeated graph-mode publication/refresh, long-budget updates/recovery with real mixed rewards, complete 27B regression, and full-dataset acceptance still require separate execution. Additional repositories require architecture, branch layout, test dependency, and resource checks. Current patch scope includes only ordinary Python/Python stubs; other languages and tasks requiring build-configuration changes are outside acceptance scope.
Run CPU contract checks from the repository root:
python -m pytest -q tests/ut/rl/agentic/swe_smith \
tests/ut/rl/agentic/codex tests/ut/rl/agentic/test_docker_workspace.pyModel mathematical contracts live in tests/ut/auto_models/models/qwen3_5/; training, publication, and recovery tests reside in corresponding tests/ut/rl/ modules.
Shared code changes must also cover Search-R1, the text gateway, Hermes evidence, and complete checkpoint regression.
CPU success does not establish acceptance for new hardware combinations, long-budget learning, or the full dataset. Before real execution, check devices, image storage,
separate output/checkpoint directories, and budgets.