Report harness timeouts as TIMED_OUT, reply from daemon outer handler, add stall/timeout regression tests - #7
Open
davidoj wants to merge 1 commit into
Open
Conversation
…, add regression tests
A parent-side daemon timeout ("Daemon timed out") was reported as FAILED on
the secure path and CRASHED on the insecure path, indistinguishable from a
wrong answer. It now maps to VerificationStatus.TIMED_OUT on both paths so a
caller can drop (not zero) timed-out grades. Only `subprocess_error` is
classified, which submission code can never write. calc_reward is unaffected
(1.0 iff PASSED).
Both daemon loops now send a best-effort "Daemon error" reply from the outer
exception handler instead of only logging, so an unexpected exception costs
one request rather than the parent's whole budget.
Tests: test_daemon_stall.py (dead child via os._exit / SIGKILL on both paths,
runaway code still bounded) guards the #5 stall fix; test_timeout_budget.py
guards warmup-stable verdicts and the TIMED_OUT mapping. Insecure-path tests
skip on platforms where the daemon child cannot set RLIMITs (macOS).
Supersedes #3 and #4.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rSFHpdtkVYpGAEafamz5R
This was referenced Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #3 and #4, whose main fixes landed in #5. This carries only the pieces #5 did not cover, rebased on current
main.What #5 already covered
DJINN_MAX_TOTAL_TIMEOUT(default 300) andDJINN_SECURE_TIMEOUT_PER_TEST(default 6). Don't score slow-but-correct submissions as wrong #3's warmup-nondeterminism test passes onmainunmodified.EOFError/OSErrorcaught aroundrecv()in both daemon loops,request_idstamped on every reply. All of Reply when a child dies instead of stalling the parent for the full budget #4's tests pass onmainunmodified.What this PR adds
1. A blown budget is
TIMED_OUT, notFAILED/CRASHEDOn
maina parent-side "Daemon timed out" comes back asFAILEDon the secure path (timeout text buried in per-test feedback) andCRASHEDon the insecure path — indistinguishable from a wrong answer. #5's description says training should drop rather than zero a timed-out grade, but a caller could only do that by string-matching feedback. Both paths now returnVerificationStatus.TIMED_OUT(already defined; already what the insecure verifier's own alarm reports).Only
subprocess_erroris classified. That key is written by the daemon loops and the parent, never by submission code, so a submission cannot spoof a timeout via its own exception text. Per-test alarm timeouts inside a batch still score that test as failed, unchanged.calc_rewardis unaffected: it returns 1.0 iffPASSED, soTIMED_OUTscores exactly asFAILED/CRASHEDdid.verify_problem_consistencycompares againstPASSEDonly.2. Daemon outer exception handler replies instead of only logging
The one piece of #4 not in
main. Theexcept Exceptionat the end of each daemon loop logged and sent nothing, so any unexpected exception (fork failure, pipe setup, …) cost the parent its whole budget. It now sends a best-effortDaemon error: …reply; the send is itself guarded, so a gone parent still yields only a log line.3. Regression tests (#5 merged without any)
djinn/tests/test_daemon_stall.py(from Reply when a child dies instead of stalling the parent for the full budget #4): dead child viaos._exitand SIGKILL on both paths returns promptly; runaway code still bounded; correct code unaffected.djinn/tests/test_timeout_budget.py(adapted from Don't score slow-but-correct submissions as wrong #3): timeout classification, budget sanity, verdict stable across warmup,TIMED_OUTon secure and insecure, daemon recovers after a timed-out request.Insecure-path tests skip on platforms where the daemon child cannot set RLIMITs (macOS:
Failed to set memory limits). Fixtures setinsecure_test_casesso thetest_case_leakinsecure verifier actually runs the probe instead of passing on zero leaked cases.Deliberately not carried over from #3
max(10, …)floor. Moot with a 6s per-test cap.Testing
python:3.12-slim, unprivileged containerOn
main,test_exceeded_budget_reports_timed_out_not_failed_securefails withFAILED/ "Daemon timed out" and the insecure analogue fails withCRASHED.🤖 Generated with Claude Code
https://claude.ai/code/session_014rSFHpdtkVYpGAEafamz5R