Repository navigation
feat(sdk): make scheduling failures actionable - #164
Conversation
Expose SchedulingError and InsufficientResourcesError with readable resource requests, rejection reasons and recovery suggestions. Explain runtime, device, placement, worker and volume blockers alongside scalar shortages, without treating global capacity maxima as usable worker capacity. Classify only service-confirmed cancelled scheduling requests. Preserve native causes, diagnostics, identities and retry/outcome guidance, including the existing same-operation policy for uncertain results. Eligible workers and incomplete diagnoses must not imply a resource shortage. Cover the pinned client's JSON/SSE parsing, attempt counts and identities, public exports, diagnostic edge cases and installed distribution imports. Document the public exceptions and update the runtime example. Refs: inclusionAI#163 Signed-off-by: wayne <rcywjo@gmail.com>
hky1999
left a comment
There was a problem hiding this comment.
Verified on 8d60cb6; approving.
The exception hierarchy is designed correctly: SchedulingError extends BackendOperationError (so existing except-clauses keep working), InsufficientResourcesError extends SchedulingError, and both are exported from akernel_sdk. The three properties (reasons, resource_shortages, suggestions) are derived lazily from the retained scheduling_failure dict via a pure classifier, with the raw diagnostics preserved and the native error as cause - nothing about the service's retry/outcome guidance is overwritten.
The classification gate is appropriately narrow: only structured, service-confirmed queue cancellations (code DEADLINE_EXCEEDED + retry never + outcome not_started, status None or 504) are classified; unknown outcomes, initialization failures, transport timeouts and backend limits keep their existing shapes. The PR body's retry-preservation claim matters most and the tests cover it: a confirmed cancellation makes exactly one attempt with no backoff, while uncertain outcomes retain three attempts and two backoffs with an identical request body/identity - verified against the actual pinned native client (adx-environment 0.1.0rc4) constructor/encoder/JSON/SSE parsing with only HTTP mocked.
The classifier itself (explain_scheduling_failure) is defensive in the right direction - malformed counters (bools, negatives, out-of-range) return the "unknown" explanation rather than coercing a diagnosis; eligible > nodeCount is treated as unusable data; rejection counts exceeding nodeCount (except no_workers) likewise. The three-way outcome is honest: eligible > 0 reports queue_timeout (workers exist, deadline hit - suggests larger schedule_timeout), no_workers/zero nodes reports registration, and all-rejected enumerates rejection keys with labels and per-reason suggestions. The known-blocker table (node_available, runtime_fit, device_fit, affinity/placement family, waiting_for_volume, volume_topology) gives good messages, and unknown future keys fall through to "other placement constraint" with the operator hint - inspectable rather than swallowed.
I particularly appreciate two honesty choices called out in code comments and the PR body: global per-dimension maxima are NOT presented as capacity available to this request (mixed blockers stay visible, so a caller cannot conclude that shrinking one resource suffices), and rejection counts are labeled as overlapping rather than a node census.
For #163's case the message reads exactly as claimed: requested 2 CPU / 8 GiB / 20 GiB, insufficient memory (14), unsupported runtime (1), with reduce/release/add-suggestions.
Verification on my side (worktree of 8d60cb6): full SDK unit suite runs 539 tests as claimed; the 15 new scheduling regressions all pass; deploy-script-check passes. The only two failures in my environment (test_lightweight_imports_do_not_load_yuanrong, test_generated_environment_imports_this_checkout_sdk) reproduce identically on the develop baseline - they are subprocess-import tests sensitive to my local interpreter mix (uv python 3.13 + borrowed site-packages), not regressions from this PR; hosted CI 43196627609 on the exact head is green.
Two non-blocking observations:
- The classifier reads a single scheduling snapshot; the PR body honestly notes that distinguishing temporary occupancy from an oversized request on compatible workers needs richer service-side diagnostics. When the service grows that, the natural extension point is here - worth keeping the dict schema documented alongside the service side.
- _count accepts 0 <= value < 2**64 but eligible == 0 with an empty rejections map returns unknown (not no_workers) - correct, since zero eligibility with no recorded reasons is genuinely unexplained; just noting the case exists deliberately.
Document the retained scheduling diagnostic schema and its resource units, overlapping rejection counts and service-side extension boundary. Cover the deliberate unknown diagnosis when workers exist but none is eligible and no rejection reasons are reported. Advance the SDK package version to 0.2.0rc7 without changing the classifier, ADX dependency or retry/outcome semantics. Validate the full SDK quality gate and independently installed wheel and source distributions. Signed-off-by: wayne <rcywjo@gmail.com>
A sandbox create that cannot find a worker currently ends with a generic
BackendOperationErrorand a long scheduling-timeout diagnostic. The SDK now raisesSchedulingError, or itsInsufficientResourcesErrorsubclass when CPU, memory or disk shortages contribute and no worker is eligible.For the mixed memory/runtime rejection in #163, the message identifies the requested 2 CPU / 8 GiB memory / 20 GiB disk, explains insufficient memory (14) and unsupported runtime (1), and recommends reducing the request, releasing idle sandboxes, or adding suitable workers.
The package version is advanced to
0.2.0rc7; the declaredadx-environment==0.1.0rc4dependency is unchanged.Closes #163.
Behavior
akernel_sdk; existingexcept BackendOperationErrorhandlers remain valid.reasons,resource_shortages, andsuggestionsfor application handling. Explain worker availability, runtime/device, placement and volume blockers alongside resource shortages.DEADLINE_EXCEEDED / never / not_started). Preserve native diagnostics/cause, status, IDs, first failure and cleanup context.The change includes the retained diagnostic schema and units, the maintained runtime example, public-export coverage and installed wheel/sdist smoke checks. An explicit regression documents that existing workers with zero eligibility and no rejection reasons remain
unknown, rather than implying missing workers or insufficient resources.Validation
uv run --isolated --no-project --with './sdk/python[dev]' make sdk-check: 540 SDK unit tests + 35 benchmark-driver tests, Ruff and mypy passed.adx-environment==0.1.0rc4constructor/encoder/JSON/SSE parsing with HTTP I/O mocked: confirmed cancellation makes one attempt with no backoff; uncertain outcomes retain three attempts, two backoffs, and the same request body/identity.twine check, installed each independently outside the checkout with declared dependencies, and ran the shared distribution smoke script, public exception checks, dependency validation and CLI help.git diff --checkpassed.Validation is local. Precisely distinguishing temporary occupancy from an oversized request on compatible workers still requires richer service-side capacity diagnostics.