Skip to content

feat(sdk): make scheduling failures actionable - #164

Merged
rcyw merged 2 commits into
inclusionAI:developfrom
rcyw:dev/sdk-scheduling-errors-163
Nov 12, 2026
Merged

rcyw merged 2 commits into
inclusionAI:developfrom
rcyw:dev/sdk-scheduling-errors-163

Conversation

@rcyw

@rcyw rcyw commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

A sandbox create that cannot find a worker currently ends with a generic BackendOperationError and a long scheduling-timeout diagnostic. The SDK now raises SchedulingError, or its InsufficientResourcesError subclass when CPU, memory or disk shortages contribute and no worker is eligible.

For the mixed memory/runtime rejection in #163, the message identifies the requested 2 CPU / 8 GiB memory / 20 GiB disk, explains insufficient memory (14) and unsupported runtime (1), and recommends reducing the request, releasing idle sandboxes, or adding suitable workers.

The package version is advanced to 0.2.0rc7; the declared adx-environment==0.1.0rc4 dependency is unchanged.

Closes #163.

Behavior

  • Export both exception types through akernel_sdk; existing except BackendOperationError handlers remain valid.
  • Expose reasons, resource_shortages, and suggestions for application handling. Explain worker availability, runtime/device, placement and volume blockers alongside resource shortages.
  • Classify only structured, service-confirmed queue cancellations (DEADLINE_EXCEEDED / never / not_started). Preserve native diagnostics/cause, status, IDs, first failure and cleanup context.
  • Preserve every retry/outcome directive and the pinned native client's attempt/backoff policy. Unknown outcomes, initialization failures, transport timeouts and backend limits retain their existing behavior.
  • Report queue-budget exhaustion when workers remain eligible, and fall back conservatively for incomplete or malformed diagnostics. Keep future rejection keys inspectable.
  • Avoid presenting global per-dimension maxima as capacity available to this request; those maxima can include incompatible workers. Mixed blockers remain visible, and a shortage classification does not imply that reducing one resource alone will make placement succeed.

The change includes the retained diagnostic schema and units, the maintained runtime example, public-export coverage and installed wheel/sdist smoke checks. An explicit regression documents that existing workers with zero eligibility and no rejection reasons remain unknown, rather than implying missing workers or insufficient resources.

Validation

  • uv run --isolated --no-project --with './sdk/python[dev]' make sdk-check: 540 SDK unit tests + 35 benchmark-driver tests, Ruff and mypy passed.
  • 16 new scheduling regression tests, including actual adx-environment==0.1.0rc4 constructor/encoder/JSON/SSE parsing with HTTP I/O mocked: confirmed cancellation makes one attempt with no backoff; uncertain outcomes retain three attempts, two backoffs, and the same request body/identity.
  • Built rc7 wheel and sdist, passed twine check, installed each independently outside the checkout with declared dependencies, and ran the shared distribution smoke script, public exception checks, dependency validation and CLI help.
  • Distribution-action YAML/Python parsing, runtime-example lint and git diff --check passed.

Validation is local. Precisely distinguishing temporary occupancy from an oversized request on compatible workers still requires richer service-side capacity diagnostics.

Expose SchedulingError and InsufficientResourcesError with readable resource
requests, rejection reasons and recovery suggestions. Explain runtime,
device, placement, worker and volume blockers alongside scalar shortages,
without treating global capacity maxima as usable worker capacity.

Classify only service-confirmed cancelled scheduling requests. Preserve
native causes, diagnostics, identities and retry/outcome guidance, including
the existing same-operation policy for uncertain results. Eligible workers
and incomplete diagnoses must not imply a resource shortage.

Cover the pinned client's JSON/SSE parsing, attempt counts and identities,
public exports, diagnostic edge cases and installed distribution imports.
Document the public exceptions and update the runtime example.

Refs: inclusionAI#163
Signed-off-by: wayne <rcywjo@gmail.com>

@hky1999 hky1999 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on 8d60cb6; approving.

The exception hierarchy is designed correctly: SchedulingError extends BackendOperationError (so existing except-clauses keep working), InsufficientResourcesError extends SchedulingError, and both are exported from akernel_sdk. The three properties (reasons, resource_shortages, suggestions) are derived lazily from the retained scheduling_failure dict via a pure classifier, with the raw diagnostics preserved and the native error as cause - nothing about the service's retry/outcome guidance is overwritten.

The classification gate is appropriately narrow: only structured, service-confirmed queue cancellations (code DEADLINE_EXCEEDED + retry never + outcome not_started, status None or 504) are classified; unknown outcomes, initialization failures, transport timeouts and backend limits keep their existing shapes. The PR body's retry-preservation claim matters most and the tests cover it: a confirmed cancellation makes exactly one attempt with no backoff, while uncertain outcomes retain three attempts and two backoffs with an identical request body/identity - verified against the actual pinned native client (adx-environment 0.1.0rc4) constructor/encoder/JSON/SSE parsing with only HTTP mocked.

The classifier itself (explain_scheduling_failure) is defensive in the right direction - malformed counters (bools, negatives, out-of-range) return the "unknown" explanation rather than coercing a diagnosis; eligible > nodeCount is treated as unusable data; rejection counts exceeding nodeCount (except no_workers) likewise. The three-way outcome is honest: eligible > 0 reports queue_timeout (workers exist, deadline hit - suggests larger schedule_timeout), no_workers/zero nodes reports registration, and all-rejected enumerates rejection keys with labels and per-reason suggestions. The known-blocker table (node_available, runtime_fit, device_fit, affinity/placement family, waiting_for_volume, volume_topology) gives good messages, and unknown future keys fall through to "other placement constraint" with the operator hint - inspectable rather than swallowed.

I particularly appreciate two honesty choices called out in code comments and the PR body: global per-dimension maxima are NOT presented as capacity available to this request (mixed blockers stay visible, so a caller cannot conclude that shrinking one resource suffices), and rejection counts are labeled as overlapping rather than a node census.

For #163's case the message reads exactly as claimed: requested 2 CPU / 8 GiB / 20 GiB, insufficient memory (14), unsupported runtime (1), with reduce/release/add-suggestions.

Verification on my side (worktree of 8d60cb6): full SDK unit suite runs 539 tests as claimed; the 15 new scheduling regressions all pass; deploy-script-check passes. The only two failures in my environment (test_lightweight_imports_do_not_load_yuanrong, test_generated_environment_imports_this_checkout_sdk) reproduce identically on the develop baseline - they are subprocess-import tests sensitive to my local interpreter mix (uv python 3.13 + borrowed site-packages), not regressions from this PR; hosted CI 43196627609 on the exact head is green.

Two non-blocking observations:

  1. The classifier reads a single scheduling snapshot; the PR body honestly notes that distinguishing temporary occupancy from an oversized request on compatible workers needs richer service-side diagnostics. When the service grows that, the natural extension point is here - worth keeping the dict schema documented alongside the service side.
  2. _count accepts 0 <= value < 2**64 but eligible == 0 with an empty rejections map returns unknown (not no_workers) - correct, since zero eligibility with no recorded reasons is genuinely unexplained; just noting the case exists deliberately.

Document the retained scheduling diagnostic schema and its resource units,
overlapping rejection counts and service-side extension boundary. Cover the
deliberate unknown diagnosis when workers exist but none is eligible and
no rejection reasons are reported.

Advance the SDK package version to 0.2.0rc7 without changing the classifier,
ADX dependency or retry/outcome semantics. Validate the full SDK quality
gate and independently installed wheel and source distributions.

Signed-off-by: wayne <rcywjo@gmail.com>
@rcyw
rcyw merged commit 7fe311e into inclusionAI:develop Nov 12, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants