[remote-controlled-man]

Summary

Stop retrying permanent execution-status callback failures in the Python SDK. Currently a 400/401/403/404/409 response or a request-construction/programming error consumes the normal five callback attempts and the 1/2/4/8-second backoff; this change returns after the first permanent failure and logs it once.

Transport errors, HTTP 408/429 and 5xx responses still retry. The existing default of five outer callback attempts, exponential backoff, timeout arguments (30 seconds normally, 5 seconds during the single shutdown attempt), successful-response handling and cancellation propagation are preserved. HTTPX interprets those scalar timeout arguments per phase; this PR does not add a total request deadline.

This addresses the Python portion of #1088. The TypeScript portion remains outside this PR, and #1099 has already fixed the execution-row race that previously made 404 retryable temporarily.

Type of change

• Bug fix

Test plan

Executed locally on Windows, Python 3.11.7, against base 2f5583e5 and reviewed head bc8149d8:

• Regression RED: restore the unpatched base implementation and run the new callback behavior cases. Eleven cases fail on actual retry-count assertions; collection succeeds. Restore the reviewed files exactly afterward.

• python -m pytest --no-cov tests/test_async_execution.py: 47 passed. Cases cover status classification, HTTPX/requests and built-in transport exceptions, permanent errors, backoff/exhaustion, shutdown and cancellation.

• python -m ruff check . with CI-pinned Ruff 0.15.22: passed.

• python -m mypy --config-file mypy.ini agentfield/: passed on 83 source files.

• Broader SDK tests with --no-cov, --ignore=tests/test_agent_mesh.py and the eight baseline exclusions below: 2537 passed, 6 skipped, 46 deselected, zero failures. Git Bash is used for CLI subprocess tests.

Eight failures were reproduced on the exact unpatched base and explicitly deselected: seven Windows test_log_writer.py pipe/CRLF cases and one test_tool_calling.py case. The 46 deselections include 38 already configured by the repository. An earlier coverage-enabled run had 2536 passed and one failure in the unchanged connection-manager health-check reconnection timing assertion; that node was retained and passed in the final run. Three isolated base repetitions also passed, so this is not claimed as a reproduced base failure. A later coverage-enabled run stopped with Windows resource exhaustion after 703 passes; the final run disables coverage collection, without adding exclusions. Both failures remain recorded.

tests/test_agent_mesh.py was not run in this continuation because of the previously observed Windows hang. Linux, other Python versions, alternate websockets versions, live control-plane tests, scripts/test-all.sh and scripts/coverage-summary.sh were not run locally; Docker is unavailable. No full unfiltered suite or hosted CI pass is claimed.

Test coverage

• I ran tests for the Python surface changed here.
• New callback branches have direct assertions in this PR.
• The repository-wide coverage gate is green in CI before requesting review.

The local configured coverage selection does not measure agentfield.agent; the callback assertions above execute it directly. Coverage baselines are unchanged. Hosted coverage results still need to be checked.

Checklist

• I read docs/CONTRIBUTING.md, docs/DEVELOPMENT.md and the applicable repository instructions.
• Commits are signed and follow conventional-commits style.
• I linked the related issue without closing its remaining TypeScript work.

The two source commits retain their original Cursor Agent authorship and are unsigned after rebasing. This contribution used AI assistance; the tests listed above were executed during preparation, independently of the original source PR's claims.

Related issues / PRs

Refs #1088. Follows the Go callback classification in #1081 and the legacy dispatch fix in #1099.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[github-actions]

Performance

SDK	Memory	Δ	Latency	Δ	Tests	Status
Python	9.1 KB	-	0.31 µs	-11%	✓	✓

✓ No regressions detected

[remote-controlled-man]

The lint-and-test (3.12) failure is unrelated to this change: FastAPI 0.145.0 (released Oct 18) removed **extra from FastAPI.__init__, and CI's unpinned pip install .[dev] picked up 0.146.0, so the 6 tests in test_agent_graceful_shutdown.py / test_did_manager_error_paths.py that pass enable_mcp=False to Agent(...) now raise TypeError. Main will hit the same failure on its next run. I'll open a separate small PR that drops the dead enable_mcp=False argument from those tests so CI goes green again.

[AbirAbbas]

Ran this against a live control plane built from current main: async execution completes normally, an unknown execution id gets exactly one 404 attempt, and a rotated key gets one 401 attempt. The new tests also fail against main's _post_execution_status the way they should (permanent cases go five rounds there).

One thing to fix before merging: the requests fallback in _is_retryable_status_callback_error only retries ConnectionError and Timeout, but requests reads the body eagerly and raises requests.exceptions.ChunkedEncodingError when a response is cut off mid-body. That's a transport failure, not a programming error. A real server that sends a truncated 503 and then a clean 200 gets one request from this branch, which then logs "not retrying" and drops the status, while main gets there on the second try. Adding ChunkedEncodingError (and arguably ContentDecodingError) to the retryable tuple, with a test that goes through the actual fallback against a server that truncates once, would cover it.

A heads-up rather than a blocker: the legacy reasoner and skill handlers in reasoners.go only log when createLegacyExecutionRecord fails and still dispatch, and a later workflow event can insert the row afterwards. In that narrow window, the first 404 here is actually transient. I think the right fix is on the control plane side (fail closed like execute_prepare.go does), so I'll track that separately.

Also, merging main will clear the red Python CI: that was the FastAPI 0.148 enable_mcp breakage, fixed in #1120.

[github-actions]

📊 Coverage gate

Thresholds from .coverage-gate.toml: per-surface ≥ 84%, aggregate ≥ 85%, max per-surface regression ≤ 1.0 pp, max aggregate regression ≤ 0.50 pp.

Surface	Current	Baseline	Δ
control-plane	88.20%	87.40%	↑ +0.80 pp	🟡
sdk-go	93.40%	92.00%	↑ +1.40 pp	🟢
sdk-python	94.72%	93.73%	↑ +0.99 pp	🟢
sdk-typescript	91.84%	90.42%	↑ +1.42 pp	🟢
web-ui	84.89%	84.79%	↑ +0.10 pp	🟡
aggregate	86.11%	85.75%	↑ +0.36 pp	🟡

✅ Gate passed

No surface regressed past the allowed threshold and the aggregate stayed above the floor.

[github-actions]

📐 Patch coverage gate

Threshold: 80% on lines this PR touches vs origin/main (from .coverage-gate.toml:thresholds.min_patch).

Surface	Touched lines	Patch coverage	Status
control-plane	11	100.00%	✅
sdk-go	0	—	➖ no changes
sdk-python	0	—	➖ no changes
sdk-typescript	0	—	➖ no changes
web-ui	0	—	➖ no changes

✅ Patch gate passed

Every surface whose lines were touched by this PR has patch coverage at or above the threshold.

[remote-controlled-man]

Thanks for live-testing this! I merged main (which picks up #1120, so Python CI is green again) and added ChunkedEncodingError and ContentDecodingError to the retryable tuple in the requests fallback, with a test that exercises the actual fallback against a server that truncates the body once and then returns 200. Agreed that the legacy 404 window is better fixed on the control-plane side. Could you take another look when you have a moment?

[AbirAbbas]

Thanks for the quick turnaround. The fallback fix checks out: a truncated 503 and an invalid-gzip 503 both deliver on the second request now, and your new test fails against the previous head. Permanent 4xx still stop after one attempt, and 408, 429 and 5xx still retry.

I pushed the control-plane side on top (8a4b07f) so this can go in as one piece. The legacy /reasoners and /skills handlers now return a 500 instead of dispatching when the execution row can't be written, for both normal and serverless nodes. That closes the window where a 404 here could have been transient. Execution IDs on that path are always generated by the server, so a client retry can't trip the new check. A real async round trip and a legacy reasoner call both still complete against a control plane built from this branch.

[AbirAbbas]

Approving at 8a4b07f. CI is green, the fallback recovers truncated and badly-encoded responses, permanent 4xx stop after one attempt, and the legacy handlers can no longer dispatch work whose execution row wasn't written.