Skip to content

fix(channels): stream the YuanBao bot creator over binary pipes so /poll keeps its events - #912

Open
Lesereingrape wants to merge 15 commits into
TencentCloud:developfrom
Lesereingrape:fix/yuanbao-creator-pipes
Open

Lesereingrape wants to merge 15 commits into
TencentCloud:developfrom
Lesereingrape:fix/yuanbao-creator-pipes

Conversation

@Lesereingrape

Copy link
Copy Markdown
Contributor

Summary

The YuanBao bot-creator endpoints read newline-delimited JSON from the child's stdout through parse_subprocess_json_lines, which is bytes-only — parse_json_lines decodes what it reads (src/octop/infra/utils/subprocess_io.py:28). The subprocess was the only one in src/ started in text mode (bufsize=1, universal_newlines=True, :972-973), so on Windows TextIOWrapper.read() handed a str to that decoder and /poll died with AttributeError: 'str' object has no attribute 'decode' at subprocess_io.py:28 on the first response that had output. The bytes were already out of the pipe at that point, and the raise happens before state["lines"].extend(new_lines) — so the events were dropped, not delayed: the scan_code the wizard is waiting for never reached the dashboard.

Its stderr=subprocess.PIPE (:971) was also never read by any code path: a creator that exits with a traceback reported status="failed" with an empty events list and nothing to show, and an undrained pipe is the deadlock CPython's subprocess docs warn about ("…the child process needs to write enough data… use stderr=STDOUT").

Both halves are one root cause — the launch configuration was written against a text/stream reader, while the pipeline is a bytes-only JSON-line reader — so the fix is to start the child the way feishu_bot_creator_start (:823-824) and infra/gateway/bot_creators/feishu_runner.py:42-48 already do: binary stdout with stderr merged into it, and the final drain decoded the same way (:1009). The JSON-line protocol, the argv, and the input sanitizers are untouched, and no other launch site in src/ is changed by this branch.

Fixes #911

Related open work

Open PR #756 (fix(channels): make the Feishu scan-to-create subprocess work on Windows, base develop) edits the same Popen call: at :972-974 it replaces env={**os.environ, "PYTHONUNBUFFERED": "1"} with a new python_subprocess_env() helper, and pins the child's stdout encoding to UTF-8 in bot_creators/yuanbao_bot_creator.py. That is a different root cause — the child failing on a cp936 console encoding — and it leaves bufsize=1, universal_newlines=True and stderr=subprocess.PIPE exactly as they are, so both halves reported above survive it. It does mean the two branches overlap on this one call site.

Re-checked before filing (2026-09-21 08:53:52Z): #756 is still open and unmerged (head=feature/feishu-channel, base=c7a095a7, last updated 2026-09-20T16:51:08Z), and this branch is rebased onto develop @ e512f05b, which does not contain it. If #756 merges first, the one-line env= change is all that has to be re-applied here — I would rather rebase onto it than have this branch silently drop that fix, so say the word and I will rebase on request. I have deliberately not rebased onto #756 while it is unmerged.

Also worth flagging: this is one of two PRs I am filing together (#910 is the other), and both add a bullet at the very top of ### 修复 in CHANGELOG.md, so whichever of the two lands second will show a one-line conflict there. Each branch keeps its own entry so that neither PR depends on the other; resolving it is a one-line fix and I am happy to rebase whichever one you merge second.

Target branch

  • Base is develop (feature / fix — default)
  • Base is main (release/* or hotfix/* only)

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Refactor / chore
  • Release / hotfix

Test plan

make is unavailable on this machine, so the .githooks/pre-commit gate (make all + dashboard build, per AGENTS.md §6/§10) never ran here — the commits were made without it. Its sub-targets were run individually and are quoted verbatim below.

The existing bot-creator tests patch subprocess.Popen wholesale, which cannot see a pipe-configuration bug. The three new cases therefore spawn a real stub child and drive the two endpoints over TestClient, following the _make_app / mock_server_and_user idiom already in that file.

RED — measured today by running this branch's tests/unit/gateway/test_channels_qr.py unchanged inside a
pristine develop @ e512f05b worktree (only the test file copied, no source change), Windows / Python 3.12:

pytest tests/unit/gateway/test_channels_qr.py -q
→ 3 failed, 22 passed
  FAILED ::test_yuanbao_creator_start_keeps_the_reader_contract   # AssertionError: parse_subprocess_json_lines
                                                                    decodes what it reads, so a text-mode stdout
                                                                    pipe raises AttributeError on the first poll
                                                                    that has output
  FAILED ::test_yuanbao_creator_poll_reports_events               # AttributeError: 'str' object has no attribute
                                                                    'decode' at src/octop/infra/utils/subprocess_io.py:28
  FAILED ::test_yuanbao_creator_poll_surfaces_stderr              # AssertionError: [] — events empty, traceback lost

Behaviour, the shared reader fed the two launch configs side by side (no server, PYTHONPATH=src python probe.py):

{'bufsize': 1, 'universal_newlines': True} -> AttributeError: 'str' object has no attribute 'decode'
{} -> [{'action': 'scan_code', 'scan_code': 'abc'}]

Same stub child, but driven through the two real endpoints (TestClient(..., raise_server_exceptions=False), so the
status the app returns is visible), pristine develop vs this branch:

develop @ b46bb8a4: (200 running, 0 events) (200 running, 0 events) (500 'Internal Server Error')
this branch       : (200 running, 0 events) (200 running, 0 events) (200 finished, 1 event)

Final state on this branch (Windows / Python 3.12; make test equivalents):

pytest tests/unit/gateway/test_channels_qr.py -q → 25 passed
pytest -n 4 -m "not live"                       → 15 failed, 3501 passed, 118 skipped (11:00)
pytest tests/unit/agents/test_memory_slim.py -q   on pristine e512f05b (same venv, same command)
→ 15 failed, 21 passed
  ↳ all 15 failures in the full run are in that one file, which #865 added to develop today;
    this branch changes no agent/memory code, so the delta vs its own base is 0 regressions
ruff check src tests            → All checks passed!
ruff format --check src tests   → 1049 files already formatted
mypy src/octop                  → Success: no issues found in 514 source files
  • Added/updated tests — 3 cases in tests/unit/gateway/test_channels_qr.py

Only Windows was measured here. Reading the two platform branches of read_available_bytes: _keeps_the_reader_contract and _surfaces_stderr should fail on Linux too (a text-mode pipe is text mode everywhere, and read_available_posix never touches stderr), while test_yuanbao_creator_poll_reports_events should pass there before the change — os.read returns bytes regardless of the wrapper — which is why the invariant test exists next to it, and why the Windows job is the one that actually catches the crash.

Checklist

  • Updated CHANGELOG.md — one bullet at the top of ### 修复 under ## [Unreleased], citing (#911), landed as commit c57dca17
  • README / docs updated (if needed) — not needed: no route, body or response shape changed

Not tested here: no live server and no real YuanBao account, so the creator child in these tests is a stub script rather than yuanbao_bot_creator.py; the >64 KB pipe-fill block is cited from the subprocess docs rather than measured (the swallowed-stderr half is measured); CI's Linux leg was not run locally.

The YuanBao bot-creator endpoints read newline-delimited JSON from the
child's stdout with `parse_subprocess_json_lines`, which decodes the bytes
it reads. The subprocess was started in text mode, so on Windows the pipe
handed a `str` to that decoder and the first poll that had output raised
`AttributeError: 'str' object has no attribute 'decode'` - taking the scan
code the whole flow waits for with it. Its stderr pipe was never drained
either, so a creator that died with a traceback reported `failed` with no
events at all.

Start the child the way the feishu creator does: binary stdout with stderr
merged into it, and decode the final drain the same way.
@jubaoliang jubaoliang closed this Sep 22, 2026
@jubaoliang jubaoliang reopened this Sep 22, 2026
…r-pipes

CHANGELOG.md only: keep the entries added on develop since e512f05, with
this fix's entry above them.
@Lesereingrape

Copy link
Copy Markdown
Contributor Author

CHANGELOG 冲突已解决(本 PR 与 #910 都会插入 ### 修复 的开头,此前在描述里预告过)。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants