test(agent): stop framing the e2e attachment marker as a confidential access code by ThomasK33 · Pull Request #218 · coder/ai-sdk

ThomasK33 · 2026-09-18T10:22:12Z

Summary

The attachment-reading test described ZEBRA-7731 as an access code and told the model to keep it confidential. That made success depend on the model's willingness to disclose a supposed credential, not just its ability to read the attachment. PR #217 records two 6/7 runs and the exact refusal, with logs and media.

Only “attaches a file and the model reads its contents” changes. The attachment is now a shipment note with a harmless reference id. The question asks for that reference id without revealing ZEBRA-7731. The secret variable becomes marker; the assertion remains expect(result.text).toContain(marker).

No other tests, tool fixtures, runtime code, agent instructions, model settings, or timeouts change.

Validation

Head: e11873de81c1a9e22ad69f22ef30f37290694c94. One commit from 50d16ea91b6f044b6917b18943f4d91de4117176.

pnpm check: passed (format, lint, typecheck).
pnpm --filter @coder/ai-sdk-agent test: 369 passed.
pnpm -r build and pnpm -r test: passed.
coder whoami: passed before each live run, using ambient authentication on https://dogfood.cdr.dev. No tokens were printed, created, revoked, or unset.
Both live suites used Node 26.9.0, pnpm 11.27.0, and the unchanged default models (haiku / sonnet).

Live suite	Start (UTC, 2026-09-18)	Result	Duration	Attachment test
Run 1	10:19:15	7 passed (7)	28.54s	passed, 2393ms
Run 2	10:20:02	7 passed (7)	27.90s	passed, 2527ms

Both native receipts: Vitest exit 0; tee exit 0. No failed post-change run or retry preceded these results.

Reproduce:

coder whoami
cd packages/agent
npx vitest run test/e2e
npx vitest run test/e2e

Verification tier: source/diff checks plus two live runs on dogfood. Raw logs, native exit receipts, and terminal casts are preserved outside the checkout. The clips below are accelerated terminal-output replays (5.80s and 5.72s), not real-time recordings or additional test invocations. They provide completion evidence; the logs retain the full emitted test output.

Run 1: complete test log

RUN  v5.0.1 /home/coder/.mux/src/ai-sdk/e2e-attachment-fixture/packages/agent

stdout | test/e2e/agent.e2e.test.ts > CoderAgent e2e (live Coder) > emits a coherent transport-event trace for a client-tool turn (#45)
[+     0ms] segment:start 
[+     0ms] http:request GET /api/v2/organizations/703f72a1-76f6-4f89-9de6-8a3989693fe5/chats/models
[+   393ms] http:response GET /api/v2/organizations/703f72a1-76f6-4f89-9de6-8a3989693fe5/chats/models → 200 (393ms)
[+   505ms] http:request POST /api/v2/chats
[+   671ms] http:response POST /api/v2/chats → 201 (166ms)
[+   671ms] ws:dial attempt=1
[+  1025ms] ws:open 
[+  1026ms] ws:event message
[+  1051ms] ws:event status:running
[+  1051ms] ws:event preview_reset
[+  2635ms] ws:event message_part
[+  2644ms] ws:event message_part
[+  2644ms] ws:event message_part
[+  2644ms] ws:event message_part
[+  2645ms] ws:event message_part
[+  2645ms] ws:event message_part
[+  2645ms] ws:event message_part
[+  2825ms] ws:event message_part
[+  2825ms] ws:event message_part
[+  3277ms] ws:event message_part
[+  3277ms] ws:event message_part
[+  3277ms] ws:event message_part
[+  3277ms] ws:event message_part
[+  3319ms] ws:event message
[+  3319ms] ws:event preview_reset
[+  3367ms] ws:event status:requires_action
[+  3367ms] ws:event action_required
[+  3367ms] segment:settle status=requires_action finish=tool-calls (3367ms)
[+  3368ms] segment:start 
[+  3368ms] http:request POST /api/v2/chats/42acdcf4-43fc-42d4-a7ee-5bdb4dfe1895/tool-results
[+  3509ms] http:response POST /api/v2/chats/42acdcf4-43fc-42d4-a7ee-5bdb4dfe1895/tool-results → 204 (141ms)
[+  3515ms] ws:event message
[+  3515ms] ws:event status:running
[+  3515ms] ws:event preview_reset
[+  3556ms] ws:event preview_reset
[+  5074ms] ws:event message_part
[+  5074ms] ws:event message_part
[+  5109ms] ws:event message
[+  5109ms] ws:event preview_reset
[+  5156ms] ws:event status:waiting
[+  5157ms] ws:close 
[+  5157ms] segment:settle status=waiting finish=stop (1789ms)

 ✓ test/e2e/agent.e2e.test.ts (7 tests) 28130ms
   ✓ CoderAgent e2e (live Coder) (7)
     ✓ generates plain text 2125ms
     ✓ streams text deltas 2699ms
     ✓ round-trips a custom (client-executed) tool 5609ms
     ✓ reuses ONE WebSocket across a multi-tool-step turn (#44) 8933ms
     ✓ emits a coherent transport-event trace for a client-tool turn (#45) 5450ms
     ✓ uploads and downloads a chat file (round-trip) 500ms
     ✓ attaches a file and the model reads its contents 2393ms

 Test Files  1 passed (1)
      Tests  7 passed (7)
   Start at  10:19:15
   Duration  28.54s (tests 99%, transform 1%)

Run 1: screenshot and accelerated terminal replay

live-1.webm

Run 2: complete test log

RUN  v5.0.1 /home/coder/.mux/src/ai-sdk/e2e-attachment-fixture/packages/agent

stdout | test/e2e/agent.e2e.test.ts > CoderAgent e2e (live Coder) > emits a coherent transport-event trace for a client-tool turn (#45)
[+     0ms] segment:start 
[+     0ms] http:request GET /api/v2/organizations/703f72a1-76f6-4f89-9de6-8a3989693fe5/chats/models
[+   405ms] http:response GET /api/v2/organizations/703f72a1-76f6-4f89-9de6-8a3989693fe5/chats/models → 200 (405ms)
[+   519ms] http:request POST /api/v2/chats
[+   811ms] http:response POST /api/v2/chats → 201 (292ms)
[+   811ms] ws:dial attempt=1
[+  1184ms] ws:open 
[+  1189ms] ws:event message
[+  1189ms] ws:event status:running
[+  1189ms] ws:event preview_reset
[+  2230ms] ws:event message_part
[+  2400ms] ws:event message_part
[+  2400ms] ws:event message_part
[+  2400ms] ws:event message_part
[+  2401ms] ws:event message_part
[+  2444ms] ws:event message
[+  2444ms] ws:event preview_reset
[+  2493ms] ws:event status:requires_action
[+  2493ms] ws:event action_required
[+  2493ms] segment:settle status=requires_action finish=tool-calls (2493ms)
[+  2496ms] segment:start 
[+  2496ms] http:request POST /api/v2/chats/a22885a9-907b-42fd-87ee-101833e81df4/tool-results
[+  2863ms] http:response POST /api/v2/chats/a22885a9-907b-42fd-87ee-101833e81df4/tool-results → 204 (367ms)
[+  2879ms] ws:event message
[+  2879ms] ws:event status:running
[+  2879ms] ws:event preview_reset
[+  2923ms] ws:event preview_reset
[+  4551ms] ws:event message_part
[+  4551ms] ws:event message_part
[+  4592ms] ws:event message
[+  4592ms] ws:event preview_reset
[+  4638ms] ws:event status:waiting
[+  4638ms] ws:close 
[+  4639ms] segment:settle status=waiting finish=stop (2143ms)

 ✓ test/e2e/agent.e2e.test.ts (7 tests) 27367ms
   ✓ CoderAgent e2e (live Coder) (7)
     ✓ generates plain text 2292ms
     ✓ streams text deltas 2453ms
     ✓ round-trips a custom (client-executed) tool 4825ms
     ✓ reuses ONE WebSocket across a multi-tool-step turn (#44) 9467ms
     ✓ emits a coherent transport-event trace for a client-tool turn (#45) 4835ms
     ✓ uploads and downloads a chat file (round-trip) 514ms
     ✓ attaches a file and the model reads its contents 2527ms

 Test Files  1 passed (1)
      Tests  7 passed (7)
   Start at  10:20:02
   Duration  27.90s (tests 98%, transform 1%)

Run 2: screenshot and accelerated terminal replay

live-2.webm

Final readiness and outcome

Reviewed source commit: e11873de81c1a9e22ad69f22ef30f37290694c94. The committed diff is byte-identical to the validated diff.
Merged via the queue at 2026-09-18 10:35:18 UTC as 758636bbc0fae91f850e1ac0cdf398b369bf9480, verified reachable on origin/main. Attribution is retained in the squash commit.
Validation: local check/build/test gates passed; agent unit 369 passed; live suites 7/7 twice, as recorded above. PR CI Required and merge-group CI passed.
Reviews: one code round + one security round (two total), zero errors/retriggers. Code verdict and security verdict both clean on the source commit.
Both summary rows had settled by 10:25:32 UTC. The final dispatch-time audit ran 469.8 seconds after the later row completed: unchanged head, zero unresolved threads, no late findings. Queue submission used expectedHeadOid.
One independent readiness pass recommended ready after the settling-window audit. The advisor agreed. No deferred code findings remain; stopping because the requested one-test fix and all merge gates are complete.
Release-please completed successfully and did open #219 — chore(agent): release 0.11.5. Its notes say “A maintenance release with no documented user-facing changes.” That release PR is left untouched; release-policy changes are outside this task.
Raw logs, native exit receipts, casts, and media are retained under $HOME/w40-scratch/e2e-attachment-fixture.tjibYa/.

Generated with xum • Model: coder:openai/gpt-6-astra • Thinking: xhigh

… access code Use a harmless shipment reference id so the live test measures attachment reading rather than willingness to disclose credentials. Keep the marker in the attachment and preserve the assertion. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `coder:openai/gpt-6-astra` • Thinking: `xhigh`_ Change-Id: I9923dc0801ad3e3d48f7f0485be05d561e9df436

ThomasK33 · 2026-09-18T10:23:09Z

@codex review

ThomasK33 · 2026-09-18T10:23:14Z

@codex security review

chatgpt-codex-connector · 2026-09-18T10:23:29Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-18T10:24:37.989180Z	`e11873d`	Manual request
🔒 Security Review	✅ Completed 2026-09-18T10:25:32.098033Z	`e11873d`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-09-18T10:24:36Z

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: e11873de81

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-18T10:25:30Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: e11873de81

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

ThomasK33 added this pull request to the merge queue Sep 18, 2026

Merged via the queue into main with commit 758636b Sep 18, 2026
7 checks passed

ThomasK33 deleted the e2e-attachment-fixture branch September 18, 2026 10:35

ThomasK33 mentioned this pull request Sep 18, 2026

chore(effect): bump published provider/sandbox pins to 0.4.17/0.4.20 #217

Merged

Navigation Menu

test(agent): stop framing the e2e attachment marker as a confidential access code - #218

test(agent): stop framing the e2e attachment marker as a confidential access code#218
ThomasK33 merged 1 commit into
mainfrom
e2e-attachment-fixture

ThomasK33 commented Sep 18, 2026 •

edited

Loading

Uh oh!

ThomasK33 commented Sep 18, 2026

Uh oh!

ThomasK33 commented Sep 18, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 18, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 18, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ThomasK33 commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Final readiness and outcome

Uh oh!

ThomasK33 commented Sep 18, 2026

Uh oh!

ThomasK33 commented Sep 18, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Sep 18, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 18, 2026

🛡️ Codex Security Review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ThomasK33 commented Sep 18, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 18, 2026 •

edited

Loading