Skip to content

test(go): 邮件用例等作业的失败记进库再拨时钟,邮件按主题挑(Closes #200) - #201

Merged
Owlbay merged 1 commit into
mainfrom
fix/flaky-mail-test-200
Nov 2, 2026
Merged

Owlbay merged 1 commit into
mainfrom
fix/flaky-mail-test-200

Conversation

@Owlbay

@Owlbay Owlbay commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

变更摘要 / Summary

只改测试 go/internal/cloud/server/mail_test.go,不动生产代码。

根因:TestUnsentEmailChangeMailIsCancelledByPasswordReset 在 FailNext(0) 并 clock.Advance(31s) 之前,没等「口令已更改」通知作业的失败写进库。作业失败时 run_at = 写入时刻的时钟 + 30 秒退避;发信返回与写库之间在 -race 的慢机器上被拉长时,写入落在拨钟之后,run_at 又被推后 30 秒,测试里没人再拨时钟,通知永远等不到(mail_test.go:752,现有 1 封)。生产里时钟是真实时间,不受影响。已在 mail.send 失败路径临时加 400ms 延迟复现同一行失败,修复后同样延迟下通过(延迟补丁未提交)。

改动:

  • 找回口令之前先等确认邮件的作业失败一次(它若在找回之后才首次执行,会发现改邮箱已作废、丢掉发件箱的行并完成,从未失败过,用例就没测到「未发出的确认邮件被作废」);找回之后再等通知作业也失败一次并记下(last_error 落库、无未尝试的 open 作业),然后才放开发信、拨时钟;等不到时失败信息里列出全部 mail.send 作业的状态;
  • 末尾等全部 mail.send 作业处理完再断言新地址没有信;noMail、noMail2 与新设备提醒用例里「睡 300ms 再断言没有」同样改为等作业处理完(mailJobsSettled);
  • 等邮件的上限与 eventuallyDB 一致放到 60 秒(条件成立即返回);找回口令防枚举用例里自带的 10 秒轮询改用 eventuallyDB;
  • 同文件里剩下的 waitMails(...)[0] 全部改为 waitSubject,按主题挑信,不按下标取。

Test-only change in go/internal/cloud/server/mail_test.go; no production code is touched.

Root cause: before calling FailNext(0) and clock.Advance(31s), TestUnsentEmailChangeMailIsCancelledByPasswordReset did not wait for the failure of the "password changed" notification job to be written to the database. A failing job gets run_at = clock at write time + 30 s backoff; when the gap between the send returning and the write is stretched on a slow -race machine, the write lands after the clock was advanced, run_at moves another 30 s out, nobody advances the clock again, and the notification never arrives (mail_test.go:752, 1 mail present). Production uses the real clock and is not affected. Reproduced the same failure line by temporarily adding a 400 ms delay on the mail.send failure path; with the fix the test passes under the same delay (the delay patch is not committed).

Changes:

  • before the password reset, wait until the confirmation mail job has failed once (if it first runs after the reset it finds the email change void, deletes the outbox row and completes without ever failing, so the test would not exercise an unsent confirmation being cancelled); after the reset wait until the notification job has also failed once and the failure is recorded (non-empty last_error, no open job without an attempt), and only then enable the sender and advance the clock; on timeout the failure message lists every mail.send job's state;
  • at the end wait until all mail.send jobs are processed before asserting that the new address got nothing; noMail, noMail2 and the new-device reminder test no longer sleep 300 ms before asserting absence (mailJobsSettled);
  • the mail wait limit is 60 s like eventuallyDB (returns as soon as the condition holds); the local 10 s poll in the reset-does-not-enumerate test now uses eventuallyDB;
  • all remaining waitMails(...)[0] picks in the file now use waitSubject, so mails are picked by subject rather than by index.

关联 issue / Related issues

Closes #200

改动范围 / Scope

  • Go 实现(中继与控制面)/ Go implementation (relay and control plane)(go/,仅测试 / tests only)

契约与协议变化 / Contract and protocol changes

  • 契约节号 / Sections: 无 / None
  • 协议版本 / Protocol version: 无 / None
  • 生成物 / Generated files: 无 / None

数据库迁移 / Database migrations

  • 编号 / Number: 无 / None
  • migrations.lock: 无 / None

测试 / Testing

  • 单元测试 / Unit tests:pnpm go:check 通过(含 go test -race ./...,一次性 PostgreSQL 17 与 Redis 8 容器)/ passes
  • 集成测试 / Integration tests: 无 / None
  • 跨仓端到端 / Cross-repo e2e: 无 / None
  • 镜像 / Images: 无 / None
  • pnpm check 通过 / passes(pnpm repo:test 无关,未跑 / unrelated, not run)
  • go test -race -count=20 -run 'Mail|Email' ./internal/cloud/server:12 个 CPU 忙循环负载下全过 / passes under 12 busy-loop CPU load(221s)

检查清单 / Checklist

  • 不涉及生产代码、契约、迁移、依赖 / No production code, contract, migration or dependency change
  • 纯测试改动,不是用户可见的变化,未记 CHANGELOG / Test-only, not user-visible, so no CHANGELOG entry

需要维护者提供的外部条件 / External prerequisites

无 / None

TestUnsentEmailChangeMailIsCancelledByPasswordReset 在放开发信并拨时钟之前,没等「口令已更改」通知作业的失败写进库。
失败的 run_at 取写入时刻的时钟加退避:写入落在拨钟之后(-race 的慢机器上发信返回到写库之间被拉长),
run_at 就又后推 30 秒,没人再拨,通知永远等不到。

现在找回口令之前先等确认邮件的作业失败过一次(它若在找回之后才第一次执行,会发现改邮箱已作废、丢掉发件箱的行并完成,
从头到尾没失败过),找回之后再等通知作业也失败一次并记下(last_error 落库),然后才 FailNext(0) 并拨时钟;
等不到时把全部 mail.send 作业的状态列进失败信息。末尾等全部 mail.send 作业处理完再断言新地址没有信。
noMail / noMail2 与新设备提醒用例的「睡 300ms 再断言没有」同样改为等作业处理完。等邮件的上限与 eventuallyDB 一致放到 60 秒
(条件成立即返回)。剩下的 waitMails(...)[0] 全部改为 waitSubject,按主题挑信。
@Owlbay
Owlbay force-pushed the fix/flaky-mail-test-200 branch from 0ac8583 to 12edc74 Compare November 2, 2026 23:21
@Owlbay
Owlbay merged commit 4c925d4 into main Nov 2, 2026
6 checks passed
@Owlbay
Owlbay deleted the fix/flaky-mail-test-200 branch November 2, 2026 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

邮件时序测试 TestUnsentEmailChangeMailIsCancelledByPasswordReset 偶发失败 / Flaky mail timing test

1 participant