Skip to content

Add mixed-brain process chaos reporting - #11814

Draft
chaptersix wants to merge 10 commits into
temporalio:mainfrom
chaptersix:mix-brain
Draft

Add mixed-brain process chaos reporting#11814
chaptersix wants to merge 10 commits into
temporalio:mainfrom
chaptersix:mix-brain

Conversation

@chaptersix

@chaptersix chaptersix commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

What changed?

  • restart either the current or previous-release Omes dev server at a configurable one-minute default cadence while both mixed-brain scenarios run
  • wait for all original membership ports after every restart and make chaos failures cancel Omes promptly
  • run Omes with structured logging; after a failed scenario, aggregate up to three normalized ERROR findings across Omes and worker records, with occurrence counts and first-seen times
  • report the final Omes fatal and recurring errors directly in the Go test failure, then place a highlighted recurring-error table in mixedbrain-summary.md / the GitHub job summary
  • scan server logs for panics and soft assertions after every setup or scenario failure; keep complete mixed-brain logs as an artifact without printing log tails inline
  • run the suite with one direct go test invocation and an 18-minute timeout, without tee, test-runner, gotestsum, JUnit, or generic test-result JSON
  • temporarily enable standalone activities only on the current server so CI exercises a known rollback-unsafe failure and validates the reporting path

This replaces and supersedes #11001. Credit to @stephanos for the original process-chaos design; this ports that intent to the current Omes devserver.Server lifecycle introduced by temporalio/omes#458.

The draft currently pins that exact Omes commit through the chaptersix/omes fork. After temporalio/omes#458 lands, the temporary replace will be removed while retaining the same exact revision.

Why?

Mixed-version failures should name a repeated actionable error in the primary test result rather than only an eventual timeout and a large log dump. The suite is a small integration test, so its targeted GitHub summary and complete log artifact are more useful than a generic JUnit pipeline. Process restarts also exercise membership and rollback seams that a continuously running pair of servers misses.

Enable standalone activities only on the current server so the mixed-brain CI run exercises the new failure reporting against a previous release where the capability remains disabled.
},
// TODO: remove this once the release server defaults to standalone activities on. Currently the downgrade version of 1.31 has SAA defaulted to off.
DynamicConfigValues: map[string]any{
"activity.enableStandalone": true,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will revert once appoarch is approved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant