Fix mislabeled "system payload" issue

[chrsmith]

⚠️ I'm a little out of my depth when it comes to the System Nexus Endpoint, and the current requirements around "system payloads". So please review this carefully to keep me honest.

I'm not sure if this is the right way we want to address this problem, of if instead we should just relax the check in the SDK. (@stuart-wells opened temporalio/api-go#309, which we may also want to take. Independently of this.)

Context

The Temporal server was labeling everything that came through the History Service's StartNexusOperation as a system payload. (#10948) However, when we landed support for visiting nested payloads in the Golang SDK (temporalio/api-go#297) it asserts that all system payloads have BOTH a "encoding":"binary/protobuf" AND "messageType" metadata key.

That's a problem.

Now, if the Temporal server were to pick up the latest api-go bits, it will introduce test failures. Because we have tests that return payloads encoded with plain/json that will fail when ran through the SDK's payload visitor at runtime.

What changed?

This PR makes the requirements surrounding a system payload clearer, and ONLY flags a System Nexus Endpoint payload as a "system payload" IFF if is a properly labeled protobuf message. Otherwise, it doesn't set the system payload tag at all.

This PR now adds a check that the payload sent to the System Nexus Endpoint is a protobuf with message type available. It also updates a testcase that was sending plain/json responses to the SNE and added a new testcase.

How did you test it?

• built
• run locally and tested manually
• covered by existing tests
• added new unit test(s)
• added new functional test(s)

Potential risks

We've already shipped code that includes the "label non-Protobuf payloads as system payloads". So it's possible that we've persisted those somewhere, and updating the api-go dependency (unless patched there) would cause problems.

[github-actions]

Claude finished @chrsmith's task in 4m 53s —— View job

---

Reviewed 7 files, 5 findings.
• branch chrsmith/fix-system-payload-issue

[chrsmith]

My BRILLIANT RELIANCE on Cunningham's Law was successful. Instead, I'll greatly scale this PR back to just:

1. Changing that test to emit an actual protobuf instead of a JSON payload.
2. Adding a validation check that SNE will only accept protobuf-encoded payloads.

That closes the gap, and ensures we don't run into some weird case like this in the future.

[chrsmith]

@bergundy PTAL

[bergundy]

Still would rather use an assertion that would log or panic over just returning plain errors but I would be okay approving this as is.