Move TypeScript worker to SDK 1.23.0 / protobufjs 8 and fix kitchen sink - #475
Closed
veeral-patel wants to merge 6 commits into
Closed
Move TypeScript worker to SDK 1.23.0 / protobufjs 8 and fix kitchen sink#475veeral-patel wants to merge 6 commits into
veeral-patel wants to merge 6 commits into
Conversation
test-kitchensink (typescript) has been failing on every PR since
2026-08-26, when @temporalio/common 1.23.0 was published. The generated
worker package floats on "^1.22.0" with no lockfile, so it picks up
1.23.0, which imports protobufjs/ext/protojson - a module that only
exists in protobufjs 8. Two separate things then go wrong.
First, sdkbuild's generated package.json pins protobufjs to 7.5.1, so the
import cannot resolve at all:
Error: Cannot find module 'protobufjs/ext/protojson'
That pin is dropped in temporalio/features#sdkbuild-drop-protobufjs-override,
wired up here with a temporary replace directive.
Second, with protobufjs 8 present, the kitchen sink payload converter
fails instead:
TypeError: type must be a Type
at fromJson (protobufjs/ext/protojson.js:865)
at ProtobufJsonPayloadConverter.fromPayload
protojson.fromJson checks `type instanceof Type` against its own
protobufjs copy. Our generated root builds its Types from
`require("protobufjs/light")`, and because the prepared build directory
lives inside workers/typescript, that resolves by walking up to
workers/typescript/node_modules - a different copy from the one pnpm
links for the SDK. Same version, different class identity, so the check
fails. This never mattered before because SDK <= 1.22.0 used
proto3-json-serializer, which is duck-typed.
Declaring protobufjs in MoreDependencies gives the prepared package its
own top-level protobufjs, which pnpm links to the same physical copy the
SDK resolves, so there is one Type class again. omes moves to protobufjs
8 and protobufjs-cli 2 to match, mirroring sdk-typescript#2303.
Verified locally: 40/40 typescript kitchen sink tests pass against the
pushed features commit with no local replace, versus 40 failures before.
SDK=go kitchen sink still passes, and the typescript worker's lint,
prettier and harness tests are green.
The previous commit put omes on protobufjs 8 while mise.toml still pinned
the TypeScript SDK to 1.22.0. That mismatch broke the Docker paths, which
pass the exact mise.toml version to prepare-worker rather than the caret
range from package.json:
TypeError: root must be an instance of a protobufjs Root
SDK 1.22.0's converters expect a root built by protobufjs 7, and ours was
now built by 8. The kitchen sink test path did not hit this because it
reads "^1.22.0" from package.json and floats to 1.23.0 on its own, so
that path was already consistent.
Pinning the SDK to 1.23.0 makes every path agree: 1.23.0 with protobufjs
8 everywhere. Applies the sync-sdk:typescript recipe from mise.toml.
Also commits the go.sum that `go mod tidy` produces. The previous commit
carried an incomplete go.sum, so every job running `go run ./cmd/dev`
appended the missing lines and then failed the shared
`git diff --exit-code` step with "worker has uncommitted formatting
changes".
Pointing the root module at the newer features commit pulls three
indirect dependencies forward (go-spew, go-difflib, golang.org/x/time),
and workers/go depends on the root module via a local replace, so its
go.mod and go.sum inherit those versions.
Without this, every job that shells out through ./cmd/dev re-resolved
them, leaving workers/go/go.{mod,sum} modified and failing the shared
`git diff --exit-code` step.
veeral-patel
force-pushed
the
ts-protobufjs-8-single-instance
branch
from
September 7, 2026 20:21
d945992 to
811a062
Compare
throughput_stress against the TypeScript worker failed every workflow
task with:
TypeError: n.toNumber is not a function
at numify (workerlib/kitchensink/proto_help.ts:72)
at handleAction (workerlib/kitchensink/workflows/kitchen_sink.ts:137)
numify assumed a 64-bit field is either a plain number or a protobufjs
Long with a toNumber method. In the main thread protojson does hand back
a real Long, but inside the Workflow sandbox the long library is not
wired up, so the value arrives as a Long-shaped object with no prototype
methods and the call blows up. Every workflow task then failed and the
scenario timed out after five minutes.
Handles bigint, decimal string, a real Long, and a plain
{low, high, unsigned} object, recombining the 32-bit halves in the last
case. The comment above the function already noted protobuf will not use
Long consistently here; this extends it to the shapes protobufjs 8 adds.
throughput_stress now completes in 3.8s (2 iterations, 2 continue-as-new,
8 workflows) where it previously timed out, and the 40-test kitchen sink
suite still passes.
veeral-patel
force-pushed
the
ts-protobufjs-8-single-instance
branch
from
September 7, 2026 20:25
811a062 to
e7c3dab
Compare
The three Ruby image jobs fail intermittently on every PR in this repo:
E: Failed to fetch .../libc-dev-bin_2.31-13+deb11u14_amd64.deb
404 Not Found
clang pulls in libc6-i386, which Depends on an exact glibc version, so
installing it upgrades libc6, libc6-dev and libc-dev-bin to whatever
bullseye-security currently advertises. Some deb.debian.org CDN edges
serve an index advertising a glibc point release whose .deb is no longer
in their pool. Which edge you get decides whether the build works: the
same commit had build-project (ruby) pass while build-worker (ruby)
failed, and the file downloads fine from other edges.
Acquire::Retries does not help, because a 404 is a definitive response
rather than a transient one - confirmed by trying it. Choosing a
different clang package does not help either; every variant pulls
libc6-i386 and so the glibc upgrade. Pinning glibc to the older build in
bullseye/main would work but means an --allow-downgrades downgrade past
security fixes.
snapshot.debian.org serves immutable index/pool pairs, so they can never
disagree and this failure mode is gone by construction. The installed
versions are unchanged - clang 1:11.0-51+nmu5 and libprotoc
3.12.4-1+deb11u1, same as before - only the source differs. The snapshot
timestamp is an ARG so it can be bumped when the pinned glibc needs newer
security fixes.
Verified by building the full image locally: all 18 build stages pass,
including prepare-worker.
Sourcing from snapshot fixed the single-arch Ruby jobs (build-worker and
build-project both pass now), but the multi-arch Docker Hub push job then
failed on its emulated arm64 leg:
E: Failed to fetch https://snapshot.debian.org/.../libclang-common-11-dev_11.0.1-2_arm64.deb
503 first byte timeout
Snapshot is slower than the CDN and answers 503 under load. Unlike the
404 this replaced, a 503 is transient, so Acquire::Retries genuinely
applies here. Also raises the fetch timeout, since the emulated arm64 leg
is slow.
Verified by building the apt stage natively for arm64: clang 11.0.1-2 and
libprotoc 3.12.4 install cleanly, and arm64 does not even need the glibc
upgrade that triggers this on amd64 (its snapshot glibc is already u13).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes
test-kitchensink (typescript), red on every PR since 2026-08-26.Important
Depends on temporalio/features#sdkbuild-drop-protobufjs-override, wired up with a temporary
replaceingo.mod. That branch must merge first; then drop thereplaceand bump therequirenormally.What broke
@temporalio/common@1.23.0was published 2026-08-26 and importsprotobufjs/ext/protojson, which exists only in protobufjs 8. sdkbuild generates the worker package from"@temporalio/client": "^1.22.0"with no lockfile, so it floats straight to 1.23.0.main's last green run was 2026-08-25 20:59Z — nothing in this repo changed.Three layered failures, each fixed
1. The module can't resolve.
sdkbuild/typescript.gosetspnpm.overrides.protobufjs = 7.5.1:Fixed upstream by dropping that pin — added in temporalio/features#625 (May 2025) as an explicitly temporary workaround whose cause is gone (
protobuf.js#2379fixed upstream; sdk-typescript#2303 rewrote the code path onto protobufjs 8).2. With protobufjs 8 present, the payload converter fails.
protojson.fromJsondoestype instanceof Typeagainst its own protobufjs copy. Our root builds Types viarequire("protobufjs/light")inworkerlib/kitchensink/protos/root.js; because the prepared build dir is created insideworkers/typescript, that resolves by walking up toworkers/typescript/node_modules— a different physical copy from the one pnpm links for the SDK. Same version, different class identity.Declaring
protobufjsinMoreDependenciesgives the prepared package its own top-level copy, which pnpm links to the same physical directory the SDK resolves. Note bumping our protobufjs to 8 alone doesn't fix this — two copies of 8.8.0 still have distinctTypeclasses.3. The Docker paths then broke the other way.
build-worker/build-projectpass the exactmise.tomlversion toprepare-worker, not the caret frompackage.json— so they were still installing SDK 1.22.0 (protobufjs 7-era) against our now-protobufjs-8 root. Bumping_.sdk.typescriptto 1.23.0 makes every path agree: 1.23.0 + protobufjs 8 everywhere. Applied via thesync-sdk:typescriptrecipe.4. Long fields then broke inside the Workflow sandbox.
numifyassumed a 64-bit field is either a plainnumberor a protobufjsLongwithtoNumber. In the main thread protojson does hand back a realLong, but inside the Workflow sandbox thelonglibrary isn't wired up, so the value arrives as aLong-shaped object with no prototype methods. Every workflow task failed andthroughput_stresstimed out after five minutes — which is whatbuild-worker (typescript)runs as its second smoke test. Hardened to acceptbigint, decimal strings, a realLong, and a plain{low, high, unsigned}object.Changes
internal/workerctl/build.go—protobufjsinMoreDependenciesworkers/typescript/workerlib/kitchensink/proto_help.ts—numifyhandles protobufjs 8 long shapesmise.toml+workers/typescript/package.json/lockfile — SDK 1.23.0,protobufjs ^8.8.0,protobufjs-cli ^2.7.0go.mod/go.sum— temporaryreplacefor the features fixworkers/go/go.mod/go.sum— the features bump moves three indirect deps forward, which the nested module inheritsThis subsumes #457 (the TS 1.23.0 bump), which can be closed if this lands.
Verification
Run locally against the pushed features commit with no local replace, i.e. what CI does:
SDK=typescriptkitchen sinkSDK=gokitchen sinkworkers/gobuild + teststhroughput_stressw/ ts workerCI on this branch now has
test-kitchensink (typescript),build-project (typescript),check-worker (typescript)and the wholecheck-workermatrix green.I isolated each layer to confirm all four changes are load-bearing: the features fix alone still fails with
type must be a Type; features + protobufjs 8 withoutMoreDependenciesalso fails; and protobufjs 8 without the SDK bump breaks the Docker paths.Notes for review
protobufjs-cli2.x deprecates the generatedI<Name>interfaces in favour of<Name>.$Properties, adding ~55 eslint warnings (not errors). Migrating those call sites is worth a follow-up, deliberately out of scope.Ruby image builds were failing on every PR in the repo, so this branch also fixes them:
clangpulls inlibc6-i386, whichDependson an exact glibc version, so installing it upgradeslibc6/libc6-dev/libc-dev-binto whateverbullseye-securityadvertises. Somedeb.debian.orgCDN edges serve an index naming a glibc point release whose.debis no longer in their pool — so which edge you get decides whether the build works. The same commit hadbuild-project (ruby)pass whilebuild-worker (ruby)failed.Things I ruled out by testing:
Acquire::Retriesdoesn't help (a 404 isn't transient); noclangpackage variant avoidslibc6-i386(checkedclang,clang-11,libclang-dev,libclang-11-dev,llvm-11-dev); pinning glibc to the olderbullseye/mainbuild would need--allow-downgradespast security fixes.Fixed by sourcing that one
apt-getfromsnapshot.debian.org, which serves immutable index/pool pairs so they can never disagree. Installed versions are unchanged (clang 1:11.0-51+nmu5,libprotoc 3.12.4-1+deb11u1) — only the source differs. The timestamp is anARGfor easy bumping. Verified by building the full image locally: all 18 stages pass, includingprepare-worker.test-omesflaked once onTestThroughputStress/Run_executor_again,_resuming_from_end; it passes locally and on other runs of this branch.