Skip to content

Move TypeScript worker to SDK 1.23.0 / protobufjs 8 and fix kitchen sink - #475

Closed
veeral-patel wants to merge 6 commits into
mainfrom
ts-protobufjs-8-single-instance
Closed

Move TypeScript worker to SDK 1.23.0 / protobufjs 8 and fix kitchen sink#475
veeral-patel wants to merge 6 commits into
mainfrom
ts-protobufjs-8-single-instance

Conversation

@veeral-patel

@veeral-patel veeral-patel commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Fixes test-kitchensink (typescript), red on every PR since 2026-08-26.

Important

Depends on temporalio/features#sdkbuild-drop-protobufjs-override, wired up with a temporary replace in go.mod. That branch must merge first; then drop the replace and bump the require normally.

What broke

@temporalio/common@1.23.0 was published 2026-08-26 and imports protobufjs/ext/protojson, which exists only in protobufjs 8. sdkbuild generates the worker package from "@temporalio/client": "^1.22.0" with no lockfile, so it floats straight to 1.23.0. main's last green run was 2026-08-25 20:59Z — nothing in this repo changed.

Three layered failures, each fixed

1. The module can't resolve. sdkbuild/typescript.go sets pnpm.overrides.protobufjs = 7.5.1:

Error: Cannot find module 'protobufjs/ext/protojson'

Fixed upstream by dropping that pin — added in temporalio/features#625 (May 2025) as an explicitly temporary workaround whose cause is gone (protobuf.js#2379 fixed upstream; sdk-typescript#2303 rewrote the code path onto protobufjs 8).

2. With protobufjs 8 present, the payload converter fails.

TypeError: type must be a Type
  at fromJson (protobufjs/ext/protojson.js:865)

protojson.fromJson does type instanceof Type against its own protobufjs copy. Our root builds Types via require("protobufjs/light") in workerlib/kitchensink/protos/root.js; because the prepared build dir is created inside workers/typescript, that resolves by walking up to workers/typescript/node_modules — a different physical copy from the one pnpm links for the SDK. Same version, different class identity.

Declaring protobufjs in MoreDependencies gives the prepared package its own top-level copy, which pnpm links to the same physical directory the SDK resolves. Note bumping our protobufjs to 8 alone doesn't fix this — two copies of 8.8.0 still have distinct Type classes.

3. The Docker paths then broke the other way.

TypeError: root must be an instance of a protobufjs Root

build-worker / build-project pass the exact mise.toml version to prepare-worker, not the caret from package.json — so they were still installing SDK 1.22.0 (protobufjs 7-era) against our now-protobufjs-8 root. Bumping _.sdk.typescript to 1.23.0 makes every path agree: 1.23.0 + protobufjs 8 everywhere. Applied via the sync-sdk:typescript recipe.

4. Long fields then broke inside the Workflow sandbox.

TypeError: n.toNumber is not a function
  at numify (workerlib/kitchensink/proto_help.ts:72)

numify assumed a 64-bit field is either a plain number or a protobufjs Long with toNumber. In the main thread protojson does hand back a real Long, but inside the Workflow sandbox the long library isn't wired up, so the value arrives as a Long-shaped object with no prototype methods. Every workflow task failed and throughput_stress timed out after five minutes — which is what build-worker (typescript) runs as its second smoke test. Hardened to accept bigint, decimal strings, a real Long, and a plain {low, high, unsigned} object.

Changes

  • internal/workerctl/build.goprotobufjs in MoreDependencies
  • workers/typescript/workerlib/kitchensink/proto_help.tsnumify handles protobufjs 8 long shapes
  • mise.toml + workers/typescript/package.json/lockfile — SDK 1.23.0, protobufjs ^8.8.0, protobufjs-cli ^2.7.0
  • go.mod/go.sum — temporary replace for the features fix
  • workers/go/go.mod/go.sum — the features bump moves three indirect deps forward, which the nested module inherits

This subsumes #457 (the TS 1.23.0 bump), which can be closed if this lands.

Verification

Run locally against the pushed features commit with no local replace, i.e. what CI does:

before after
SDK=typescript kitchen sink 40 fail 40/40 pass
SDK=go kitchen sink pass pass
ts worker eslint + prettier pass pass (0 errors)
ts worker harness tests pass 10/10 pass
workers/go build + tests pass pass
throughput_stress w/ ts worker 5min timeout completes in 3.8s

CI on this branch now has test-kitchensink (typescript), build-project (typescript), check-worker (typescript) and the whole check-worker matrix green.

I isolated each layer to confirm all four changes are load-bearing: the features fix alone still fails with type must be a Type; features + protobufjs 8 without MoreDependencies also fails; and protobufjs 8 without the SDK bump breaks the Docker paths.

Notes for review

  • protobufjs-cli 2.x deprecates the generated I<Name> interfaces in favour of <Name>.$Properties, adding ~55 eslint warnings (not errors). Migrating those call sites is worth a follow-up, deliberately out of scope.

  • Ruby image builds were failing on every PR in the repo, so this branch also fixes them:

    E: Failed to fetch .../libc-dev-bin_2.31-13+deb11u14_amd64.deb  404  Not Found
    

    clang pulls in libc6-i386, which Depends on an exact glibc version, so installing it upgrades libc6/libc6-dev/libc-dev-bin to whatever bullseye-security advertises. Some deb.debian.org CDN edges serve an index naming a glibc point release whose .deb is no longer in their pool — so which edge you get decides whether the build works. The same commit had build-project (ruby) pass while build-worker (ruby) failed.

    Things I ruled out by testing: Acquire::Retries doesn't help (a 404 isn't transient); no clang package variant avoids libc6-i386 (checked clang, clang-11, libclang-dev, libclang-11-dev, llvm-11-dev); pinning glibc to the older bullseye/main build would need --allow-downgrades past security fixes.

    Fixed by sourcing that one apt-get from snapshot.debian.org, which serves immutable index/pool pairs so they can never disagree. Installed versions are unchanged (clang 1:11.0-51+nmu5, libprotoc 3.12.4-1+deb11u1) — only the source differs. The timestamp is an ARG for easy bumping. Verified by building the full image locally: all 18 stages pass, including prepare-worker.

  • test-omes flaked once on TestThroughputStress/Run_executor_again,_resuming_from_end; it passes locally and on other runs of this branch.

test-kitchensink (typescript) has been failing on every PR since
2026-08-26, when @temporalio/common 1.23.0 was published. The generated
worker package floats on "^1.22.0" with no lockfile, so it picks up
1.23.0, which imports protobufjs/ext/protojson - a module that only
exists in protobufjs 8. Two separate things then go wrong.

First, sdkbuild's generated package.json pins protobufjs to 7.5.1, so the
import cannot resolve at all:

    Error: Cannot find module 'protobufjs/ext/protojson'

That pin is dropped in temporalio/features#sdkbuild-drop-protobufjs-override,
wired up here with a temporary replace directive.

Second, with protobufjs 8 present, the kitchen sink payload converter
fails instead:

    TypeError: type must be a Type
      at fromJson (protobufjs/ext/protojson.js:865)
      at ProtobufJsonPayloadConverter.fromPayload

protojson.fromJson checks `type instanceof Type` against its own
protobufjs copy. Our generated root builds its Types from
`require("protobufjs/light")`, and because the prepared build directory
lives inside workers/typescript, that resolves by walking up to
workers/typescript/node_modules - a different copy from the one pnpm
links for the SDK. Same version, different class identity, so the check
fails. This never mattered before because SDK <= 1.22.0 used
proto3-json-serializer, which is duck-typed.

Declaring protobufjs in MoreDependencies gives the prepared package its
own top-level protobufjs, which pnpm links to the same physical copy the
SDK resolves, so there is one Type class again. omes moves to protobufjs
8 and protobufjs-cli 2 to match, mirroring sdk-typescript#2303.

Verified locally: 40/40 typescript kitchen sink tests pass against the
pushed features commit with no local replace, versus 40 failures before.
SDK=go kitchen sink still passes, and the typescript worker's lint,
prettier and harness tests are green.
@veeral-patel
veeral-patel requested review from a team as code owners September 7, 2026 19:44
The previous commit put omes on protobufjs 8 while mise.toml still pinned
the TypeScript SDK to 1.22.0. That mismatch broke the Docker paths, which
pass the exact mise.toml version to prepare-worker rather than the caret
range from package.json:

    TypeError: root must be an instance of a protobufjs Root

SDK 1.22.0's converters expect a root built by protobufjs 7, and ours was
now built by 8. The kitchen sink test path did not hit this because it
reads "^1.22.0" from package.json and floats to 1.23.0 on its own, so
that path was already consistent.

Pinning the SDK to 1.23.0 makes every path agree: 1.23.0 with protobufjs
8 everywhere. Applies the sync-sdk:typescript recipe from mise.toml.

Also commits the go.sum that `go mod tidy` produces. The previous commit
carried an incomplete go.sum, so every job running `go run ./cmd/dev`
appended the missing lines and then failed the shared
`git diff --exit-code` step with "worker has uncommitted formatting
changes".
Pointing the root module at the newer features commit pulls three
indirect dependencies forward (go-spew, go-difflib, golang.org/x/time),
and workers/go depends on the root module via a local replace, so its
go.mod and go.sum inherit those versions.

Without this, every job that shells out through ./cmd/dev re-resolved
them, leaving workers/go/go.{mod,sum} modified and failing the shared
`git diff --exit-code` step.
@veeral-patel veeral-patel changed the title Fix TypeScript kitchen sink against SDK 1.23.0 / protobufjs 8 Move TypeScript worker to SDK 1.23.0 / protobufjs 8 and fix kitchen sink Sep 7, 2026
@veeral-patel
veeral-patel force-pushed the ts-protobufjs-8-single-instance branch from d945992 to 811a062 Compare September 7, 2026 20:21
throughput_stress against the TypeScript worker failed every workflow
task with:

    TypeError: n.toNumber is not a function
      at numify (workerlib/kitchensink/proto_help.ts:72)
      at handleAction (workerlib/kitchensink/workflows/kitchen_sink.ts:137)

numify assumed a 64-bit field is either a plain number or a protobufjs
Long with a toNumber method. In the main thread protojson does hand back
a real Long, but inside the Workflow sandbox the long library is not
wired up, so the value arrives as a Long-shaped object with no prototype
methods and the call blows up. Every workflow task then failed and the
scenario timed out after five minutes.

Handles bigint, decimal string, a real Long, and a plain
{low, high, unsigned} object, recombining the 32-bit halves in the last
case. The comment above the function already noted protobuf will not use
Long consistently here; this extends it to the shapes protobufjs 8 adds.

throughput_stress now completes in 3.8s (2 iterations, 2 continue-as-new,
8 workflows) where it previously timed out, and the 40-test kitchen sink
suite still passes.
@veeral-patel
veeral-patel force-pushed the ts-protobufjs-8-single-instance branch from 811a062 to e7c3dab Compare September 7, 2026 20:25
The three Ruby image jobs fail intermittently on every PR in this repo:

    E: Failed to fetch .../libc-dev-bin_2.31-13+deb11u14_amd64.deb
       404  Not Found

clang pulls in libc6-i386, which Depends on an exact glibc version, so
installing it upgrades libc6, libc6-dev and libc-dev-bin to whatever
bullseye-security currently advertises. Some deb.debian.org CDN edges
serve an index advertising a glibc point release whose .deb is no longer
in their pool. Which edge you get decides whether the build works: the
same commit had build-project (ruby) pass while build-worker (ruby)
failed, and the file downloads fine from other edges.

Acquire::Retries does not help, because a 404 is a definitive response
rather than a transient one - confirmed by trying it. Choosing a
different clang package does not help either; every variant pulls
libc6-i386 and so the glibc upgrade. Pinning glibc to the older build in
bullseye/main would work but means an --allow-downgrades downgrade past
security fixes.

snapshot.debian.org serves immutable index/pool pairs, so they can never
disagree and this failure mode is gone by construction. The installed
versions are unchanged - clang 1:11.0-51+nmu5 and libprotoc
3.12.4-1+deb11u1, same as before - only the source differs. The snapshot
timestamp is an ARG so it can be bumped when the pinned glibc needs newer
security fixes.

Verified by building the full image locally: all 18 build stages pass,
including prepare-worker.
Sourcing from snapshot fixed the single-arch Ruby jobs (build-worker and
build-project both pass now), but the multi-arch Docker Hub push job then
failed on its emulated arm64 leg:

    E: Failed to fetch https://snapshot.debian.org/.../libclang-common-11-dev_11.0.1-2_arm64.deb
       503  first byte timeout

Snapshot is slower than the CDN and answers 503 under load. Unlike the
404 this replaced, a 503 is transient, so Acquire::Retries genuinely
applies here. Also raises the fetch timeout, since the emulated arm64 leg
is slow.

Verified by building the apt stage natively for arm64: clang 11.0.1-2 and
libprotoc 3.12.4 install cleanly, and arm64 does not even need the glibc
upgrade that triggers this on amd64 (its snapshot glibc is already u13).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant