Skip to content

CI: Run every SDK's JavaScript builder tests, not only .NET's - #29

Merged
jwrosewell merged 3 commits into
mainfrom
ci/consumer-tests-every-builder
Sep 18, 2026
Merged

jwrosewell merged 3 commits into
mainfrom
ci/consumer-tests-every-builder

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Why

Every builder that embeds these templates is expected to render the same
script from the same evidence. Until now the consumer job built one of them,
.NET, so a change that suited .NET and not another builder could not be seen
here at all.

That is what happened. A change here was correct for .NET, reached a package
registry through another SDK, and broke the session storage cache for pages
using it: the record a page keeps of its request inputs carried a value that
differs on every page view, so it could never match, the cached response was
thrown away and a request went on every page view for the life of the tab.

Four SDKs embed the templates. All four now get a job.

Job Template path in that SDK
pipeline-dotnet FiftyOne.Pipeline.Elements/FiftyOne.Pipeline.JavaScriptBuilderElement/Templates
pipeline-java pipeline.javascriptbuilder/src/main/resources/fiftyone/pipeline/javascriptbuilder/templates
pipeline-node fiftyone.pipeline.core/javascript-templates
pipeline-python fiftyone_pipeline_core/src/fiftyone_pipeline_core/js_templates

Each keeps the guard the .NET job already had, which reports when the copy
changed nothing so a green run cannot be mistaken for a check of this change
when it actually exercised the revision that SDK pins.

Java runs on 11 rather than 8 deliberately. The module drops
JavaScriptBuilderTests at test compilation on 8, because Selenium's classes
are built for 11:

<testExclude>**/JavaScriptBuilderTests.java</testExclude>

So a run on 8 reports a pass without ever having built the tests that drive
the template. That is worth knowing separately from this change.

What this does not do

It runs each SDK's own tests, so it is only as good as those tests, and only
.NET drives the template in a real browser today. It would not by itself
have caught the change that prompted it
, because the SDK that broke has no
test that loads a second page view. What it does is make a template change
visible to every builder that embeds it, which is the precondition for any of
them catching anything.

The check that would settle it is rendering one fixed set of evidence through
all four and comparing the output, since all four are meant to produce the
same script. That needs a way to render from fixed evidence in each builder,
which none of them exposes today, and it is a larger piece of work in four
repositories rather than one file here.

Verified

The workflow parses and declares the four jobs:

pipeline-dotnet -> pipeline-dotnet JavaScript builder | 7 steps
pipeline-java   -> pipeline-java JavaScript builder   | 7 steps
pipeline-node   -> pipeline-node JavaScript builder   | 5 steps
pipeline-python -> pipeline-python JavaScript builder | 5 steps

The test commands are the ones I ran by hand in each SDK today whilst fixing
the builders, so they are known to work rather than inferred from the
repositories' own CI.

The three new jobs have not run yet. This pull request is their first run,
which is the point at which the paths and commands are proved rather than
asserted.

Merge order

The three ready pull requests in this repository take this order:

29, 32, 31

This one goes first, so nothing has to merge before it.

This one changes only .github/workflows/consumer-tests.yml, which neither of the others touches.

No branch had to be changed for this order. Every pair was measured with git merge-tree, the three were then merged into main in that order as real merges with no conflict, and npm test in tests on the result reports 207 checks, 0 failures. Smallest and most self-contained first. Checked on 17 September 2026.

The templates are consumed as a submodule, so a change here is only ever
exercised by the SDKs that embed it. Four do, being .NET, Java, Node and
Python, and until now the consumer job built one of them.

That is how a change that was correct for .NET reached a package registry
and broke the session storage cache for Python. Every builder is expected
to render the same script from the same evidence, and one of them did not,
which nothing here could see because nothing here built it.

Each SDK now gets a job that checks it out, swaps in the revision under
review and runs its JavaScript builder tests, with the same guard the .NET
job already had that says so when the copy changed nothing and the run
would otherwise report a pass about the revision the SDK pins rather than
about this change.

Java runs on 11 rather than 8 on purpose. The module drops
JavaScriptBuilderTests at test compilation on 8, because Selenium's classes
are built for 11, so a run on 8 reports a pass without having built the
tests that drive the template at all.

What this does not do. It runs each SDK's own tests, so it is only as good
as those tests, and only .NET drives the template in a real browser today.
It would not by itself have caught the change that prompted it, because
Python has no test that loads a second page view. What it does is make a
template change visible to every builder that embeds it, which is the
precondition for any of them catching anything.

The check that would settle it is rendering one fixed set of evidence
through all four and comparing the output, since all four are meant to
produce the same script. That needs a way to render from fixed evidence in
each builder, which none of them expose today.
Six SDKs render these templates, not four. PHP and Rust carry a copy of the
file rather than a submodule, so a change here never reaches them on its
own and the copies drift. Today the PHP copy matches the create last merge
without the fix to the inputs record, and the Rust copy is the revision
before create last, so Rust serves a different script from the other five.

Both now get a job that swaps the revision under review into the copy and
runs the SDK's tests. Because a copy that differs is itself a difference in
what that SDK serves, the job reports it as a warning rather than a notice.
@jwrosewell

Copy link
Copy Markdown
Contributor Author

The three red consumer jobs, and what each one was

All seven checks now pass. None of the three failures was in this branch's
workflow, and two of them were already fixed in the consumer before this was
looked at.

rust JavaScript builder

no_url_means_no_update_section asserted that a script rendered with updates
off never mentions processRequest, and the template on this branch still
named that function in a comment sitting outside the {{#_updateEnabled}}
section. That is exactly what
#30 fixed on
main, and this branch had not taken it. main is now merged in, all three
uses of the name sit inside {{#_updateEnabled}}, and the job passes.

pipeline-java JavaScript builder

JavaScriptBuilderElement_ParametersExcludeSessionAndSequence failed with
"the script should assign its parameters ==> expected: not ". At the
revision of pipeline-java the run checked out, the test looked for a line
containing the literal parameters = {, and the template declares
var renderedParameters = function() { return {...}; };, so nothing matched
and the search returned nothing to assert against. pipeline-java's main has
since changed that test to match on the word and a brace rather than the
exact name, which is why the job passes on a fresh run.

pipeline-node JavaScript builder

The same check, "the script parameters leave out the session id and the
sequence", timing out at five seconds rather than reporting a failure,
because the assertion threw inside the promise and done was never called.
pipeline-node's main has since given that test a host, a protocol and an
endpoint, so the parameters are rendered at all, and matched the declaration
with a regular expression instead of the old name. It passes on a fresh run
too.

Worth noting for whoever reviews this

The java job's swap step reported the template it copies in as 763 lines
added and 97 removed against the revision pipeline-java pins, which is the
gap this workflow exists to show. Two of the three failures were a consumer
test naming a declaration the template has since renamed, and both were
found here rather than after a package carrying the change was published,
which is what the workflow's own comment says it is for.

Evidence

Run 35186467794 on 4378f81, all seven checks green, being pipeline-dotnet,
pipeline-java, pipeline-node, pipeline-python, pipeline-php-core and rust,
with the template tests alongside.

@jwrosewell
jwrosewell marked this pull request as ready for review September 17, 2026 20:01
@jwrosewell
jwrosewell merged commit 6f2121a into main Sep 18, 2026
7 checks passed
@jwrosewell
jwrosewell deleted the ci/consumer-tests-every-builder branch September 18, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant