Skip to content

feat(maestro): support evalScript inline JavaScript expressions - #2158

Open
Rohit3523 wants to merge 3 commits into
callstack:mainfrom
Rohit3523:feat/maestro-evalscript
Open

feat(maestro): support evalScript inline JavaScript expressions#2158
Rohit3523 wants to merge 3 commits into
callstack:mainfrom
Rohit3523:feat/maestro-evalscript

Conversation

@Rohit3523

Copy link
Copy Markdown

Summary

Adds evalScript to the supported Maestro YAML subset. The command evaluates a
single-line JavaScript expression with flow env values and prior output
leaves in scope, then folds the assigned output object back into the flat
string-key variable model so later steps resolve ${output.x} and
${output.list.length}.

env:
  MY_NAME: John
---
- evalScript: ${output.uppercaseName = MY_NAME.toUpperCase()}
- inputText: ${output.uppercaseName}

Evaluation is host-side in the compute engine (node:vm, matching runScript
trust semantics); the command is never dispatched to the device port. Without a
device round trip, the engine merges output leaves directly into flow scope and
marks the step executed. repeat.times: ${output.list.length} works because
assigned arrays fold to index and length leaves. --from resume refuses to
skip evalScript steps, since they produce output.

Validation

  • pnpm test:maestro-compat: 335 unit tests pass (new parser, engine, and
    engine-eval-script eval tests).
  • pnpm maestro:conformance: 57 pass. upstream/053_repeat_times reclassifies
    from we-reject (evalScript unsupported) to identical; the declared
    divergence is removed.
  • Live Android evidence: installed the test-app release build on
    emulator-5554, ran an eval-script flow; output.sum = 1 + 2 then
    inputText: ${output.sum} landed the computed 3 in the focused field
    (field value Ada Lovelace3), proving arbitrary-expression evaluation and
    ${output.x} consumption end-to-end on a device.

Tradeoffs and follow-ups

  • evalScript output is string-typed across steps (runScript parity): JS
    object/array identity is not preserved, so ${output.list.length} and
    indexed leaves resolve but in-flow mutation like output.list.push() does
    not.
  • evalScript is the only command whose payload runs as JavaScript; all other
    fields stay literal/${VAR}-lookup-only (assertTrue scoped to lookups,
    repeat.while unsupported).
  • No new issue to close; ADR 0015 and the help maestro support matrix updated
    to document the evalScript boundary.

13 files touched, +298/−15, within the Maestro compatibility engine and its
conformance/docs. No scope growth outside that family.

@thymikee

Copy link
Copy Markdown
Member

Reviewed exact head 1b4e08204425c6a16175426469e3734942424e54. This is blocked by a security boundary, plus one vacuous regression.

P0 — remote caller-supplied YAML gains daemon-host code execution. Maestro source bundles are sent to the daemon; engine-eval-script.ts passes the inline expression to vm.runInNewContext. The supplied { ...values, output } context is escapable through the global constructor chain (reproduced by reading process.versions.node), which enables filesystem/environment access, child processes, and private-network requests. node:vm is explicitly not a security mechanism, and without microtaskMode: 'afterEvaluate' promise work can also escape the timeout. The existing daemon-local runScript trust contract does not transfer to inline remote payloads. The smallest safe contract is to reject evalScript for remotely supplied flows; remote support requires an OS/process-isolated evaluator with strict CPU, memory, filesystem, and network boundaries.

P2 — resume safety is untested. replay-plan-resume.ts adds evalScript to the no-skip condition, but the resume fixture only contains runScript; deleting the new condition still leaves the suite green. Add an explicit --from/plan-digest regression with a preceding evalScript.

After redesigning the trust boundary, add planted-red remote adversarial tests for process/filesystem/child-process/private-network access and execution-budget escape, then run exact-head CI/Size and repeat device evidence against the shipped path. Reference: https://nodejs.org/api/vm.html

@thymikee

thymikee commented Sep 4, 2026

Copy link
Copy Markdown
Member

Sentinel recheck: BLOCKED. The existing P0 remains: remote caller-supplied Maestro YAML reaches vm.runInNewContext, and the constructor chain escapes to daemon-host process; node:vm is not an isolation boundary. Reject evalScript for remote flows or use real process/OS isolation before shipping.

The existing P2 also remains: the added evalScript no-skip condition has no explicit resume regression.

One additional consistency defect: scripts/maestro-conformance/build-manifest.mjs and the generated corpus manifest still say upstream flow 053 uses unsupported evalScript, while this PR removes that declared divergence. Update/regenerate the manifest after the trust-boundary redesign.

The exact head still has no GitHub checks. Do not apply ready-for-human.

Add evalScript as a supported Maestro command: a single-line JavaScript
expression evaluated against flow env and prior output leaves, with the
assigned output object folded back into the flat string-key variable
model so ${output.x} and ${output.list.length} resolve in later steps.
The command is handled host-side by the compute engine (node:vm) and
never dispatched to the device port.

Update the support matrix, ADR 0015, the conformance expected-divergence
(upstream/053_repeat_times is now identical), and add parser, engine, and
eval-module unit tests.
Reject evalScript when trustedScripts is false (remote HTTP surface
via publicNetworkOnly), since node:vm is not a security sandbox.
Add engine adversarial refusal matrix and handler wiring test,
explicit runScript/evalScript resume regression, and fix stale
upstream 053 manifest note. Also fix lint/typecheck in touched files.
@Rohit3523
Rohit3523 force-pushed the feat/maestro-evalscript branch from 04ad0b9 to 582eb29 Compare September 6, 2026 14:33
@Rohit3523

Copy link
Copy Markdown
Author

Done with the changes, Can you review it, Thanks. I actually fixed it last week but forgot to commit it.

@thymikee

thymikee commented Sep 6, 2026

Copy link
Copy Markdown
Member

The remote-script restriction and the earlier resume/manifest gaps are addressed at 582eb29. One correctness issue remains: evalScript merges flattened output into the old variables without removing missing keys. After assigning output.list = [1, 2, 3], then output.list = [4], a later ${output.list.2} still resolves to 3. Replace the previous output namespace when committing the result, and add a regression for array shrinkage and deleted values. This head also has no CI checks yet.

evalScript flattened the new output object correctly, but merged it
over the old variables, so shrunken arrays and deleted leaves survived
(e.g. output.list=[1,2,3] then [4] still resolved output.list.2=3).
Commit via replaceOutput that clears output/output.* before writing.
@Rohit3523

Copy link
Copy Markdown
Author

done with the fix :)

@thymikee

thymikee commented Sep 6, 2026

Copy link
Copy Markdown
Member

The stale-output issue is fixed at 2edf630. Replacing the output namespace removes old array entries and deleted properties, and the added regressions exercise that path. No further finding in this delta; this head still has no CI checks, so it is not merge-ready yet.

@thymikee

thymikee commented Sep 7, 2026

Copy link
Copy Markdown
Member

Code review is clear at 2edf630, and the reported Android flow covers expression output reaching the device. This is ready for human review. There are still no CI checks reported on this head, so CI validation remains unknown.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 7, 2026
@thymikee

thymikee commented Sep 7, 2026

Copy link
Copy Markdown
Member

The new coverage failures look related: the fuzz model still treats a newly supported command as unsupported, and the replay documentation no longer matches the compatibility help. Please update those expectations and docs, then rerun coverage. Android smoke failed at automation-press, which looks unrelated. Code-review readiness is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants