feat(maestro): support evalScript inline JavaScript expressions - #2158
feat(maestro): support evalScript inline JavaScript expressions#2158Rohit3523 wants to merge 3 commits into
Conversation
|
Reviewed exact head P0 — remote caller-supplied YAML gains daemon-host code execution. Maestro source bundles are sent to the daemon; P2 — resume safety is untested. After redesigning the trust boundary, add planted-red remote adversarial tests for process/filesystem/child-process/private-network access and execution-budget escape, then run exact-head CI/Size and repeat device evidence against the shipped path. Reference: https://nodejs.org/api/vm.html |
|
Sentinel recheck: BLOCKED. The existing P0 remains: remote caller-supplied Maestro YAML reaches The existing P2 also remains: the added One additional consistency defect: The exact head still has no GitHub checks. Do not apply |
Add evalScript as a supported Maestro command: a single-line JavaScript
expression evaluated against flow env and prior output leaves, with the
assigned output object folded back into the flat string-key variable
model so ${output.x} and ${output.list.length} resolve in later steps.
The command is handled host-side by the compute engine (node:vm) and
never dispatched to the device port.
Update the support matrix, ADR 0015, the conformance expected-divergence
(upstream/053_repeat_times is now identical), and add parser, engine, and
eval-module unit tests.
Reject evalScript when trustedScripts is false (remote HTTP surface via publicNetworkOnly), since node:vm is not a security sandbox. Add engine adversarial refusal matrix and handler wiring test, explicit runScript/evalScript resume regression, and fix stale upstream 053 manifest note. Also fix lint/typecheck in touched files.
04ad0b9 to
582eb29
Compare
|
Done with the changes, Can you review it, Thanks. I actually fixed it last week but forgot to commit it. |
|
The remote-script restriction and the earlier resume/manifest gaps are addressed at 582eb29. One correctness issue remains: evalScript merges flattened output into the old variables without removing missing keys. After assigning output.list = [1, 2, 3], then output.list = [4], a later ${output.list.2} still resolves to 3. Replace the previous output namespace when committing the result, and add a regression for array shrinkage and deleted values. This head also has no CI checks yet. |
evalScript flattened the new output object correctly, but merged it over the old variables, so shrunken arrays and deleted leaves survived (e.g. output.list=[1,2,3] then [4] still resolved output.list.2=3). Commit via replaceOutput that clears output/output.* before writing.
|
done with the fix :) |
|
The stale-output issue is fixed at 2edf630. Replacing the output namespace removes old array entries and deleted properties, and the added regressions exercise that path. No further finding in this delta; this head still has no CI checks, so it is not merge-ready yet. |
|
Code review is clear at 2edf630, and the reported Android flow covers expression output reaching the device. This is ready for human review. There are still no CI checks reported on this head, so CI validation remains unknown. |
|
The new coverage failures look related: the fuzz model still treats a newly supported command as unsupported, and the replay documentation no longer matches the compatibility help. Please update those expectations and docs, then rerun coverage. Android smoke failed at automation-press, which looks unrelated. Code-review readiness is unchanged. |
Summary
Adds
evalScriptto the supported Maestro YAML subset. The command evaluates asingle-line JavaScript expression with flow env values and prior
outputleaves in scope, then folds the assigned
outputobject back into the flatstring-key variable model so later steps resolve
${output.x}and${output.list.length}.Evaluation is host-side in the compute engine (
node:vm, matchingrunScripttrust semantics); the command is never dispatched to the device port. Without a
device round trip, the engine merges output leaves directly into flow scope and
marks the step executed.
repeat.times: ${output.list.length}works becauseassigned arrays fold to index and
lengthleaves.--fromresume refuses toskip evalScript steps, since they produce output.
Validation
pnpm test:maestro-compat: 335 unit tests pass (new parser, engine, andengine-eval-scripteval tests).pnpm maestro:conformance: 57 pass.upstream/053_repeat_timesreclassifiesfrom
we-reject(evalScript unsupported) toidentical; the declareddivergence is removed.
emulator-5554, ran an eval-script flow;output.sum = 1 + 2theninputText: ${output.sum}landed the computed3in the focused field(field value
Ada Lovelace3), proving arbitrary-expression evaluation and${output.x}consumption end-to-end on a device.Tradeoffs and follow-ups
object/array identity is not preserved, so
${output.list.length}andindexed leaves resolve but in-flow mutation like
output.list.push()doesnot.
fields stay literal/
${VAR}-lookup-only (assertTruescoped to lookups,repeat.whileunsupported).help maestrosupport matrix updatedto document the evalScript boundary.
13 files touched, +298/−15, within the Maestro compatibility engine and its
conformance/docs. No scope growth outside that family.