diff --git a/crates/bashkit/docs/jq.md b/crates/bashkit/docs/jq.md index 61eacdc04..f761d661e 100644 --- a/crates/bashkit/docs/jq.md +++ b/crates/bashkit/docs/jq.md @@ -27,6 +27,10 @@ Implemented: | `-e`, `--exit-status` | Set exit code based on the output | | `-j` | Like `-r` but suppresses trailing newlines | | `--tab` | Use tabs for indentation | +| `-a`, `--ascii-output` | Escape non-ASCII characters as `\uXXXX` (strings stay quoted even with `-r`) | +| `--raw-output0` | Like `-r` with a NUL after each output; a string containing NUL is an error | +| `--seq` | Write RS (0x1e) before each output; RS separates input values | +| `--stream` | Feed each input as path events (`[path, leaf]`, `[path]`), as `tostream` does | | `--arg name value` | Bind `$name` to a string | | `--argjson name json` | Bind `$name` to a parsed JSON value | | `-f FILE`, `--from-file FILE` | Read the filter from FILE (VFS); all positionals become input files | @@ -98,11 +102,30 @@ The limit counts real in-memory size, which is several times the JSON text: a 5.6 MB array of 100,000 small objects fits the default, much larger inputs need a larger `max_live_intermediate_bytes`. +## Location + +`$__loc__` is `{"file":"","line":N}`, N being the filter line it +appears on. + +## Destructuring alternatives + +`. as [$a, $b] ?// {a: $a} | ...` tries each pattern in turn, as in jq: +every variable named in any pattern is bound (null when its pattern did not +set it), and an error moves on to the next pattern. + +## Messages and halt + +`stderr`, `debug` and `halt_error` write to the jq command's stderr. +`halt` and `halt_error` end the jq command (not the shell or host) with +their exit code, skipping any remaining input. + ## Known gaps Bashkit's jq is intentionally minimal in places where the host model differs from upstream jq: +- `--seq` reads RS as whitespace; jq's rules for abandoned text between + separators are not reproduced. - Exotic numeric formatting modes (`@base32`, `@base64d`, etc.) follow whatever `jaq-json` ships. diff --git a/crates/bashkit/src/builtins/jq/altpat.rs b/crates/bashkit/src/builtins/jq/altpat.rs new file mode 100644 index 000000000..15a99e4fe --- /dev/null +++ b/crates/bashkit/src/builtins/jq/altpat.rs @@ -0,0 +1,250 @@ +//! jq's destructuring alternatives: `T as P1 ?// P2 | BODY`. +//! +//! Important decisions: +//! - jaq does not parse `?//`, so the filter text is rewritten before it is +//! compiled, the same way as `$__loc__` (see `loc.rs`): +//! +//! ```text +//! T as $alt | . as $dot | null as $v1 | ... | +//! try ($alt as P1 | BODY) +//! catch ($dot | $alt as P2 | BODY) +//! ``` +//! +//! That is jq's rule: every variable of every pattern is bound (null when +//! its pattern did not bind it), an error in a pattern or in BODY moves on +//! to the next pattern, and the last pattern's error propagates. +//! - BODY is copied once per pattern. The rewrite runs at most +//! `MAX_REWRITES` times so nested alternatives cannot blow up the +//! filter; anything left over fails to compile as before. +//! - Strings and comments are masked before searching, so `?//` inside +//! them is left alone. Forms the scanner cannot place (`reduce`/ +//! `foreach` with alternatives) are left unchanged. + +use std::borrow::Cow; + +const MAX_REWRITES: usize = 16; + +/// Copy of `src` with string literals and comments blanked out (same +/// length; newlines kept), so structural searches only see code. +fn mask(src: &str) -> Vec { + let b = src.as_bytes(); + let mut m = b.to_vec(); + let mut i = 0; + let mut in_str = false; + while i < b.len() { + let c = b[i]; + if in_str { + if c == b'\\' && i + 1 < b.len() { + m[i] = b' '; + m[i + 1] = b' '; + i += 2; + continue; + } + if c == b'"' { + in_str = false; + } else if c != b'\n' { + m[i] = b' '; + } + } else if c == b'"' { + in_str = true; + } else if c == b'#' { + while i < b.len() && b[i] != b'\n' { + m[i] = b' '; + i += 1; + } + continue; + } + i += 1; + } + m +} + +fn is_ident(b: u8) -> bool { + b.is_ascii_alphanumeric() || b == b'_' +} + +fn word_at(m: &[u8], i: usize, w: &str) -> bool { + let w = w.as_bytes(); + m[i..].starts_with(w) + && (i == 0 || !is_ident(m[i - 1]) && m[i - 1] != b'$') + && m.get(i + w.len()).is_none_or(|&c| !is_ident(c)) +} + +/// One rewrite of the first `?//`, or None when there is none to rewrite. +fn rewrite_once(src: &str, n: usize) -> Option { + let m = mask(src); + let alt = (0..m.len().saturating_sub(2)).find(|&i| m[i..].starts_with(b"?//"))?; + + // Back to the `as` that starts the patterns, at the same depth. + let mut depth = 0i32; + let mut as_pos = None; + let mut i = alt; + while i > 0 { + i -= 1; + match m[i] { + b')' | b']' | b'}' => depth += 1, + b'(' | b'[' | b'{' => { + depth -= 1; + if depth < 0 { + return None; + } + } + b'a' if depth == 0 && word_at(&m, i, "as") => { + as_pos = Some(i); + break; + } + _ => {} + } + } + let as_pos = as_pos?; + + // Patterns, split on `?//`, up to the `|` that starts the body. + let mut patterns = Vec::new(); + let mut start = as_pos + 2; + let mut depth = 0i32; + let mut i = start; + let pipe = loop { + let c = *m.get(i)?; + match c { + b'(' if depth == 0 => return None, + b'(' | b'[' | b'{' => depth += 1, + b')' | b']' | b'}' => { + depth -= 1; + if depth < 0 { + return None; + } + } + b'?' if depth == 0 && m[i..].starts_with(b"?//") => { + patterns.push(src[start..i].trim()); + start = i + 3; + i += 3; + continue; + } + b'|' if depth == 0 && m.get(i + 1) != Some(&b'=') => { + patterns.push(src[start..i].trim()); + break i; + } + _ => {} + } + i += 1; + }; + if patterns.len() < 2 || patterns.iter().any(|p| p.is_empty()) { + return None; + } + + // The body runs to an unmatched closer, or `;`/`then`/`elif`/`else`/ + // `end`/`catch` outside any bracket or `if`. + let body_start = pipe + 1; + let mut depth = 0i32; + let mut ifs = 0i32; + let mut i = body_start; + let body_end = loop { + let Some(&c) = m.get(i) else { break m.len() }; + match c { + b'(' | b'[' | b'{' => depth += 1, + b')' | b']' | b'}' => { + if depth == 0 { + break i; + } + depth -= 1; + } + b';' if depth == 0 && ifs == 0 => break i, + b'i' if word_at(&m, i, "if") => ifs += 1, + b'e' if word_at(&m, i, "end") => { + if depth == 0 && ifs == 0 { + break i; + } + ifs -= 1; + } + _ if depth == 0 + && ifs == 0 + && ["then", "elif", "else", "catch"] + .iter() + .any(|w| word_at(&m, i, w)) => + { + break i; + } + _ => {} + } + i += 1; + }; + let body = &src[body_start..body_end]; + + // Every `$name` in any pattern is bound, null by default. + let mut vars: Vec<&str> = Vec::new(); + for p in &patterns { + let pb = p.as_bytes(); + let pm = mask(p); + let mut j = 0; + while j < pb.len() { + if pm[j] == b'$' { + let e = (j + 1..pb.len()) + .find(|&k| !is_ident(pb[k])) + .unwrap_or(pb.len()); + let v = &p[j..e]; + if v.len() > 1 && !vars.contains(&v) { + vars.push(v); + } + j = e; + } else { + j += 1; + } + } + } + + let alt_var = format!("$__bk_alt{n}__"); + let dot_var = format!("$__bk_dot{n}__"); + let mut chain = format!("{alt_var} as {} |{body}", patterns[patterns.len() - 1]); + for p in patterns[..patterns.len() - 1].iter().rev() { + chain = format!("try ({alt_var} as {p} |{body}) catch ({dot_var} | {chain})"); + } + let nulls: String = vars.iter().map(|v| format!("null as {v} | ")).collect(); + Some(format!( + "{}as {alt_var} | . as {dot_var} | {nulls}({chain}){}", + &src[..as_pos], + &src[body_end..] + )) +} + +/// Rewrite every `as P1 ?// P2 | BODY` in the filter (see module docs). +pub(super) fn expand_alternatives(src: &str) -> Cow<'_, str> { + if !src.contains("?//") { + return Cow::Borrowed(src); + } + let mut cur = src.to_string(); + for n in 0..MAX_REWRITES { + match rewrite_once(&cur, n) { + Some(next) => cur = next, + None => break, + } + } + Cow::Owned(cur) +} + +#[cfg(test)] +mod tests { + use super::expand_alternatives; + + #[test] + fn leaves_filters_without_alternatives() { + assert_eq!(expand_alternatives(". as [$a] | $a"), ". as [$a] | $a"); + assert_eq!(expand_alternatives("\"?//\""), "\"?//\""); + } + + #[test] + fn rewrites_two_patterns() { + let out = expand_alternatives(".[] as [$x, $y] ?// $x | [$x, $y]"); + assert_eq!( + out, + ".[] as $__bk_alt0__ | . as $__bk_dot0__ | null as $x | null as $y | \ + (try ($__bk_alt0__ as [$x, $y] | [$x, $y]) catch ($__bk_dot0__ | \ + $__bk_alt0__ as $x | [$x, $y]))" + ); + } + + #[test] + fn body_stops_at_closing_paren() { + let out = expand_alternatives("(. as [$a] ?// $a | $a), 2"); + assert!(out.ends_with("| $a))), 2"), "{out}"); + } +} diff --git a/crates/bashkit/src/builtins/jq/args.rs b/crates/bashkit/src/builtins/jq/args.rs index 59fef157b..5b0d7b93e 100644 --- a/crates/bashkit/src/builtins/jq/args.rs +++ b/crates/bashkit/src/builtins/jq/args.rs @@ -45,6 +45,14 @@ pub(super) struct JqArgs<'a> { pub null_input: bool, pub slurp: bool, pub sort_keys: bool, + /// `-a`: escape non-ASCII characters as `\uXXXX`. + pub ascii_output: bool, + /// `--raw-output0`: like `-r`, but a NUL after each output. + pub raw_output0: bool, + /// `--seq`: RS (0x1e) before each output; RS separates inputs. + pub seq: bool, + /// `--stream`: feed each input as `[path, leaf]` / `[path]` events. + pub stream: bool, /// `-e` flag — set exit status from output. pub exit_status: bool, pub indent: Indent, @@ -99,6 +107,10 @@ const HELP_TEXT: &str = "Usage: jq [OPTIONS...] FILTER [FILE...]\n\n\ \t-e, --exit-status\tset exit status code based on output\n\ \t-S, --sort-keys\t\tsort object keys in output\n\ \t-j, --join-output\tlike -r without trailing newline\n\ + \t-a, --ascii-output\toutput strings by only ASCII characters using escape sequences\n\ + \t--raw-output0\t\timplies -r and output NUL after each output\n\ + \t--seq\t\t\tparse input/output as application/json-seq\n\ + \t--stream\t\tparse the input value in streaming fashion\n\ \t--tab\t\t\tuse tabs for indentation\n\ \t--indent N\t\tuse N spaces for indentation (0..7, default 2)\n\ \t--arg name value\tset variable $name to string value\n\ @@ -138,6 +150,10 @@ pub(super) fn parse<'a>(args: &'a [String]) -> ParseOutcome<'a> { compact_output: false, null_input: false, slurp: false, + ascii_output: false, + raw_output0: false, + seq: false, + stream: false, sort_keys: false, exit_status: false, indent: Indent::Spaces(2), @@ -203,15 +219,13 @@ pub(super) fn parse<'a>(args: &'a [String]) -> ParseOutcome<'a> { "--exit-status" => out.exit_status = true, "--tab" => out.indent = Indent::Tab, "--join-output" => out.join_output = true, - "--ascii-output" - | "-a" - | "-C" - | "-M" - | "--color-output" - | "--monochrome-output" - | "--unbuffered" => { + "--ascii-output" | "-a" => out.ascii_output = true, + "--raw-output0" => out.raw_output0 = true, + "--seq" => out.seq = true, + "--stream" => out.stream = true, + "-C" | "--color-output" | "--monochrome-output" | "-M" | "--unbuffered" => { // Recognised but no-op: rendering is not TTY-aware in this - // sandbox, and ASCII-only output isn't implemented yet. + // sandbox. } "--indent" => match args.get(i + 1) { Some(n) => match n.parse::() { @@ -330,7 +344,8 @@ pub(super) fn parse<'a>(args: &'a [String]) -> ParseOutcome<'a> { 's' => out.slurp = true, 'e' => out.exit_status = true, 'j' => out.join_output = true, - 'a' | 'C' | 'M' => {} // ASCII / color / monochrome — accept silently + 'a' => out.ascii_output = true, + 'C' | 'M' => {} // color / monochrome — accept silently unknown => { return ParseOutcome::Done(unknown_option(&format!("-{unknown}"))); } @@ -666,10 +681,13 @@ mod tests { } #[test] - fn ascii_output_silently_accepted() { - match parse_strs(&["-a", "."]) { - ParseOutcome::Args(a) => assert_eq!(a.filter, "."), - _ => panic!("expected Args (-a should be accepted)"), + fn output_mode_flags_parse() { + match parse_strs(&["-a", "--raw-output0", "--seq", "--stream", "."]) { + ParseOutcome::Args(a) => { + assert_eq!(a.filter, "."); + assert!(a.ascii_output && a.raw_output0 && a.seq && a.stream); + } + _ => panic!("expected Args"), } } diff --git a/crates/bashkit/src/builtins/jq/compat.rs b/crates/bashkit/src/builtins/jq/compat.rs index 6afc33c4e..a4632d879 100644 --- a/crates/bashkit/src/builtins/jq/compat.rs +++ b/crates/bashkit/src/builtins/jq/compat.rs @@ -15,6 +15,12 @@ //! `truncate_stream`: jq 1.7's definitions (capture groups in `scan`, //! null for an unmatched group, null joins as "", non-strings pass //! through the trim filters). +//! - `halt_error`: jq prints a string as is, null as nothing, and other +//! values as JSON plus a newline, then halts (the run loop turns the +//! halt into the command's exit code, never a process exit). +//! - `implode` / `strptime` / `fromdate` / `@base64d`: jq's handling of +//! invalid code points (U+FFFD), its strptime error text, and unpadded +//! base64 input. //! - `@tsv` / `@csv`: jaq-std doesn't define them. Strict variants reject //! non-scalars with a runtime error matching real jq's wording. //! - `input_filename` / `input_line_number`: bashkit threads these as @@ -111,6 +117,36 @@ def fromstream(f): def truncate_stream(stream): . as $n | null | stream | if (.[0] | length) > $n then setpath([0]; .[0][$n:]) else empty end; +def _bk_implode: implode; +def implode: + map(if type == "number" and (. > 1114111 or (. >= 55296 and . <= 57343)) + then 65533 else . end) + | _bk_implode; +def _bk_strptime($f): strptime($f); +def strptime($f): + if type != "string" or ($f | type) != "string" then + error("strptime/1 requires string inputs and arguments") + else + . as $s + | try _bk_strptime($f) + catch error("date \($s | tojson) does not match format \($f | tojson)") + end; +def fromdateiso8601: strptime("%Y-%m-%dT%H:%M:%SZ") | mktime; +def fromdate: fromdateiso8601; +def @base64d: + tostring | . as $s + | (sub("=+$"; "")) as $t + | ($t | length % 4) as $r + | if $r == 1 then error("string (\($s | tojson)) trailing base64 byte found") + else + ($t + (if $r == 2 then "==" elif $r == 3 then "=" else "" end)) + | try decode_base64 + catch error("string (\($s | tojson)) is not valid base64 data") + end; +def halt_error($code): + (if type == "string" then . elif . == null then empty else tojson + "\n" end + | stderr_empty), halt($code); +def halt_error: halt_error(5); def @tsv: [.[] | if type == "string" then diff --git a/crates/bashkit/src/builtins/jq/convert.rs b/crates/bashkit/src/builtins/jq/convert.rs index 2a5bf221b..59bedbde3 100644 --- a/crates/bashkit/src/builtins/jq/convert.rs +++ b/crates/bashkit/src/builtins/jq/convert.rs @@ -326,6 +326,44 @@ pub(super) fn parse_json_stream_partial(input: &str) -> (Vec, Option Vec { + fn walk(v: &JqJson, path: &mut Vec, out: &mut Vec) { + let children: Vec<(JqJson, &JqJson)> = match v { + JqJson::Array(a) => a + .iter() + .enumerate() + .map(|(i, x)| (JqJson::Number(i.to_string()), x)) + .collect(), + JqJson::Object(o) => o + .iter() + .map(|(k, x)| (JqJson::String(k.clone()), x)) + .collect(), + _ => Vec::new(), + }; + if children.is_empty() { + out.push(JqJson::Array(vec![JqJson::Array(path.clone()), v.clone()])); + return; + } + let mut last = None; + for (k, x) in children { + path.push(k); + walk(x, path, out); + last = path.pop(); + } + if let Some(k) = last { + let mut closing = path.clone(); + closing.push(k); + out.push(JqJson::Array(vec![JqJson::Array(closing)])); + } + } + let mut out = Vec::new(); + walk(v, &mut Vec::new(), &mut out); + out +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/bashkit/src/builtins/jq/loc.rs b/crates/bashkit/src/builtins/jq/loc.rs new file mode 100644 index 000000000..f4b383d5a --- /dev/null +++ b/crates/bashkit/src/builtins/jq/loc.rs @@ -0,0 +1,140 @@ +//! jq's `$__loc__`: `{"file":"","line":N}` for the line where it +//! appears in the filter. +//! +//! Important decisions: +//! - jaq has no `$__loc__`, so the filter text is rewritten before it is +//! compiled: each `$__loc__` in code becomes the object literal. A small +//! scanner skips string literals (but not their `\(...)` interpolations) +//! and comments, and counts lines. +//! - `{$__loc__}` is jq's object shorthand for `{"__loc__": $__loc__}`; it +//! is written out as that pair. + +use std::borrow::Cow; + +const TOKEN: &str = "$__loc__"; + +fn is_ident(b: u8) -> bool { + b.is_ascii_alphanumeric() || b == b'_' +} + +/// Replace every `$__loc__` in filter code with its location object. +pub(super) fn expand_loc(src: &str) -> Cow<'_, str> { + if !src.contains(TOKEN) { + return Cow::Borrowed(src); + } + let b = src.as_bytes(); + let mut out: Vec = Vec::with_capacity(src.len() + 64); + // Open brackets: `{`, `[`, `(`, `"` (string), `I` (interpolation). + let mut stack: Vec = Vec::new(); + let mut line = 1usize; + let mut last_sig = 0u8; + let mut i = 0; + while i < b.len() { + let c = b[i]; + if stack.last() == Some(&b'"') { + match c { + b'\\' if b.get(i + 1) == Some(&b'(') => { + stack.push(b'I'); + out.extend_from_slice(b"\\("); + last_sig = b'('; + i += 2; + continue; + } + b'\\' if i + 1 < b.len() => { + out.extend_from_slice(&b[i..i + 2]); + i += 2; + continue; + } + b'"' => { + stack.pop(); + last_sig = b'"'; + } + b'\n' => line += 1, + _ => {} + } + out.push(c); + i += 1; + continue; + } + match c { + b'#' => { + let end = b[i..] + .iter() + .position(|&x| x == b'\n') + .map_or(b.len(), |p| i + p); + out.extend_from_slice(&b[i..end]); + i = end; + continue; + } + b'"' | b'{' | b'[' | b'(' => stack.push(c), + b'}' | b']' | b')' => { + stack.pop(); + } + b'\n' => line += 1, + b'$' if src[i..].starts_with(TOKEN) + && !b.get(i + TOKEN.len()).is_some_and(|&x| is_ident(x)) => + { + let obj = format!("{{\"file\":\"\",\"line\":{line}}}"); + let next_sig = b[i + TOKEN.len()..] + .iter() + .copied() + .find(|x| !x.is_ascii_whitespace()) + .unwrap_or(0); + let shorthand = stack.last() == Some(&b'{') + && matches!(last_sig, b'{' | b',') + && matches!(next_sig, b'}' | b','); + if shorthand { + out.extend_from_slice(format!("\"__loc__\": {obj}").as_bytes()); + } else { + out.extend_from_slice(format!("({obj})").as_bytes()); + } + last_sig = b')'; + i += TOKEN.len(); + continue; + } + _ => {} + } + out.push(c); + if !c.is_ascii_whitespace() { + last_sig = c; + } + i += 1; + } + // Only ASCII was inserted and input bytes were copied whole. + Cow::Owned(String::from_utf8(out).unwrap_or_else(|_| src.to_string())) +} + +#[cfg(test)] +mod tests { + use super::expand_loc; + + #[test] + fn replaces_code_occurrences_with_line() { + assert_eq!( + expand_loc("1\n| $__loc__.line"), + "1\n| ({\"file\":\"\",\"line\":2}).line" + ); + } + + #[test] + fn leaves_strings_comments_and_longer_names() { + let src = "\"$__loc__\", $__loc__x # $__loc__"; + assert_eq!(expand_loc(src), src); + } + + #[test] + fn expands_inside_interpolation() { + assert_eq!( + expand_loc("\"\\($__loc__)\""), + "\"\\(({\"file\":\"\",\"line\":1}))\"" + ); + } + + #[test] + fn object_shorthand_becomes_pair() { + assert_eq!( + expand_loc("{a: 1, $__loc__}"), + "{a: 1, \"__loc__\": {\"file\":\"\",\"line\":1}}" + ); + } +} diff --git a/crates/bashkit/src/builtins/jq/messages.rs b/crates/bashkit/src/builtins/jq/messages.rs new file mode 100644 index 000000000..203e53a97 --- /dev/null +++ b/crates/bashkit/src/builtins/jq/messages.rs @@ -0,0 +1,68 @@ +//! jq's `stderr`, `debug` and `halt_error` messages. +//! +//! Important decisions: +//! - jaq-std's `stderr_empty` / `debug_empty` natives write to the host +//! process (`log`). bashkit replaces them so the text lands in the jq +//! command's own stderr, in order with its error messages, and never +//! reaches the host. +//! - Natives are plain fn pointers, so the text goes through a +//! thread-local buffer. The filter runs synchronously on one thread; +//! `take` drains it after each result. +//! - The buffer is capped (`MAX_MESSAGE_BYTES`) so a looping `debug` cannot +//! grow memory without bound; past the cap text is dropped. + +use super::convert::val_to_jq_capped; +use super::format::{Indent, render}; +use super::jaq_json::Val; +use std::cell::RefCell; + +/// Cap on buffered stderr text per jq run. +pub(super) const MAX_MESSAGE_BYTES: usize = 1 << 20; + +thread_local! { + static BUF: RefCell = const { RefCell::new(String::new()) }; +} + +fn push(s: &str) { + BUF.with(|b| { + let mut b = b.borrow_mut(); + let room = MAX_MESSAGE_BYTES.saturating_sub(b.len()); + if s.len() <= room { + b.push_str(s); + } else { + let mut end = room; + while !s.is_char_boundary(end) { + end -= 1; + } + b.push_str(&s[..end]); + } + }); +} + +fn compact(v: &Val) -> String { + match val_to_jq_capped(v, MAX_MESSAGE_BYTES) { + Some(j) => render(&j, Indent::Compact), + None => "".to_string(), + } +} + +/// `stderr`: a string as is, anything else as compact JSON, no newline. +pub(super) fn stderr(v: &Val) { + match v { + Val::TStr(s) | Val::BStr(s) => { + let bytes: &[u8] = (**s).as_ref(); + push(&String::from_utf8_lossy(bytes)); + } + _ => push(&compact(v)), + } +} + +/// `debug`: `["DEBUG:",]` and a newline. +pub(super) fn debug(v: &Val) { + push(&format!("[\"DEBUG:\",{}]\n", compact(v))); +} + +/// Drain the buffered text. +pub(super) fn take() -> String { + BUF.with(|b| std::mem::take(&mut *b.borrow_mut())) +} diff --git a/crates/bashkit/src/builtins/jq/mod.rs b/crates/bashkit/src/builtins/jq/mod.rs index 2e8caab50..18243e99d 100644 --- a/crates/bashkit/src/builtins/jq/mod.rs +++ b/crates/bashkit/src/builtins/jq/mod.rs @@ -29,12 +29,15 @@ use crate::error::Result; use crate::interpreter::ExecResult; use crate::limits::ExecutionLimits; +mod altpat; mod args; mod compat; mod convert; mod errors; mod format; mod input; +mod loc; +mod messages; // Vendored jaq-json (MIT, Michael Färber, https://github.com/01mf02/jaq), // see jaq_json/UPSTREAM_VERSION and knowledge/runtimes/jaq-json-vendor.md. // Kept byte-close to upstream so `scripts/sync-jaq-json.sh` can merge new @@ -64,7 +67,7 @@ use compat::{ ARGS_VAR_NAME, ENV_VAR_NAME, FILENAME_VAR_NAME, LINENO_VAR_NAME, PUBLIC_ENV_VAR_NAME, build_compat_prefix, }; -use convert::{JqJson, jq_to_val, parse_json_stream, val_to_jq_capped}; +use convert::{JqJson, jq_to_val, parse_json_stream, stream_events, val_to_jq_capped}; use errors::{format_compile_errors, format_load_errors, format_runtime_error_at}; use format::{Indent, render, sort_keys as sort_jq_keys}; @@ -261,6 +264,8 @@ async fn run_jq(ctx: Context<'_>, parsed: JqArgs<'_>) -> Result { } None => std::borrow::Cow::Borrowed(parsed.filter), }; + let filter_text = loc::expand_loc(&filter_text); + let filter_text = altpat::expand_alternatives(&filter_text); let compat_filter = format!("{prefix}\n{filter_text}"); let filter_src = compat_filter.as_str(); @@ -321,30 +326,62 @@ async fn run_jq(ctx: Context<'_>, parsed: JqArgs<'_>) -> Result { .into_iter() .map(|(name, arity, run)| (name, arity, jaq_core::Native::::new(run))) .collect(); - // SECURITY (TM-INF-023, #1571): replace the upstream `halt` native. - // jaq-std's impl calls `std::process::exit(...)`, which would tear - // down the entire embedding process — a sandbox escape via DoS for - // any caller hosting bashkit. Strip it from the `funs` chain and add - // a safe stub so the wrapper defs in jaq-std's `defs.jq` - // (`def halt: halt(0);`, `def halt_error(...): ..., halt(...);`) - // still resolve, but produce a normal jq runtime error instead of - // killing the host. - let safe_halt_run: jaq_core::RunPtr = |mut cv| { - let _ = cv.0.pop_var(); - jaq_core::native::bome(Err(jaq_core::Error::str( - "halt is disabled in the bashkit sandbox", - ))) - }; - let safe_halt: jaq_core::native::Fun = ( - "halt", - jaq_core::native::v(1), - jaq_core::Native::::new(safe_halt_run), - ); + // SECURITY (TM-INF-023, #1571): jaq-std's `halt` native only raises + // `Exn::halt`; it is `jaq_core::unwrap_valr` that turns that into + // `std::process::exit`. The run loop below never calls `unwrap_valr`: + // it ends the jq command with the halt code instead, so `halt` and + // `halt_error` behave like jq without touching the host process. + // + // jaq-std's `stderr_empty`/`debug_empty` write to the host (`log`); + // ours buffer the text for this command's stderr (see `messages`). + let message_funs: Vec> = vec![ + ( + "stderr_empty", + jaq_core::native::v(0), + jaq_core::Native::::new(|cv| { + messages::stderr(&cv.1); + Box::new(std::iter::empty()) + }), + ), + // jq accepts base64 with non-zero trailing bits ("YW" is "a"). + ( + "decode_base64", + jaq_core::native::v(0), + jaq_core::Native::::new(|cv| { + use base64::Engine; + use base64::engine::{DecodePaddingMode, GeneralPurpose, GeneralPurposeConfig}; + use jaq_std::ValT; + const LENIENT: GeneralPurpose = GeneralPurpose::new( + &base64::alphabet::STANDARD, + GeneralPurposeConfig::new() + .with_decode_allow_trailing_bits(true) + .with_decode_padding_mode(DecodePaddingMode::Indifferent), + ); + jaq_core::native::bome(cv.1.try_as_utf8_bytes().and_then(|s| { + LENIENT + .decode(s) + .map_err(|e| jaq_core::Error::str(e.to_string())) + .map(Val::from_utf8_bytes) + })) + }), + ), + ( + "debug_empty", + jaq_core::native::v(0), + jaq_core::Native::::new(|cv| { + messages::debug(&cv.1); + Box::new(std::iter::empty()) + }), + ), + ]; let native_funs = jaq_core::funs::() .chain(jaq_std::funs::().filter(|(name, _, _)| { - *name != "env" && *name != "halt" && !regex_compat::SHADOWED_NATIVE_NAMES.contains(name) + !matches!( + *name, + "env" | "stderr_empty" | "debug_empty" | "decode_base64" + ) && !regex_compat::SHADOWED_NATIVE_NAMES.contains(name) })) - .chain(std::iter::once(safe_halt)) + .chain(message_funs) .chain(input_funs) .chain(regex_funs) .chain(self::jaq_json::funs::()); @@ -396,16 +433,37 @@ async fn run_jq(ctx: Context<'_>, parsed: JqArgs<'_>) -> Result { } else { let mut slurped: Vec = Vec::new(); for (name, text) in &sources { + // `--seq`: RS (0x1e) separates values; read it as whitespace. + let seq_text; + let text = if parsed.seq { + seq_text = text.replace('\x1e', " "); + &seq_text + } else { + text + }; let (vals, err) = parse_jq_json_stream_partial(&ctx, text)?; + let lines = input::value_lines(text); + // `--stream`: each value becomes its path events. + let vals: Vec<(JqJson, usize)> = vals + .into_iter() + .enumerate() + .flat_map(|(i, v)| { + let line = lines.get(i).copied().unwrap_or(0); + if parsed.stream { + stream_events(&v).into_iter().map(|e| (e, line)).collect() + } else { + vec![(v, line)] + } + }) + .collect(); if parsed.slurp { - slurped.extend(vals); + slurped.extend(vals.into_iter().map(|(v, _)| v)); } else { - let lines = input::value_lines(text); - for (i, v) in vals.iter().enumerate() { + for (v, line) in &vals { items.push(FilterInput { value: jq_to_val(v), filename: name_val(name), - lineno: lines.get(i).copied().unwrap_or(0), + lineno: *line, }); } } @@ -509,12 +567,15 @@ async fn run_jq(ctx: Context<'_>, parsed: JqArgs<'_>) -> Result { // runtime error, else 0; with `-e`, 1 when the last output was null or // false and 4 when nothing was ever output. let mut status: Option = None; + // `halt`/`halt_error` end the whole command with this code. + let mut halted: Option = None; + messages::take(); // THREAT[TM-DOS-110]: the meter aborts a run by unwinding where jaq // gives it no error channel (see jaq_json::meter::Abort). let run_filter = std::panic::AssertUnwindSafe(|| -> Result> { let mut first = true; - loop { + 'inputs: loop { let jaq_input: Val = if null_input { if !first { break; @@ -553,7 +614,19 @@ async fn run_jq(ctx: Context<'_>, parsed: JqArgs<'_>) -> Result { // The value may have been cut short at the limit. return size_error(&meter).map(Some); } - match jaq_core::unwrap_valr(result) { + stderr_out.push_str(&messages::take()); + // Never `jaq_core::unwrap_valr`: it exits the process on halt. + let result = match result { + Ok(val) => Ok(val), + Err(exn) => match exn.get_err() { + Ok(e) => Err(e), + Err(exn) => { + halted = Some(exn.get_halt().unwrap_or(5)); + break 'inputs; + } + }, + }; + match result { Ok(val) => { let Some(mut jq) = val_to_jq_capped(&val, max_output_bytes) else { return Ok(Some(ExecResult::err( @@ -568,19 +641,38 @@ async fn run_jq(ctx: Context<'_>, parsed: JqArgs<'_>) -> Result { input_status = Some(i32::from(jq.is_null() || jq.is_false())); } - let effective_raw = parsed.raw_output || parsed.join_output; - let formatted = if effective_raw { - if let JqJson::String(s) = &jq { + // With -a, jq JSON-encodes strings even under -r. + let effective_raw = + (parsed.raw_output || parsed.join_output || parsed.raw_output0) + && !parsed.ascii_output; + let formatted = match &jq { + JqJson::String(s) if effective_raw => { + if parsed.raw_output0 && s.contains('\0') { + let loc = location(¤t.borrow()); + stderr_out.push_str(&format!( + "jq: error (at {loc}): Cannot dump a string \ + containing NUL with --raw-output0 option\n" + )); + input_status = Some(5); + break; + } s.clone() - } else { - render(&jq, indent) } + _ => render(&jq, indent), + }; + let formatted = if parsed.ascii_output { + ascii_escape(&formatted) } else { - render(&jq, indent) + formatted }; + if parsed.seq { + output.push('\x1e'); + } output.push_str(&formatted); - if !parsed.join_output { + if parsed.raw_output0 { + output.push('\0'); + } else if !parsed.join_output { output.push('\n'); } @@ -640,8 +732,9 @@ async fn run_jq(ctx: Context<'_>, parsed: JqArgs<'_>) -> Result { }, } + stderr_out.push_str(&messages::take()); let mut code = status.unwrap_or(if parsed.exit_status { 4 } else { 0 }); - if let Some(e) = parse_error { + if let Some(e) = parse_error.filter(|_| halted.is_none()) { stderr_out.push_str(&e); stderr_out.push('\n'); code = 5; @@ -649,6 +742,9 @@ async fn run_jq(ctx: Context<'_>, parsed: JqArgs<'_>) -> Result { if input_failed { code = 2; } + if let Some(h) = halted { + code = h; + } Ok(ExecResult { stdout: output.into(), stderr: stderr_out.into(), @@ -733,3 +829,24 @@ fn build_args_obj(positional: &[JqJson], named: &[(String, JqJson)]) -> JqJson { ("named".to_string(), JqJson::Object(named_map)), ]) } + +/// `-a`: escape every non-ASCII character as `\uXXXX` (UTF-16 surrogate +/// pairs above the BMP). Rendered JSON has non-ASCII text only inside +/// strings, so escaping the whole rendering is safe. +fn ascii_escape(s: &str) -> String { + if s.is_ascii() { + return s.to_string(); + } + let mut out = String::with_capacity(s.len() + 16); + for c in s.chars() { + if c.is_ascii() { + out.push(c); + } else { + let mut buf = [0u16; 2]; + for unit in c.encode_utf16(&mut buf) { + out.push_str(&format!("\\u{unit:04x}")); + } + } + } + out +} diff --git a/crates/bashkit/src/builtins/jq/tests.rs b/crates/bashkit/src/builtins/jq/tests.rs index 7c5d78ddf..cff7dbf39 100644 --- a/crates/bashkit/src/builtins/jq/tests.rs +++ b/crates/bashkit/src/builtins/jq/tests.rs @@ -1430,46 +1430,61 @@ async fn double_dash_separator() { // Security: jq filters must not terminate the host process (#1571) // ========================================================================= +// The tests reaching their assertions at all proves the host wasn't killed: +// `halt` ends the jq command with its code, never the process. #[tokio::test] -async fn halt_does_not_terminate_host_process() { - // Regression for #1571: the upstream `halt` native calls - // `std::process::exit(...)`, which would tear down the embedding - // process. We strip it from the funs chain; calling `halt` therefore - // surfaces as an ordinary jq compile/runtime error rather than process - // termination. - // - // The test reaching the assertion at all proves the host wasn't killed. - let result = run_jq_result("halt", r#"{}"#).await.unwrap(); - assert_ne!( - result.exit_code, 0, - "halt should fail safely; stdout=<{}> stderr=<{}>", - result.stdout, result.stderr - ); +async fn halt_ends_the_command_not_the_host() { + let result = run_jq_result_with_args(&["-n", "1, halt, 2"], "") + .await + .unwrap(); + assert_eq!(result.stdout, "1\n"); + assert_eq!(result.exit_code, 0); + let result = run_jq_result_with_args(&["-n", "halt(7)"], "") + .await + .unwrap(); + assert_eq!(result.exit_code, 7); } #[tokio::test] -async fn halt_with_arg_does_not_terminate_host_process() { - // Regression for #1571: the explicit-arity form `halt(N)` must also - // be neutered, not just the wrapper-def `halt`. - let result = run_jq_result("halt(7)", r#"{}"#).await.unwrap(); - assert_ne!( - result.exit_code, 0, - "halt(7) should fail safely; stdout=<{}> stderr=<{}>", - result.stdout, result.stderr - ); +async fn halt_stops_remaining_inputs() { + let result = run_jq_result_with_args(&["-c", "if . == 2 then halt else . end"], "1 2 3") + .await + .unwrap(); + assert_eq!(result.stdout, "1\n"); + assert_eq!(result.exit_code, 0); } #[tokio::test] -async fn halt_error_does_not_terminate_host_process() { - // Regression for #1571: `halt_error` is a jq-syntax def in jaq-std - // that ultimately calls `halt(...)`. Stripping the native makes the - // whole family fail closed. - let result = run_jq_result("halt_error", r#""boom""#).await.unwrap(); - assert_ne!( - result.exit_code, 0, - "halt_error should fail safely; stdout=<{}> stderr=<{}>", - result.stdout, result.stderr - ); +async fn halt_error_prints_message_and_exits() { + let result = run_jq_result("halt_error", r#""boom\n""#).await.unwrap(); + assert_eq!(result.stderr, "boom\n"); + assert_eq!(result.exit_code, 5); + let result = run_jq_result("halt_error(3)", r#"{"a":1}"#).await.unwrap(); + assert_eq!(result.stderr, "{\"a\":1}\n"); + assert_eq!(result.exit_code, 3); + let result = run_jq_result("try halt_error catch 9", "null") + .await + .unwrap(); + assert_eq!(result.stdout, ""); + assert_eq!(result.exit_code, 5); +} + +#[tokio::test] +async fn debug_and_stderr_write_to_jq_stderr() { + let result = run_jq_result_with_args(&["-c", "debug, debug(\"m\"), stderr"], "[1]") + .await + .unwrap(); + assert_eq!(result.stdout, "[1]\n[1]\n[1]\n"); + assert_eq!(result.stderr, "[\"DEBUG:\",[1]]\n[\"DEBUG:\",\"m\"]\n[1]"); +} + +#[tokio::test] +async fn debug_output_is_capped() { + let result = run_jq_result_with_args(&["-n", "[range(200000)] | debug | length"], "") + .await + .unwrap(); + assert_eq!(result.stdout, "200000\n"); + assert!(result.stderr.len() <= super::messages::MAX_MESSAGE_BYTES); } // ========================================================================= @@ -2141,3 +2156,121 @@ async fn tostream_round_trips() { "[[[0],1],[[1,\"a\"],2],[[1,\"a\"]],[[2],[]],[[2]]]\n[1,{\"a\":2},[]]\n[[[0],2],[[0]]]\ntrue\n" ); } + +#[test] +fn run_loop_never_calls_unwrap_valr() { + // THREAT[TM-INF-023]: `jaq_core::unwrap_valr` exits the process on halt. + assert!(!include_str!("mod.rs").contains("unwrap_valr(")); +} + +#[tokio::test] +async fn ascii_output_escapes_non_ascii() { + let out = run_jq_with_args(&["-ac", "."], r#"{"s":"olá ☃ 😀"}"#) + .await + .unwrap(); + assert_eq!(out, "{\"s\":\"ol\\u00e1 \\u2603 \\ud83d\\ude00\"}\n"); + // -a keeps strings JSON-encoded even under -r, like jq. + let out = run_jq_with_args(&["-ra", "."], "\"é\"").await.unwrap(); + assert_eq!(out, "\"\\u00e9\"\n"); +} + +#[tokio::test] +async fn raw_output0_separates_with_nul() { + let out = run_jq_with_args(&["--raw-output0", ".[]"], r#"[1,"x",{}]"#) + .await + .unwrap(); + assert_eq!(out, "1\0x\0{}\0"); + let result = run_jq_result_with_args(&["--raw-output0", ".[]"], r#"["a\u0000b","c"]"#) + .await + .unwrap(); + assert_eq!(result.stdout, ""); + assert_eq!( + result.stderr, + "jq: error (at :0): Cannot dump a string containing NUL with --raw-output0 option\n" + ); + assert_eq!(result.exit_code, 5); +} + +#[tokio::test] +async fn stream_flag_feeds_path_events() { + let out = run_jq_with_args(&["-c", "--stream", "."], r#"{"a":[1,{"b":null}],"c":"d"}"#) + .await + .unwrap(); + assert_eq!( + out, + "[[\"a\",0],1]\n[[\"a\",1,\"b\"],null]\n[[\"a\",1,\"b\"]]\n[[\"a\",1]]\n[[\"c\"],\"d\"]\n[[\"c\"]]\n" + ); + let out = run_jq_with_args(&["-c", "--stream", "."], "1 [] {}") + .await + .unwrap(); + assert_eq!(out, "[[],1]\n[[],[]]\n[[],{}]\n"); + let out = run_jq_with_args(&["-nc", "--stream", "fromstream(inputs)"], r#"{"a":[1,2]}"#) + .await + .unwrap(); + assert_eq!(out, "{\"a\":[1,2]}\n"); +} + +#[tokio::test] +async fn seq_flag_uses_record_separators() { + let out = run_jq_with_args(&["--seq", "-c", "."], "\x1e[1]\n\x1e2\n") + .await + .unwrap(); + assert_eq!(out, "\x1e[1]\n\x1e2\n"); +} + +#[tokio::test] +async fn destructuring_alternatives_follow_jq() { + let out = run_jq_with_args( + &["-c", ".[] as [$x, $y] ?// $x | [$x, $y]"], + r#"[1, {"b": 2}, [4, 5]]"#, + ) + .await + .unwrap(); + assert_eq!(out, "[1,null]\n[{\"b\":2},null]\n[4,5]\n"); + // An error in the body moves on to the next pattern; the last one's + // error propagates. + let result = run_jq_result_with_args( + &[ + "-c", + ".[] as [$a] ?// $a | if $a == 1 then error(\"x\") else $a end", + ], + "[[1]]", + ) + .await + .unwrap(); + assert_eq!(result.stdout, "[1]\n"); + let result = run_jq_result_with_args(&["-c", ". as [$a] ?// $a | error(\"e\")"], "[1]") + .await + .unwrap(); + assert_eq!(result.stderr, "jq: error (at :0): e\n"); + let result = run_jq_result_with_args(&["-c", ". as [$a] ?// {a: $a} | error(\"e\")"], "[1]") + .await + .unwrap(); + assert_eq!( + result.stderr, + "jq: error (at :0): Cannot index array with string \"a\"\n" + ); +} + +#[tokio::test] +async fn decoding_builtins_follow_jq() { + let out = run_jq_with_args( + &[ + "-nc", + "([55296, 65] | implode), (\"YW\" | @base64d), (\"YQ\" | @base64d), \ + (try (\"!!\" | @base64d) catch .), (try (\"Y\" | @base64d) catch .), \ + (try (\"x\" | strptime(\"%Y\")) catch .), (try (\"2026-01-15\" | fromdate) catch .), \ + (\"2026-01-15T12:00:00Z\" | fromdate)", + ], + "", + ) + .await + .unwrap(); + assert_eq!( + out, + "\"\u{fffd}A\"\n\"a\"\n\"a\"\n\"string (\\\"!!\\\") is not valid base64 data\"\n\ + \"string (\\\"Y\\\") trailing base64 byte found\"\n\ + \"date \\\"x\\\" does not match format \\\"%Y\\\"\"\n\ + \"date \\\"2026-01-15\\\" does not match format \\\"%Y-%m-%dT%H:%M:%SZ\\\"\"\n1768478400\n" + ); +} diff --git a/knowledge/log.md b/knowledge/log.md index dc86babee..ac241828c 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -2,6 +2,11 @@ ## 2026-10-07 +* **Fix**: jq `implode` maps invalid code points to U+FFFD, `strptime`/`fromdate` fail with jq's `date "X" does not match format "F"`, and `@base64d` accepts unpadded input and non-zero trailing bits with jq's error texts. +* **Feature**: jq destructuring alternatives `T as P1 ?// P2 | BODY`: every pattern variable is bound (null when unbound), an error in a pattern or the body moves to the next pattern, the last error propagates. jaq does not parse `?//`; `builtins/jq/altpat.rs` rewrites it to `try`/`catch` before compiling (body copied per pattern, at most 16 rewrites). +* **Feature**: jq `$__loc__` (`{"file":"","line":N}`), including `{$__loc__}` shorthand and string interpolation. jaq has no `$__loc__`, so the filter text is rewritten before compiling (`builtins/jq/loc.rs` skips strings and comments). +* **Feature**: jq `-a`/`--ascii-output` (escape non-ASCII, quoted even with `-r`), `--raw-output0` (NUL after each output, NUL in a string is an error), `--seq` (RS before each output, RS read as a separator) and `--stream` (inputs become `tostream` path events, also with `-n`/`-s`). `-a` was accepted and ignored before. +* **Fix**: jq `halt`, `halt_error`, `stderr` and `debug` work like jq. `halt` ends the jq command (not the host) with its code and skips the remaining inputs; `halt_error` prints a string as is or other values as JSON to stderr; `stderr`/`debug` write to the command's stderr in order with its errors (capped buffer) instead of the host logger. TM-INF-023 updated: the guard is now that the run loop never calls `jaq_core::unwrap_valr`. * **Fix**: jq builtins follow jq 1.7: `join` prints null as "" and fails on arrays/objects, `ltrimstr`/`rtrimstr` pass non-strings through, `scan` with capture groups yields the groups, `match` reports an unmatched group (offset -1, string null) so `capture` gives null, `from_entries` takes `key`/`Key`/`name`/`Name` and `value`/`Value`, object keys must be strings, `tostream`/`fromstream`/`truncate_stream`/`IN` are defined, DEL prints as `\u007f`. Debian-oracle jq score 79.9% to 84.0%. * **Fix**: jq assignments through null work like jq: `null | .a.b.c = 1`, `.[2] = 1`, `.a += 1` and slice updates create the containers (they failed with `Cannot iterate over null`); `|= empty` on a missing path keeps null; an index past the end pads the array with null; fractional update indexes truncate; jq's error texts for negative and too-large indexes and for iterating a scalar. Debian-oracle jq score 75.8% to 79.9%. * **Fix**: sed follows GNU on character escapes, `-z`, unreadable inputs and long loops. `\xHH`, `\dNNN`, `\oNNN`, `\cX` work in regexes, replacements and `y` and always produce a literal character (`s/z/\x26/` inserts `&`); `\t`/`\n` work inside bracket expressions (`s/^[ \t]*//`). Under `-z`, `N`/`G`/`H` join with NUL, `D`/`P` split on it, and `M` anchors match around it. An unreadable input file is reported and skipped (exit 2 at the end). The loop budget now counts per input line read, so `$!N;P;D` no longer stopped after 1000 lines. Debian-oracle sed score 94.2% to 99.4%. diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index d66c27c0c..d04e9e1ec 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -448,7 +448,7 @@ execute permission (mode & 0o111); exit 127 missing / 126 non-executable; sheban | TM-INF-020 | `template` exposes env vars via `{{var}}` | Template builtin looks up variables from env as fallback after shell vars and JSON data | Same as TM-INF-001 (caller controls env) | **CALLER RISK** | | TM-INF-021 | Stack backtrace information disclosure | Panics leak internal source paths, dependency versions, and function names via stderr | Custom panic hook suppresses backtraces in CLI | **MITIGATED** | | TM-INF-022 | Library Debug shapes leak via stderr | Builtins wrapping external libs (jaq, regex, serde_json, semver, chrono, …) formatted errors with `{:?}`, dumping internal struct shapes (e.g. the jaq `File` struct incl. spliced compat-defs source) into agent-visible stderr | Three-layer enforcement, all run by `cargo test`: (1) static, `builtins::tests::no_debug_fmt_in_builtin_source` forbids `{:?}` / `{:#?}` / `{name:?}` in every `crates/bashkit/src/builtins/*.rs` (per-line opt-out: `// debug-ok: `); (2) dynamic per-tool, each tool's `mod tests` calls `bashkit::testing::assert_no_leak` with malformed inputs; (3) fuzz, every cargo-fuzz target plus 5 proptest cases call `bashkit::testing::assert_fuzz_invariants` and check the same invariants on stderr/stdout | **FIXED** | -| TM-INF-023 | jq `halt` / `halt_error` terminate the host process | jaq-std 3.0's `halt(N)` native calls `std::process::exit()`; untrusted jq filters could exit the entire embedding process, escaping the `ExecResult` sandbox boundary (process-wide DoS) | Strip the upstream `halt` native from `jaq_std::funs::()` and chain in a safe replacement returning `Error::str("halt is disabled in the bashkit sandbox")`; wrapper defs (`halt`, `halt_error`) still resolve, so callers see a normal jq runtime error (exit 5). Regression tests in `builtins::jq::tests`: `halt_does_not_terminate_host_process`, `halt_with_arg_does_not_terminate_host_process`, `halt_error_does_not_terminate_host_process` | **FIXED** | +| TM-INF-023 | jq `halt` / `halt_error` terminate the host process | jaq-std 3.0's `halt(N)` raises `Exn::halt`, and `jaq_core::unwrap_valr` turns that into `std::process::exit()`; untrusted jq filters could exit the entire embedding process, escaping the `ExecResult` sandbox boundary (process-wide DoS). jaq-std's `stderr_empty`/`debug_empty` also wrote filter values to the host (`log`) | The jq run loop never calls `unwrap_valr`: it matches the exception itself and ends only the jq command, with the halt code as its exit status (jq semantics). `stderr`/`debug`/`halt_error` text goes to the command's stderr through a capped per-run buffer (`builtins/jq/messages.rs`), never the host. Regression tests in `builtins::jq::tests`: `halt_ends_the_command_not_the_host`, `halt_stops_remaining_inputs`, `halt_error_prints_message_and_exits`, `debug_output_is_capped`, `run_loop_never_calls_unwrap_valr` (static guard) | **FIXED** | | TM-INF-024 | Host env side-channel via clap `Arg::env(...)` | uutils-ported clap builtins (currently `ls`: `.env("TABSIZE")`/`.env("TIME_STYLE")`) resolved defaults from `std::env`, not `ctx.env`, (1) presence-probe of host env vars via `ls` behaviour; (2) a host-set `TIME_STYLE` became a value source for an unimplemented option, tripping the unsupported-option gate on every plain `ls` for unrelated tenants | Four-layer fix: (1) codegen, `bashkit-coreutils-port` strips runtime `.env(...)` calls and emits a sidecar `_ENV_DEFAULTS: &[EnvDefault]` table per stripped annotation; (2) virtual-env shim, `crate::builtins::clap_env::apply_env_defaults` rewrites argv from `ctx.env` (never `std::env`) before `try_get_matches_from`, emulating clap's "argv > env > default" precedence; (3) static guards, `no_clap_env_in_generated_parsers` forbids runtime `.env(` in `generated/*.rs`; `every_generated_parser_emits_env_defaults_table` enforces the sidecar on every util; `no_host_env_reads_in_generated_code` recursively forbids `std::env::var*` reads in `generated/**`, covering vendored uucore module bodies (which have no `ctx` and so cannot reach the virtual env) — `bashkit-coreutils-port`'s `fold_host_env` folds names allow-listed in `vendored.toml`'s `host_env` to "unset" at port time and aborts the port on any other env read, so upstream adding one (as it did with a `POSIXLY_CORRECT` probe in `format/argument.rs`) fails drift loudly; (4) defence-in-depth, workspace `clap` drops the `env` cargo feature, so a re-introduced `.env(...)` fails to compile. Coverage: `ls_ignores_host_time_style_and_tabsize` (host env ignored) and `ls_honors_virtual_env_time_style` (virtual env honoured) | **FIXED** | | TM-INF-025 | Untrusted generated Rust runs in drift CI with repository write token | The coreutils argument-drift workflow regenerates Rust from third-party `uutils/coreutils` `uu_app()` then builds/tests bashkit; if the generator preserved arbitrary upstream statements and the job held `contents: write`/`pull-requests: write`, malicious upstream code could execute with repository write impact | Two-layer fix: (1) `bashkit-coreutils-port` validates args-mode `uu_app()` before emission, accepts only a single tail clap `Command` builder chain, or `let = Command::new(...); .(...)` with the tail chained off the let-bound ident; both shapes pass a disallowed-method/closure/macro/block visitor. Regression tests `rejects_executable_statements_in_uu_app`, `rejects_let_with_non_command_initializer`, `rejects_tail_not_chained_off_let_binding`, `rejects_three_statement_body`. (2) `.github/workflows/coreutils-args-drift.yml` separates privilege: regeneration/build/test runs with `contents: read` + `persist-credentials: false`; the `open-pr` job has write permission but never builds or executes generated Rust, only commits the tested artifact | **FIXED** | | TM-INF-026 | Fork-PR secret exfiltration via approved CI run | GitHub's first-time-contributor gate runs the workflow **from the PR head**, and approved contributors auto-run thereafter. The `examples` job in `ci.yml` set `DOPPLER_TOKEN` at job level and `ANTHROPIC_API_KEY` at step level, so a fork PR could edit any `examples/*.rs`/`build.rs`/script the job runs and exfiltrate. `DOPPLER_TOKEN` is the master key, it fetches every other secret in the Doppler config, so one leak compromises all. Every `doppler run --` also injected the entire config into the step env | Trusted-run gates exclude PR execution from secret-backed examples. Dedicated credential-fetch steps receive `DOPPLER_TOKEN`, request one API key, mask it, and exit before build/example processes run. Execution steps receive only the requested API key through step output/env; Docker inherits it by name, without putting its value in argv. No broad token remains in the execution shell or a live Doppler parent. This is process-lifetime isolation on a trusted runner, not a sandbox against compromised dependencies persisting across steps. Regression tests execute the workflow scripts and inspect child/parent environments: `scripts/tests/test_maintenance_security.py`. CI has contents-read permission only and checkout credentials are not persisted anywhere they could reach untrusted code (TM-INF-034). Release examples install the committed dependency lockfile with lifecycle scripts disabled before linking the built artifact; `scripts/tests/test_ci_supply_chain.py` enforces both boundaries | **FIXED** |