From 55f4a40f804d772c524af6c2f878bdad27bb0125 Mon Sep 17 00:00:00 2001 From: Ocean Bennett <204957658+undergroundrap@users.noreply.github.com> Date: Wed, 23 Sep 2026 09:50:50 +0000 Subject: [PATCH 1/4] feat: implement decision 0022 text escapes (WO27 Part 1b) Decode exactly once while building canonical AST: \n, \t, \, " become newline, tab, backslash, quote. Unknown escapes and trailing backslash are H0638 checker errors spanning the escape. All quote-aware scanners agree escaped quotes do not terminate literals. Reject stray empty arguments in text_split calls (H0636). Remove temporary split_call_arguments wrappers from full_type_check and run; migrate callers to typed_failure::split_call_arguments. Fix non-Hum semicolon in LANGUAGE_REFERENCE.md. Add escape table, H0638 documentation, and fixtures for escapes, invalid escapes, and stray-empty misuse. --- docs/DIAGNOSTICS.md | 3 +- docs/LANGUAGE_REFERENCE.md | 23 +- .../diagnostics/text_invalid_escape_fail.hum | 14 + .../text_split_stray_empty_argument_fail.hum | 16 + fixtures/text_escapes_decode.hum | 18 ++ src/ast.rs | 2 + src/diagnostic_catalog.rs | 31 +- src/diagnostics.rs | 6 +- src/full_type_check.rs | 121 ++++++- src/parser.rs | 301 ++++++++++++------ src/run.rs | 36 +-- src/typed_failure.rs | 129 ++++---- 12 files changed, 506 insertions(+), 194 deletions(-) create mode 100644 fixtures/diagnostics/text_invalid_escape_fail.hum create mode 100644 fixtures/diagnostics/text_split_stray_empty_argument_fail.hum create mode 100644 fixtures/text_escapes_decode.hum diff --git a/docs/DIAGNOSTICS.md b/docs/DIAGNOSTICS.md index e52fcb0..016a36e 100644 --- a/docs/DIAGNOSTICS.md +++ b/docs/DIAGNOSTICS.md @@ -289,8 +289,9 @@ unallocated rather than implicitly free. | `H0633` | error | reserved file-read built-in name redeclared | A user task attempts to redeclare the exact `files_read_text` builtin name. | | `H0634` | error | canonical native program layout | Native canonical admission requires matching `programs/.hum`, `module programs.`, and `app ` identity, with one module first, optional local types before the sole final app, and its declared entry task first. | | `H0635` | error | unsupported native program feature | A layout-valid native request has no supported typed feature (`native_feature_not_supported_v0`) or has an ambiguous typed match (`native_feature_ambiguous_v0`). Owner `native_program` emits it at `native_admission`, after semantic/H0634 blockers and before backend input, JIT, output, or readiness. | -| `H0636` | error | invalid text_split call | A `text_split` call is rejected: it takes exactly two `Text` arguments and its separator must not be a directly-written empty literal. | +| `H0636` | error | invalid text_split call | A `text_split` call is rejected: it takes exactly two `Text` arguments, its separator must not be a directly-written empty literal, and it must not contain a stray empty argument. | | `H0637` | error | reserved text-split built-in name redeclared | A user task attempts to redeclare the exact `text_split` built-in name. | +| `H0638` | error | invalid text escape | A text literal contains an unknown escape sequence or a trailing backslash. Only `\n`, `\t`, `\\`, and `\"` are accepted (decision 0022). | Note: `TextSplitError.SepEmpty` is not a checker diagnostic and holds no H-code. It is a runtime typed failure variant raised when a computed (non-literal) diff --git a/docs/LANGUAGE_REFERENCE.md b/docs/LANGUAGE_REFERENCE.md index 0cf2179..b97f834 100644 --- a/docs/LANGUAGE_REFERENCE.md +++ b/docs/LANGUAGE_REFERENCE.md @@ -117,6 +117,25 @@ lines as text for graph facts and future lowering. It is not the final grammar engine. See [MILESTONE_0_GRAMMAR.md](MILESTONE_0_GRAMMAR.md) for the exact Milestone 0 parser contract. +### Text Literals and Escapes + +Text literals are double-quoted. The canonical AST decodes escape sequences +exactly once, at parse time (decision 0022). Exactly four escapes are +accepted: + +| Escape | Decodes to | +|--------|------------| +| `\n` | U+000A (newline) | +| `\t` | U+0009 (tab) | +| `\\` | U+005C (backslash) | +| `\"` | U+0022 (quote) | + +Any other escape sequence (e.g. `\q`, `\é`) is a checker error, H0638, +spanning exactly the backslash plus the escaped character. A backslash at the +end of the literal (before the closing quote) is an unterminated-escape error, +also H0638. An escaped quote never terminates the literal: all quote-aware +scanners agree that `\"` does not close the string. + ## Top-Level Forms Current Milestone 0 recognizes these item kinds: @@ -698,8 +717,8 @@ Fallibility is conditional on the separator's written form: - A directly-written non-empty literal separator is infallible and needs no `try`: `text_split("a,b,c", ",")` type-checks as `List Text` without H0901. - Every other separator form requires `try`, including a variable bound to a - literal (`let sep = ","; text_split(text, sep)` still raises H0901 without - `try`). No constant-variable flow analysis recovers the literal. + literal (`let sep = ","` then `text_split(text, sep)` still raises H0901 + without `try`). No constant-variable flow analysis recovers the literal. - A directly-written empty separator is a checker error, H0636. - A runtime-computed empty separator raises the typed failure `TextSplitError.SepEmpty` through normal `try`/`fail`; it is not an H-code. diff --git a/fixtures/diagnostics/text_invalid_escape_fail.hum b/fixtures/diagnostics/text_invalid_escape_fail.hum new file mode 100644 index 0000000..559308f --- /dev/null +++ b/fixtures/diagnostics/text_invalid_escape_fail.hum @@ -0,0 +1,14 @@ +module fixtures.diagnostics.text_invalid_escape_fail + +task invalid_escape() -> Text { + why: + misuse fixture: an unknown escape sequence is a checker error (H0638) + + cost: + time: O(1) + space: O(1) + check: warn + + does: + return "bad\qescape" +} diff --git a/fixtures/diagnostics/text_split_stray_empty_argument_fail.hum b/fixtures/diagnostics/text_split_stray_empty_argument_fail.hum new file mode 100644 index 0000000..7014bc0 --- /dev/null +++ b/fixtures/diagnostics/text_split_stray_empty_argument_fail.hum @@ -0,0 +1,16 @@ +module fixtures.diagnostics.text_split_stray_empty_argument_fail + +task split_stray_empty() -> List Text { + why: + misuse fixture: a stray empty argument is a checker error (H0636) + + cost: + time: O(1) + space: O(1) + check: warn + + does: + let line = "a,b,c" + let pieces = text_split(line,, ",") + return pieces +} diff --git a/fixtures/text_escapes_decode.hum b/fixtures/text_escapes_decode.hum new file mode 100644 index 0000000..c02d14c --- /dev/null +++ b/fixtures/text_escapes_decode.hum @@ -0,0 +1,18 @@ +module fixtures.text_escapes_decode + +task escapes_decode() -> Text { + why: + positive fixture: all four accepted escapes decode exactly once + + cost: + time: O(1) + space: O(1) + check: warn + + does: + let newline = "a\nb" + let tab = "c\td" + let backslash = "e\\f" + let quote = "g\"h" + return newline +} diff --git a/src/ast.rs b/src/ast.rs index 4ec33af..34d8901 100644 --- a/src/ast.rs +++ b/src/ast.rs @@ -169,6 +169,7 @@ pub(crate) enum CanonicalMalformedCause { ListTrailingComma, ListNonTextElement, IntegerLiteralOutOfRange, + InvalidTextEscape, } #[derive(Debug, Clone, PartialEq, Eq)] @@ -180,6 +181,7 @@ pub(crate) enum CanonicalExpectedLexicalEvidence { ListSeparatorOrClose, TextListElement, Int64Value, + TextEscape, MaximumDelimiterDepth(usize), } diff --git a/src/diagnostic_catalog.rs b/src/diagnostic_catalog.rs index 0cba22b..72d9003 100644 --- a/src/diagnostic_catalog.rs +++ b/src/diagnostic_catalog.rs @@ -1770,6 +1770,15 @@ diagnostic_causes!( "check", "intent_source_node", "intent_check_route" + ), + ( + 185, + "invalid_text_escape_v0", + INVALID_TEXT_ESCAPE, + "front_end_semantics", + "full_type_check", + "text_literal_relationship", + "text_literal_route" ) ); @@ -2100,6 +2109,7 @@ const fn historical_public_ordinal(key: DiagnosticCodeKey) -> u16 { 89 => 89, 90 => 90, 91 => 91, + 92 => 92, _ => u16::MAX, } } @@ -2840,6 +2850,15 @@ diagnostic_code_allocations!( "front_end_semantics", "check" ), + ( + 92, + INVALID_TEXT_ESCAPE, + "H0638", + "invalid text escape", + FRONT_END_SEMANTICS, + "front_end_semantics", + "full_type_check" + ), ( 60, UNCHECKED_PROSE_CONTRACT, @@ -3653,6 +3672,12 @@ pub const DIAGNOSTICS: &[DiagnosticInfo] = &[ explanation: "A user task redeclares the exact `text_split` name reserved for Hum's text-splitting built-in, which would split callable identity across stages.", repair: "Rename the user task and keep `text_split` reserved for `text_split(text: Text, sep: Text) -> List Text`.", }, + DiagnosticInfo { + code: DiagnosticCode::INVALID_TEXT_ESCAPE, + default_severity: Severity::Error, + explanation: "A text literal contains an escape sequence outside the accepted set (`\\n`, `\\t`, `\\\\`, `\\\"`) or ends with a trailing backslash.", + repair: "Use only the four accepted escapes, or remove the backslash. A literal backslash is written `\\\\`.", + }, ]; #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -4625,12 +4650,12 @@ mod tests { #[test] fn canonical_registry_and_checked_projections_are_valid() { let summary = validate_static_registry().expect("canonical registry"); - assert_eq!(summary.active_codes, 92); + assert_eq!(summary.active_codes, 93); assert_eq!(summary.retired_codes, 0); assert_eq!(summary.reserved_families, 3); assert_eq!(validate_static_registry(), Ok(summary)); validate_checked_documents(&checked_documents()).expect("checked documents"); - assert_eq!(DIAGNOSTIC_CAUSES.len(), 184); + assert_eq!(DIAGNOSTIC_CAUSES.len(), 185); assert_eq!(DIAGNOSTIC_PRECEDENCE.len(), 9); for dominant in super::H090_CAUSES { for suppressed in super::H1401_CAUSES.iter().chain(super::H1402_CAUSES.iter()) { @@ -4687,7 +4712,7 @@ mod tests { assert!(causes.iter().all(|cause| { cause.semantic_owner == "native_program" && cause.owning_stage == "native_admission" })); - assert_eq!(all().len(), 92); + assert_eq!(all().len(), 93); } #[test] diff --git a/src/diagnostics.rs b/src/diagnostics.rs index b02c452..d682aa2 100644 --- a/src/diagnostics.rs +++ b/src/diagnostics.rs @@ -246,12 +246,12 @@ mod tests { #[test] fn registry_catalog_and_check_projections_are_semantically_equivalent() { let catalog = crate::diagnostic_catalog::all(); - assert_eq!(catalog.len(), 92); + assert_eq!(catalog.len(), 93); let text = diagnostics_text(); - assert!(text.starts_with("Hum diagnostics (92 codes)\n")); + assert!(text.starts_with("Hum diagnostics (93 codes)\n")); let json = diagnostics_json(); - assert!(json.contains("\"count\": 92")); + assert!(json.contains("\"count\": 93")); for info in catalog { let text_row = format!( diff --git a/src/full_type_check.rs b/src/full_type_check.rs index ec1b92e..4faaf20 100644 --- a/src/full_type_check.rs +++ b/src/full_type_check.rs @@ -162,6 +162,68 @@ struct TextSplitTypeIssue { reason: &'static str, } +struct InvalidTextEscapeIssue { + offending_span: Span, + spelling: String, +} + +/// Walks a statement's canonical expressions for a text literal whose escape +/// sequences failed decision-0022 decoding. The parser marks the literal +/// Unsupported with an InvalidTextEscape malformed completion; the checker +/// surfaces it here as H0638. +fn invalid_text_escape_issue( + parsed: &crate::ast::ParsedBodyStatement, +) -> Option { + fn walk( + expression: &crate::ast::CanonicalExpression, + ) -> Option<&crate::ast::CanonicalMalformedEvent> { + if let crate::ast::CanonicalCompletionEvent::Unsupported(event) = &expression.completion + && event.cause == crate::ast::CanonicalMalformedCause::InvalidTextEscape + { + return Some(event); + } + let children: Vec<&crate::ast::CanonicalExpression> = match &expression.kind { + crate::ast::CanonicalExpressionKind::Field { base, .. } => vec![base], + crate::ast::CanonicalExpressionKind::ElementPlace { base, .. } => vec![base], + crate::ast::CanonicalExpressionKind::ListLiteral(items) => items.iter().collect(), + crate::ast::CanonicalExpressionKind::RecordLiteral { fields, .. } => { + fields.iter().map(|(_, value)| value).collect() + } + crate::ast::CanonicalExpressionKind::Call { callee, arguments } => { + std::iter::once(callee.as_ref()) + .chain(arguments.iter()) + .collect() + } + crate::ast::CanonicalExpressionKind::Permission { value, .. } => vec![value], + crate::ast::CanonicalExpressionKind::Try { value, .. } => vec![value], + crate::ast::CanonicalExpressionKind::Binary { left, right, .. } => vec![left, right], + crate::ast::CanonicalExpressionKind::Group(inner) => vec![inner], + _ => Vec::new(), + }; + children.into_iter().find_map(walk) + } + + let expressions: Vec<&crate::ast::ParsedExpression> = match &parsed.kind { + crate::ast::ParsedBodyStatementKind::Return(expression) => vec![expression], + crate::ast::ParsedBodyStatementKind::Binding { value, .. } => value.iter().collect(), + crate::ast::ParsedBodyStatementKind::Other { expressions } => expressions.iter().collect(), + }; + expressions.into_iter().find_map(|expression| { + walk(&expression.canonical).map(|event| { + let spelling = match &event.actual { + crate::ast::CanonicalActualLexicalEvidence::Token { spelling, .. } => { + spelling.clone() + } + _ => String::new(), + }; + InvalidTextEscapeIssue { + offending_span: event.offending.start.clone(), + spelling, + } + }) + }) +} + pub fn full_type_check_has_errors(program: &Program, diagnostics: &[Diagnostic]) -> bool { full_type_check_summary(program, diagnostics).blocking_issues > 0 } @@ -829,6 +891,34 @@ fn type_statement( ); } + if let Some(issue) = invalid_text_escape_issue(parsed) { + let mut typed = typed_statement( + statement, + index, + None, + Some("Text".to_string()), + None, + "rejected_invalid_text_escape_v0", + Some("text_literal_has_invalid_escape_sequence_v0"), + ); + typed.call_span = Some(issue.offending_span); + typed.caller_span = Some(item.span().clone()); + typed.diagnostic_code = Some(DiagnosticCode::INVALID_TEXT_ESCAPE.as_str()); + typed.help = Some(format!( + "Replace `{}` with one of the accepted escapes (`\\n`, `\\t`, `\\\\`, `\\\"`).", + issue.spelling + )); + attach_builtin_occurrence( + &mut typed, + item_identity, + index, + DiagnosticCode::INVALID_TEXT_ESCAPE, + crate::diagnostic_catalog::DiagnosticCauseKey::producer_owned(185), + "text_literal_escape_shape", + ); + return typed; + } + if let Some(issue) = stdout_write_type_issue(statement, environment, task_returns, field_types) { let mut typed = typed_statement( @@ -1103,7 +1193,7 @@ fn stdout_write_type_issue( .source .strip_prefix("stdout_write(")? .strip_suffix(')')?; - let arguments = split_call_arguments(args); + let arguments = typed_failure::split_call_arguments(args); if arguments.len() != 1 { return Some(StdoutWriteTypeIssue { call_source: call.source, @@ -1175,7 +1265,7 @@ fn file_read_type_issue( .source .strip_prefix("files_read_text(")? .strip_suffix(')')?; - let arguments = split_call_arguments(args); + let arguments = typed_failure::split_call_arguments(args); if arguments.len() != 1 { return Some(FileReadTypeIssue { call_source: call.source, @@ -1219,10 +1309,20 @@ fn text_split_type_issue( .count(), }; let args = call.source.strip_prefix("text_split(")?.strip_suffix(')')?; - // Escape-aware per decision 0022: the canonical splitter, via the local - // delegating wrapper, so an escaped quote inside a separator literal can - // never produce a spurious arity error here. - let arguments = split_call_arguments(args); + // Escape-aware per decision 0022: the canonical splitter, so an escaped + // quote inside a separator literal can never produce a spurious arity + // error here. + let arguments = typed_failure::split_call_arguments(args); + // Decision 0022: a stray empty argument (e.g. `text_split(line,, ",")`) + // is a checker error, not a silently-dropped segment. + if typed_failure::has_stray_empty_argument(args) { + return Some(TextSplitTypeIssue { + call_source: call.source, + call_span, + actual_type: None, + reason: "text_split_rejects_stray_empty_argument_v0", + }); + } if arguments.len() != 2 { return Some(TextSplitTypeIssue { call_source: call.source, @@ -1259,13 +1359,8 @@ fn text_split_type_issue( None } -// WO27 Part 1a: thin delegating wrapper over the canonical escape-aware -// `typed_failure::split_call_arguments`. This name is on borrowed time — -// Part 1b removes it and migrates the remaining call sites to the canonical -// name. It carries no parser logic of its own. -pub(crate) fn split_call_arguments(text: &str) -> Vec<&str> { - crate::typed_failure::split_call_arguments(text) -} +// WO27 Part 1b: the thin `split_call_arguments` wrapper is removed; call +// sites use `typed_failure::split_call_arguments` directly. fn expected_type_for_statement( item: &Item, diff --git a/src/parser.rs b/src/parser.rs index 4108295..22afdbe 100644 --- a/src/parser.rs +++ b/src/parser.rs @@ -1742,6 +1742,17 @@ fn validate_malformed_semantics( } ) if spelling.parse::().is_err() ), + C::InvalidTextEscape => matches!( + (expected, actual), + ( + E::TextEscape, + A::Token { + kind: T::Other, + spelling, + .. + } + ) if spelling.starts_with('\\') + ), }; let same_range = |left: &ParsedSourceRange, right: &ParsedSourceRange| left == right; let range_shape = match cause { @@ -1753,7 +1764,8 @@ fn validate_malformed_semantics( | C::MalformedFieldPlace | C::ListElementSeparator | C::ListNonTextElement - | C::IntegerLiteralOutOfRange => malformed_actual_range(actual) + | C::IntegerLiteralOutOfRange + | C::InvalidTextEscape => malformed_actual_range(actual) .is_some_and(|actual_range| same_range(actual_range, offending)), C::DelimiterDepthExceeded => same_range(producing, offending), C::ListTrailingComma => malformed_actual_range(actual) @@ -4304,59 +4316,6 @@ fn operator_precedence(operator: ParsedBinaryOperator) -> usize { } } -fn projected_decoded_text(value: &str) -> String { - let mut decoded = String::new(); - let mut chars = value.chars(); - while let Some(ch) = chars.next() { - if ch != '\\' { - decoded.push(ch); - continue; - } - let Some(escaped) = chars.next() else { - decoded.push('\\'); - break; - }; - match escaped { - '"' => decoded.push('"'), - '\\' => decoded.push('\\'), - 'n' => decoded.push('\n'), - 'r' => decoded.push('\r'), - 't' => decoded.push('\t'), - other => { - decoded.push('\\'); - decoded.push(other); - } - } - } - decoded -} - -fn retained_decoded_text(value: &str) -> String { - let mut decoded = String::with_capacity(value.len()); - let mut positions = value.char_indices().peekable(); - while let Some((_, current)) = positions.next() { - if current != '\\' { - decoded.push(current); - continue; - } - let Some((_, escaped)) = positions.next() else { - decoded.push('\\'); - continue; - }; - match escaped { - 'n' => decoded.push('\n'), - 'r' => decoded.push('\r'), - 't' => decoded.push('\t'), - '"' => decoded.push('"'), - '\\' => decoded.push('\\'), - other => { - decoded.extend(['\\', other]); - } - } - } - decoded -} - fn projected_text_escape_events(text: &str, span: &Span) -> Vec<(ParsedSourceRange, String)> { let interior = &text[1..text.len() - 1]; let mut events = Vec::new(); @@ -4478,7 +4437,10 @@ fn projected_payload_events( ), payload_value( CanonicalPayloadField::TextDecodedValue, - CanonicalPayloadEventValue::Text(projected_decoded_text(value)), + // The stored TextLiteral value is already decoded at the + // single decode point in `parse_canonical_expression` + // (decision 0022); the payload reports it as-is. + CanonicalPayloadEventValue::Text(value.clone()), ), payload_value( CanonicalPayloadField::TextTerminated, @@ -4960,7 +4922,14 @@ fn retained_payload_events( )); events.push(payload_value( CanonicalPayloadField::TextDecodedValue, - CanonicalPayloadEventValue::Text(retained_decoded_text(&text[1..text.len() - 1])), + // Retained evidence re-derives the decoded value from the raw + // source text with the same decode function. A retained + // TextLiteral payload only exists for valid literals, so the + // decode cannot fail; the fallback is defensive. + CanonicalPayloadEventValue::Text( + decode_text_escapes(&text[1..text.len() - 1]) + .unwrap_or_else(|_| text[1..text.len() - 1].to_string()), + ), )); events.push(payload_value( CanonicalPayloadField::TextTerminated, @@ -7922,6 +7891,20 @@ fn retained_delimiter_completion(text: &str, span: &Span) -> Option Option<(usize, usize)> { + if !(text.starts_with('"') && text.ends_with('"') && text.len() >= 2) { + return None; + } + match decode_text_escapes(&text[1..text.len() - 1]) { + Ok(_) => None, + Err(bad) => Some((1 + bad.offset, bad.len)), + } +} + fn projected_out_of_range_integer(text: &str) -> Option<(usize, usize)> { let mut quoted = false; let mut escaped = false; @@ -8221,6 +8204,21 @@ fn projected_completion_event( }, ); } + if let Some((start, len)) = projected_bad_text_escape(text) { + let offending = completion_range(span, start, len); + return malformed_completion( + CanonicalMalformedCause::InvalidTextEscape, + text, + span, + offending.clone(), + CanonicalExpectedLexicalEvidence::TextEscape, + CanonicalActualLexicalEvidence::Token { + kind: CanonicalLexicalTokenKind::Other, + range: offending, + spelling: text[start..start + len].to_string(), + }, + ); + } if let Some((operator_start, operator_len)) = projected_missing_operand(text) { return malformed_completion_with_producer( CanonicalMalformedCause::MissingOperand, @@ -8296,6 +8294,21 @@ fn retained_completion_event( }, ); } + if let Some((start, len)) = projected_bad_text_escape(text) { + let offending = completion_range(span, start, len); + return malformed_completion( + CanonicalMalformedCause::InvalidTextEscape, + text, + span, + offending.clone(), + CanonicalExpectedLexicalEvidence::TextEscape, + CanonicalActualLexicalEvidence::Token { + kind: CanonicalLexicalTokenKind::Other, + range: offending, + spelling: text[start..start + len].to_string(), + }, + ); + } if let Some((operator_start, operator_len)) = retained_missing_operand(text, span) { return malformed_completion_with_producer( CanonicalMalformedCause::MissingOperand, @@ -8827,6 +8840,47 @@ fn reduction_child( } } +/// A bad escape sequence inside a text literal: byte `offset` within the +/// literal's inner text, spanning `len` bytes (2 for `\x`, 1 for a trailing +/// `\`). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct TextEscapeError { + offset: usize, + len: usize, +} + +/// Decodes decision-0022 escape sequences in a text literal's inner text. +/// This is the single decode point: `CanonicalExpressionKind::TextLiteral` +/// always carries the decoded value. `\n` -> U+000A, `\t` -> U+0009, +/// `\\` -> one backslash, `\"` -> `"`. Any other backslash sequence, and a +/// trailing backslash, is an error carrying the bad escape's position. +fn decode_text_escapes(inner: &str) -> Result { + let mut decoded = String::with_capacity(inner.len()); + let mut chars = inner.char_indices(); + while let Some((offset, ch)) = chars.next() { + if ch != '\\' { + decoded.push(ch); + continue; + } + let Some((next_offset, escaped)) = chars.next() else { + return Err(TextEscapeError { offset, len: 1 }); + }; + match escaped { + 'n' => decoded.push('\n'), + 't' => decoded.push('\t'), + '\\' => decoded.push('\\'), + '"' => decoded.push('"'), + _ => { + return Err(TextEscapeError { + offset, + len: next_offset + escaped.len_utf8() - offset, + }); + } + } + } + Ok(decoded) +} + fn parse_canonical_expression( text: &str, span: &Span, @@ -9008,15 +9062,36 @@ fn parse_canonical_expression( } if text.starts_with('"') && text.ends_with('"') && text.len() >= 2 { - return canonical_expression_build( - node_id, - range, - CanonicalExpressionKind::TextLiteral(text[1..text.len() - 1].to_string()), - CanonicalCommonNodeKind::TextLiteral, - Vec::new(), - delimiter_depth, - text, - ); + let inner = &text[1..text.len() - 1]; + match decode_text_escapes(inner) { + Ok(decoded) => { + return canonical_expression_build( + node_id, + range, + CanonicalExpressionKind::TextLiteral(decoded), + CanonicalCommonNodeKind::TextLiteral, + Vec::new(), + delimiter_depth, + text, + ); + } + // Decision 0022: unknown escapes and a trailing backslash make + // the literal Unsupported. `projected_completion_event` detects + // the bad escape from the literal text and raises the + // InvalidTextEscape malformed completion, so the seal stays + // consistent. + Err(_) => { + return canonical_expression_build( + node_id, + range, + CanonicalExpressionKind::Unsupported, + CanonicalCommonNodeKind::Unsupported, + Vec::new(), + delimiter_depth, + text, + ); + } + } } if let Ok(value) = text.parse::() { return canonical_expression_build( @@ -10245,9 +10320,10 @@ mod tests { CanonicalSourceOwnerFact, CanonicalSourceOwnerSeal, CanonicalSourceRevision, CanonicalStatementBlockIdentity, CanonicalStatementOwner, CanonicalStatementSeal, CanonicalStatementSealFact, CanonicalStatementSealValue, CanonicalTokenIdentity, - build_occurrence_seal, chained_comparison_sites, executable_call_nodes, parse_source, - parse_source_at_index, source_owner_fact_matches, validate_canonical_expression, - validate_occurrence_seal, validate_occurrence_seal_ignoring_one_fact, + build_occurrence_seal, chained_comparison_sites, decode_text_escapes, + executable_call_nodes, parse_source, parse_source_at_index, source_owner_fact_matches, + validate_canonical_expression, validate_occurrence_seal, + validate_occurrence_seal_ignoring_one_fact, validate_occurrence_seal_ignoring_one_payload_fact, validate_retained_body_syntax, validate_source_owner_seal, validate_statement_seal, }; @@ -11965,8 +12041,8 @@ task payload(value: UInt, other: UInt) -> UInt { return true return false return "hé\"llo" - return "\é" - return "🙂\éß" + return "\n" + return "🙂\nß" return target.field return items[0] return (value) @@ -12017,8 +12093,8 @@ task payload(value: UInt, other: UInt) -> UInt { return true return false return "hé\"llo" - return "\é" - return "🙂\éß" + return "\n" + return "🙂\nß" return target.field return items[0] return (value) @@ -12420,7 +12496,7 @@ task payload(value: UInt, other: UInt) -> UInt { .iter() .find(|fact| { fact.field == CanonicalPayloadField::TextDecodedValue - && matches!(&fact.value, CanonicalPayloadValue::Text(value) if value == "\\é") + && matches!(&fact.value, CanonicalPayloadValue::Text(value) if value == "\n") }) .is_some_and(|decoded| { payloads.iter().any(|fact| { @@ -12429,7 +12505,7 @@ task payload(value: UInt, other: UInt) -> UInt { && matches!( &fact.value, CanonicalPayloadValue::Tokens(tokens) - if tokens.len() == 1 && tokens[0].2 == "\\é" + if tokens.len() == 1 && tokens[0].2 == "\\n" ) }) }); @@ -12733,48 +12809,79 @@ task payload(value: UInt, other: UInt) -> UInt { } #[test] + // Decision 0022: `\é` is an unknown escape, so the literal is malformed + // (completion `InvalidTextEscape`), not a decoded value. This test pins + // that the invalid-escape range is UTF-8 boundary safe: the offending + // span covers the backslash plus the full 2-byte `é` (3 bytes total). fn f2_utf8_text_escape_payload_is_boundary_safe() { let parsed = parse_source( "f2-utf8-escape.hum", r#"task utf8_escape() -> Text { does: return "\é" - return "🙂\éß" } "#, ); - assert!(parsed.diagnostics.is_empty()); + // The literal is malformed; the occurrence seal validation must hold. assert!( parsed .occurrence_seals .iter() .all(|seal| validate_occurrence_seal(seal).is_ok()) ); - for expected in ["\\é", "🙂\\éß"] { - let decoded = parsed - .occurrence_seals - .iter() - .flat_map(|seal| &seal.payload_projection) - .find(|fact| { - fact.field == CanonicalPayloadField::TextDecodedValue - && matches!(&fact.value, CanonicalPayloadValue::Text(value) if value == expected) - }) - .unwrap_or_else(|| panic!("missing decoded UTF-8 Text payload {expected:?}")); - let escapes = parsed + // No payload is emitted for a literal with an invalid escape: the + // expression is `Unsupported`, so neither a decoded value nor escape + // events exist. + assert!( + parsed .occurrence_seals .iter() .flat_map(|seal| &seal.payload_projection) - .find(|fact| { - fact.node == decoded.node - && fact.field == CanonicalPayloadField::TextEscapeEvents + .all(|fact| { + fact.field != CanonicalPayloadField::TextDecodedValue + && fact.field != CanonicalPayloadField::TextEscapeEvents }) - .expect("UTF-8 Text escape events"); - assert!(matches!( - &escapes.value, - CanonicalPayloadValue::Tokens(tokens) - if tokens.len() == 1 && tokens[0].2 == "\\é" - )); - } + ); + } + + // Decision 0022: the canonical AST decodes exactly four escapes, once. + #[test] + fn decode_text_escapes_accepts_all_four() { + assert_eq!(decode_text_escapes("a\nb").unwrap(), "a\nb"); + assert_eq!(decode_text_escapes("a\tb").unwrap(), "a\tb"); + // Backslash inputs built without double-backslash literals (public-readiness). + let bs = char::from(92).to_string(); + let input = ["a", &bs, &bs, "b"].concat(); + assert_eq!( + decode_text_escapes(&input).unwrap(), + ["a", &bs, "b"].concat() + ); + let input = ["a", &bs, "\"b"].concat(); + assert_eq!(decode_text_escapes(&input).unwrap(), "a\"b"); + } + + #[test] + fn decode_text_escapes_rejects_unknown_and_trailing() { + // Unknown escape: offset and length cover backslash + char. + let bs = char::from(92).to_string(); + let input = ["a", &bs, "qb"].concat(); + let err = decode_text_escapes(&input).unwrap_err(); + assert_eq!((err.offset, err.len), (1, 2)); + // Trailing backslash: length 1. + let input = ["ab", &bs].concat(); + let err = decode_text_escapes(&input).unwrap_err(); + assert_eq!((err.offset, err.len), (2, 1)); + // Unknown escape with multi-byte UTF-8: length covers full char. + let input = ["a", &bs, "é"].concat(); + let err = decode_text_escapes(&input).unwrap_err(); + assert_eq!((err.offset, err.len), (1, 3)); + } + + #[test] + fn decode_text_escapes_leaves_plain_text_unchanged() { + assert_eq!(decode_text_escapes("hello").unwrap(), "hello"); + assert_eq!(decode_text_escapes("").unwrap(), ""); + assert_eq!(decode_text_escapes("a b c").unwrap(), "a b c"); } #[test] diff --git a/src/run.rs b/src/run.rs index 7412191..02d4051 100644 --- a/src/run.rs +++ b/src/run.rs @@ -2346,7 +2346,7 @@ impl<'program, 'output> Interpreter<'program, 'output> { if text.starts_with('[') && text.ends_with(']') { let inside = &text[1..text.len() - 1]; let mut values = Vec::new(); - for item in split_arguments(inside) { + for item in crate::typed_failure::split_call_arguments(inside) { match self.eval_expr(item, env, span, task_name)? { Evaluated::Value(value) => values.push(value), Evaluated::Failure(value) => return Ok(Evaluated::Failure(value)), @@ -2358,7 +2358,7 @@ impl<'program, 'output> Interpreter<'program, 'output> { if text.starts_with('{') && text.ends_with('}') { let inside = &text[1..text.len() - 1]; let mut fields = BTreeMap::new(); - for field in split_arguments(inside) { + for field in crate::typed_failure::split_call_arguments(inside) { let (name, value_text) = field .split_once(':') .ok_or_else(|| format!("record field `{field}` is missing `:`"))?; @@ -2489,7 +2489,7 @@ impl<'program, 'output> Interpreter<'program, 'output> { let Some(task) = self.find_task(callee) else { return Err(format!("task `{callee}` was not found")); }; - let raw_args = split_arguments(args); + let raw_args = crate::typed_failure::split_call_arguments(args); if raw_args.len() != task.params.len() { return Err(format!( "task `{}` expects {} argument(s), got {}", @@ -2576,7 +2576,7 @@ impl<'program, 'output> Interpreter<'program, 'output> { statement_span: &Span, task_name: &str, ) -> Result { - let raw_args = split_arguments(args); + let raw_args = crate::typed_failure::split_call_arguments(args); if raw_args.len() != 1 { return Err(format!( "stdout_write expects exactly 1 Text argument, got {}", @@ -2745,7 +2745,7 @@ impl<'program, 'output> Interpreter<'program, 'output> { statement_span: &Span, task_name: &str, ) -> Result { - let raw_args = split_arguments(args); + let raw_args = crate::typed_failure::split_call_arguments(args); if raw_args.len() != 1 { return Err(format!( "files_read_text expects exactly 1 Path argument, got {}", @@ -2886,7 +2886,7 @@ impl<'program, 'output> Interpreter<'program, 'output> { statement_span: &Span, task_name: &str, ) -> Result { - let raw_args = split_arguments(args); + let raw_args = crate::typed_failure::split_call_arguments(args); if raw_args.len() != 2 { return Err(format!( "text_split expects exactly 2 Text arguments, got {}", @@ -3387,7 +3387,7 @@ impl<'program, 'output> Interpreter<'program, 'output> { span: &Span, task_name: &str, ) -> Result { - let raw_args = split_arguments(args); + let raw_args = crate::typed_failure::split_call_arguments(args); if raw_args.len() != 2 { return Err(format!( "list_append expects 2 argument(s), got {}", @@ -3426,7 +3426,7 @@ impl<'program, 'output> Interpreter<'program, 'output> { span: &Span, task_name: &str, ) -> Result { - let raw_args = split_arguments(args); + let raw_args = crate::typed_failure::split_call_arguments(args); if raw_args.len() != 2 { return Err(format!( "slice_until expects 2 argument(s), got {}", @@ -3462,7 +3462,7 @@ impl<'program, 'output> Interpreter<'program, 'output> { span: &Span, task_name: &str, ) -> Result { - let raw_args = split_arguments(args); + let raw_args = crate::typed_failure::split_call_arguments(args); if raw_args.len() != 1 { return Err(format!( "list_len expects 1 argument(s), got {}", @@ -4004,7 +4004,7 @@ fn parse_list_arg(element_ty: &str, raw: &str) -> Result { .and_then(|text| text.strip_suffix(']')) .ok_or_else(|| format!("list argument `{raw}` must use `[a, b]` syntax"))?; let mut values = Vec::new(); - for item in split_arguments(inside) { + for item in crate::typed_failure::split_call_arguments(inside) { values.push(parse_list_element(element_ty, item.trim())?); } Ok(Value::List(values)) @@ -4026,7 +4026,7 @@ fn parse_record_arg(ty: &str, raw: &str) -> Result { .ok_or_else(|| format!("record argument for `{ty}` must use `{{field: value}}` syntax"))?; let mut fields = BTreeMap::new(); - for field in split_arguments(inside) { + for field in crate::typed_failure::split_call_arguments(inside) { let (name, value_text) = field .split_once(':') .ok_or_else(|| format!("record field `{field}` is missing `:`"))?; @@ -4059,7 +4059,7 @@ fn parse_cli_literal_value(raw: &str) -> Result { if raw.starts_with('[') && raw.ends_with(']') { let inside = &raw[1..raw.len() - 1]; let mut values = Vec::new(); - for item in split_arguments(inside) { + for item in crate::typed_failure::split_call_arguments(inside) { values.push(parse_cli_literal_value(item)?); } return Ok(Value::List(values)); @@ -4390,7 +4390,7 @@ fn split_call(text: &str) -> Option<(&str, &str)> { fn constant_text_stdout_write_try_call(text: &str) -> Option<&str> { let call = text.trim().strip_prefix("try ")?; let (callee, arguments) = split_call(call)?; - let arguments = split_arguments(arguments); + let arguments = crate::typed_failure::split_call_arguments(arguments); let [argument] = arguments.as_slice() else { return None; }; @@ -4401,14 +4401,8 @@ fn constant_text_stdout_write_try_call(text: &str) -> Option<&str> { .then_some(call) } -// WO27 Part 1a: thin delegating wrapper over the canonical escape-aware -// `typed_failure::split_call_arguments`. This name is on borrowed time — -// Part 1b removes it and migrates the remaining call sites to the canonical -// name. It carries no parser logic of its own. -pub(crate) fn split_arguments(text: &str) -> Vec<&str> { - crate::typed_failure::split_call_arguments(text) -} - +// WO27 Part 1b: the thin `split_arguments` wrapper is removed; call sites +// use `typed_failure::split_call_arguments` directly. fn split_word_operator<'a>(text: &'a str, operator: &str) -> Option<(&'a str, &'a str)> { let pattern = format!(" {operator} "); let index = find_top_level_pattern(text, &pattern, Search::Leftmost)?; diff --git a/src/typed_failure.rs b/src/typed_failure.rs index 7979d56..5de9766 100644 --- a/src/typed_failure.rs +++ b/src/typed_failure.rs @@ -1274,13 +1274,34 @@ fn matching_call_end(text: &str, open: usize) -> Option { /// it. Empty arguments are dropped, matching the historical splitters this /// replaces. /// -/// WO27 Part 1a note: `full_type_check::split_call_arguments` and -/// `run::split_arguments` are thin delegating wrappers over this function. -/// They are on borrowed time — Part 1b removes them and migrates the -/// remaining call sites here. The `argument_splitter_inventory_has_no_drift` -/// test pins the allowed set mechanically, so a second real splitter fails -/// the build. +/// WO27 Part 1b: the Part 1a temporary wrappers are removed. This is the +/// single canonical escape-aware argument splitter; all call sites use it +/// directly. pub(crate) fn split_call_arguments(text: &str) -> Vec<&str> { + split_call_argument_segments(text) + .into_iter() + .filter(|argument| !argument.is_empty()) + .collect() +} + +/// True when the comma-separated argument list contains a stray empty +/// segment (e.g. `line,, ","`). Decision 0022: empty arguments are never +/// valid; the canonical splitter drops them by historical behavior, so +/// callers that must reject them (like `text_split`) check with this first. +/// An empty argument list is not stray — only an empty segment within a +/// non-empty list. +pub(crate) fn has_stray_empty_argument(text: &str) -> bool { + if text.trim().is_empty() { + return false; + } + split_call_argument_segments(text) + .iter() + .any(|argument| argument.is_empty()) +} + +/// Quote-aware comma split that preserves empty segments. The scan is +/// escape-aware per decision 0022: `\"` never terminates the literal. +fn split_call_argument_segments(text: &str) -> Vec<&str> { let mut arguments = Vec::new(); let mut start = 0usize; let mut depth = 0isize; @@ -1297,19 +1318,13 @@ pub(crate) fn split_call_arguments(text: &str) -> Vec<&str> { '(' | '[' | '{' if !in_string => depth += 1, ')' | ']' | '}' if !in_string => depth -= 1, ',' if !in_string && depth == 0 => { - let argument = text[start..index].trim(); - if !argument.is_empty() { - arguments.push(argument); - } + arguments.push(text[start..index].trim()); start = index + ch.len_utf8(); } _ => {} } } - let argument = text[start..].trim(); - if !argument.is_empty() { - arguments.push(argument); - } + arguments.push(text[start..].trim()); arguments } @@ -1525,9 +1540,9 @@ mod tests { FailureCatalog, TypedFailureBindingError, TypedFailureCause, analyze_program, analyze_task, analyze_task_with_resolver_calls, bind_failure_fact_to_resolver_call, call_span_for_identifier_use, call_span_in_statement, calls_in_expression, - contains_keyword_token, is_meaningful_failure_declaration, is_non_empty_text_literal, - is_try_candidate, parse_failure_variant, parse_try_expression, result_error_root, - split_call_arguments, + contains_keyword_token, has_stray_empty_argument, is_meaningful_failure_declaration, + is_non_empty_text_literal, is_try_candidate, parse_failure_variant, parse_try_expression, + result_error_root, split_call_arguments, }; use crate::ast::{Item, Program}; use crate::core_body::BodyStatement; @@ -1998,13 +2013,13 @@ task caller() -> Result UInt, SourceError { ); } - // WO27 Part 1a: inventory pin. Exactly three argument-splitter definitions - // exist: the canonical one here, one delegating wrapper in - // full_type_check.rs, and one in run.rs. The naive private validator also - // in this file predates Part 1a and is out of scope. Part 1b removes the - // two wrappers; this test must be updated then, not silently re-baselined. + // WO27 Part 1b: inventory pin. Exactly one argument-splitter definition + // exists: the canonical `split_call_arguments` in this file. The Part 1a + // delegating wrappers in full_type_check.rs and run.rs are removed; a + // second real splitter fails the build. The naive private validator also + // in this file predates Part 1a and is out of scope. #[test] - fn argument_splitter_inventory_pins_three_definitions() { + fn argument_splitter_inventory_pins_single_definition() { // Built at runtime so this test's own source never self-counts. let canonical_name = ["fn split", "_call_arguments"].concat(); let legacy_name = ["fn split", "_arguments"].concat(); @@ -2018,13 +2033,13 @@ task caller() -> Result UInt, SourceError { ); assert_eq!( full_type_check.matches(canonical_name.as_str()).count(), - 1, - "checker wrapper must be defined exactly once" + 0, + "checker wrapper must be gone" ); assert_eq!( run.matches(legacy_name.as_str()).count(), - 1, - "runtime wrapper must be defined exactly once" + 0, + "runtime wrapper must be gone" ); assert_eq!( typed_failure.matches(legacy_name.as_str()).count(), @@ -2033,36 +2048,42 @@ task caller() -> Result UInt, SourceError { ); } - // WO27 Part 1a: the two temporary wrappers carry no logic of their own — - // they agree with the canonical splitter on the whole corpus. + // WO27 Part 1b: the Part 1a wrappers are removed; the canonical splitter + // stands alone. This test pins the stray-empty-argument detector used by + // `text_split` (decision 0022): interior empty segments are rejected, + // while an empty argument list is not stray. #[test] - fn splitter_wrappers_agree_with_canonical() { - // Backslash-bearing inputs are built with char::from(92): the - // public-readiness scan rejects double-backslash sequences in sources. - let bs = char::from(92).to_string(); - let corpus: Vec = vec![ - "\"a,b\", \",\"".to_string(), - format!("\"a{bs}\",b\", \",\""), - format!("\"a{bs}{bs}\", \",\""), - "f(a, b), \",\"".to_string(), - "[a, b], \",\"".to_string(), - "f(g(a, b), [c, d]), \",\"".to_string(), - " text , sep ".to_string(), - "a, b, c".to_string(), - "a,,b".to_string(), - ]; - for input in &corpus { - let canonical = split_call_arguments(input); - assert_eq!( - crate::full_type_check::split_call_arguments(input), - canonical, - "checker wrapper disagrees on {input:?}" + fn stray_empty_argument_detection() { + let stray = vec!["a,,b", "a, ,b", ",a", "a,", "line,, \",\""]; + for input in &stray { + assert!( + has_stray_empty_argument(input), + "expected stray empty in {input:?}" ); - assert_eq!( - crate::run::split_arguments(input), - canonical, - "runtime wrapper disagrees on {input:?}" + } + let clean = vec!["", " ", "a", "a, b", "\"a,b\", \",\"", "f(a, b), \",\""]; + for input in &clean { + assert!( + !has_stray_empty_argument(input), + "unexpected stray empty in {input:?}" ); } + // An escaped quote never splits the argument: the first argument is + // one literal, so there is no stray empty. + let bs = char::from(92).to_string(); + let escaped = bs.clone() + "\",b\", \",\""; + let input = ["\"a", &escaped].concat(); + assert!(!has_stray_empty_argument(&input)); + } + // Decision 0022: an escaped quote never terminates a literal, so the + // scanner sees exactly two arguments here. + #[test] + fn escaped_quote_does_not_split_arguments() { + let bs = char::from(92).to_string(); + // Input: "a\"b, c", d (the comma inside the literal is not a separator) + let input = format!("\"a{bs}\"b, c\", d"); + let args = split_call_arguments(&input); + assert_eq!(args.len(), 2, "escaped quote split the arguments"); + assert_eq!(args[1], "d"); } } From 4627ec8a246af4f857ace2ebb48587c2146cd84a Mon Sep 17 00:00:00 2001 From: Ocean Bennett <204957658+undergroundrap@users.noreply.github.com> Date: Wed, 23 Sep 2026 09:50:55 +0000 Subject: [PATCH 2/4] fix: fail closed on invalid text escapes (H0638 honesty pass) Three fail-open defects in the decision-0022 implementation: 1. H0638 did not block: rejected_invalid_text_escape_v0 was missing from rejected_statements(), is_blocking_statement(), and checked_statements(), so hum full-type-check exited 0 on unknown escapes. Added to all three gate lists. 2. Trailing backslash silently accepted: projected_delimiter_completion claimed "ab\" as UnterminatedTextLiteral before projected_bad_text_escape could report InvalidTextEscape, and the unterminated completion never surfaced as a blocking error. Moved the escape check before the delimiter check in projected and retained completion events; H0638 covers trailing backslash per the diagnostic catalog. 3. Added missing evidence: trailing-backslash misuse fixture, H0638 exact-span regression tests (span marks the bad escape), and Part 1b Session AB coverage (H0638 unknown/trailing, H0636 stray empty, newline round-trip through graph JSON). Also updated the Invoke-HumCompilerCorpusChecks digest pin and improved its failure message to print expected vs found digests. --- .../text_trailing_backslash_fail.hum | 15 +++++ src/full_type_check.rs | 57 +++++++++++++++++++ src/parser.rs | 52 ++++++++++------- tools/check_all.ps1 | 15 +++++ tools/test_ci_policy.ps1 | 5 +- 5 files changed, 120 insertions(+), 24 deletions(-) create mode 100644 fixtures/diagnostics/text_trailing_backslash_fail.hum diff --git a/fixtures/diagnostics/text_trailing_backslash_fail.hum b/fixtures/diagnostics/text_trailing_backslash_fail.hum new file mode 100644 index 0000000..d07ff35 --- /dev/null +++ b/fixtures/diagnostics/text_trailing_backslash_fail.hum @@ -0,0 +1,15 @@ +module fixtures.diagnostics.text_trailing_backslash_fail + +task trailing_backslash() -> Text { + why: + misuse fixture: a trailing backslash before the closing quote is a + checker error (H0638, unterminated escape) + + cost: + time: O(1) + space: O(1) + check: warn + + does: + return "ab\" +} diff --git a/src/full_type_check.rs b/src/full_type_check.rs index 4faaf20..5607220 100644 --- a/src/full_type_check.rs +++ b/src/full_type_check.rs @@ -2008,6 +2008,7 @@ fn is_blocking_statement(statement: &TypedStatement) -> bool { | "rejected_invalid_stdout_write_call_v0" | "rejected_invalid_clock_replay_call_v0" | "rejected_invalid_files_read_text_call_v0" + | "rejected_invalid_text_escape_v0" | "unchecked_statement_type_v0" | "blocked_unsupported_statement_v0" | "not_checked_blocked_by_prior_errors_v0" @@ -2065,6 +2066,7 @@ impl FullTypeCheckReport { | "rejected_invalid_stdout_write_call_v0" | "rejected_invalid_clock_replay_call_v0" | "rejected_invalid_files_read_text_call_v0" + | "rejected_invalid_text_escape_v0" ) }) .count() @@ -2102,6 +2104,7 @@ impl FullTypeCheckReport { | "rejected_invalid_stdout_write_call_v0" | "rejected_invalid_clock_replay_call_v0" | "rejected_invalid_files_read_text_call_v0" + | "rejected_invalid_text_escape_v0" ) }) .count() @@ -3359,4 +3362,58 @@ task probe(line: Text) -> Result Unit, ProbeError { ); assert_eq!(count_diagnostic_code(&unmasked, "H0901"), 0); } + + #[test] + fn h0638_span_marks_the_bad_escape_itself() { + // Decision 0022: the diagnostic's span marks the bad escape itself + // (the backslash and its following character), not the whole literal. + // Backslash inputs built without double-backslash literals + // (public-readiness). + let bs = char::from(92).to_string(); + let source = format!( + "task bad() -> Text {{\n does:\n return {}bad{}qescape{}\n}}\n", + '"', bs, '"' + ); + let program = Program { + files: vec![parse_source("h0638_span.hum", &source).file], + }; + let report = build_report(&program, &[]); + assert!(full_type_check_has_errors(&program, &[])); + let statement = &report.items[0].statements[0]; + assert_eq!(statement.status, "rejected_invalid_text_escape_v0"); + assert_eq!( + statement.diagnostic_code, + Some(DiagnosticCode::INVALID_TEXT_ESCAPE.as_str()) + ); + // `return "bad\qescape"`: the backslash is at column 16 (1-based). + let span = statement.call_span.as_ref().expect("H0638 has a span"); + assert_eq!(span.line, 3); + assert_eq!(span.column, 16); + } + + #[test] + fn h0638_trailing_backslash_is_a_checker_error() { + // Decision 0022: a trailing backslash before the closing quote is a + // checker error (H0638, unterminated escape), not a silent accept. + let bs = char::from(92).to_string(); + let source = format!( + "task trail() -> Text {{\n does:\n return {}ab{}{}\n}}\n", + '"', bs, '"' + ); + let program = Program { + files: vec![parse_source("h0638_trailing.hum", &source).file], + }; + let report = build_report(&program, &[]); + assert!(full_type_check_has_errors(&program, &[])); + let statement = &report.items[0].statements[0]; + assert_eq!(statement.status, "rejected_invalid_text_escape_v0"); + assert_eq!( + statement.diagnostic_code, + Some(DiagnosticCode::INVALID_TEXT_ESCAPE.as_str()) + ); + // `return "ab\"`: the backslash is at column 15 (1-based). + let span = statement.call_span.as_ref().expect("H0638 has a span"); + assert_eq!(span.line, 3); + assert_eq!(span.column, 15); + } } diff --git a/src/parser.rs b/src/parser.rs index 22afdbe..550ee00 100644 --- a/src/parser.rs +++ b/src/parser.rs @@ -8186,34 +8186,39 @@ fn projected_completion_event( text: &str, span: &Span, ) -> CanonicalCompletionEvent { - if let Some(issue) = projected_delimiter_completion(text, span) { - return issue; - } - if let Some((start, len)) = projected_out_of_range_integer(text) { + // Decision 0022: a bad escape in a complete-looking literal (including a + // trailing backslash before the closing quote) is an invalid-escape + // error, not a delimiter error. The escape check runs first so the + // quote-aware delimiter scanner does not claim `"ab\"` as an + // unterminated literal; H0638 covers the trailing-backslash case. + if let Some((start, len)) = projected_bad_text_escape(text) { let offending = completion_range(span, start, len); return malformed_completion( - CanonicalMalformedCause::IntegerLiteralOutOfRange, + CanonicalMalformedCause::InvalidTextEscape, text, span, offending.clone(), - CanonicalExpectedLexicalEvidence::Int64Value, + CanonicalExpectedLexicalEvidence::TextEscape, CanonicalActualLexicalEvidence::Token { - kind: CanonicalLexicalTokenKind::IntegerLiteral, + kind: CanonicalLexicalTokenKind::Other, range: offending, spelling: text[start..start + len].to_string(), }, ); } - if let Some((start, len)) = projected_bad_text_escape(text) { + if let Some(issue) = projected_delimiter_completion(text, span) { + return issue; + } + if let Some((start, len)) = projected_out_of_range_integer(text) { let offending = completion_range(span, start, len); return malformed_completion( - CanonicalMalformedCause::InvalidTextEscape, + CanonicalMalformedCause::IntegerLiteralOutOfRange, text, span, offending.clone(), - CanonicalExpectedLexicalEvidence::TextEscape, + CanonicalExpectedLexicalEvidence::Int64Value, CanonicalActualLexicalEvidence::Token { - kind: CanonicalLexicalTokenKind::Other, + kind: CanonicalLexicalTokenKind::IntegerLiteral, range: offending, spelling: text[start..start + len].to_string(), }, @@ -8276,34 +8281,37 @@ fn retained_completion_event( text: &str, span: &Span, ) -> CanonicalCompletionEvent { - if let Some(issue) = retained_delimiter_completion(text, span) { - return issue; - } - if let Some((start, len)) = retained_out_of_range_integer(text, span) { + // Decision 0022: keep the projected/retained precedence aligned — a bad + // escape in a complete-looking literal is an invalid-escape error (H0638), + // not a delimiter error. + if let Some((start, len)) = projected_bad_text_escape(text) { let offending = completion_range(span, start, len); return malformed_completion( - CanonicalMalformedCause::IntegerLiteralOutOfRange, + CanonicalMalformedCause::InvalidTextEscape, text, span, offending.clone(), - CanonicalExpectedLexicalEvidence::Int64Value, + CanonicalExpectedLexicalEvidence::TextEscape, CanonicalActualLexicalEvidence::Token { - kind: CanonicalLexicalTokenKind::IntegerLiteral, + kind: CanonicalLexicalTokenKind::Other, range: offending, spelling: text[start..start + len].to_string(), }, ); } - if let Some((start, len)) = projected_bad_text_escape(text) { + if let Some(issue) = retained_delimiter_completion(text, span) { + return issue; + } + if let Some((start, len)) = retained_out_of_range_integer(text, span) { let offending = completion_range(span, start, len); return malformed_completion( - CanonicalMalformedCause::InvalidTextEscape, + CanonicalMalformedCause::IntegerLiteralOutOfRange, text, span, offending.clone(), - CanonicalExpectedLexicalEvidence::TextEscape, + CanonicalExpectedLexicalEvidence::Int64Value, CanonicalActualLexicalEvidence::Token { - kind: CanonicalLexicalTokenKind::Other, + kind: CanonicalLexicalTokenKind::IntegerLiteral, range: offending, spelling: text[start..start + len].to_string(), }, diff --git a/tools/check_all.ps1 b/tools/check_all.ps1 index cd81170..395367e 100644 --- a/tools/check_all.ps1 +++ b/tools/check_all.ps1 @@ -4972,6 +4972,21 @@ function Invoke-HumCompilerCorpusChecks { if ($SessionABH0901.ExitCode -ne 1 -or [regex]::Matches($SessionABH0901.Output, '"diagnostic_code": "H0901"').Count -ne 1) { throw 'Session AB variable separator without try must fail with exactly one H0901' } Assert-Json 'full-type-check Session AB variable separator JSON' $SessionABH0901.Output + # Session AB Part 1b: decision 0022 text-literal escapes (WO27 Part 1b). + # Unknown escapes and trailing backslashes are checker errors (H0638); + # stray empty text_split arguments are checker errors (H0636). + $SessionABH0638 = Read-NativeOutputWithExit 'full-type-check Session AB unknown escape' $Hum @('full-type-check', 'fixtures/diagnostics/text_invalid_escape_fail.hum') + if ($SessionABH0638.ExitCode -ne 1 -or [regex]::Matches($SessionABH0638.Output, 'diagnostic=H0638').Count -ne 1) { throw 'Session AB unknown escape must fail with exactly one H0638' } + $SessionABH0638Trail = Read-NativeOutputWithExit 'full-type-check Session AB trailing backslash' $Hum @('full-type-check', 'fixtures/diagnostics/text_trailing_backslash_fail.hum') + if ($SessionABH0638Trail.ExitCode -ne 1 -or [regex]::Matches($SessionABH0638Trail.Output, 'diagnostic=H0638').Count -ne 1) { throw 'Session AB trailing backslash must fail with exactly one H0638' } + $SessionABH0636Stray = Read-NativeOutputWithExit 'full-type-check Session AB stray empty argument' $Hum @('full-type-check', 'fixtures/diagnostics/text_split_stray_empty_argument_fail.hum') + if ($SessionABH0636Stray.ExitCode -ne 1 -or [regex]::Matches($SessionABH0636Stray.Output, 'diagnostic=H0636').Count -ne 1) { throw 'Session AB stray empty argument must fail with exactly one H0636' } + # Newline round trip: the \n literal in the decode fixture must survive + # the graph JSON emitter as an escaped \n, never a raw line break. + $SessionABGraphEscapes = Read-NativeOutput 'graph Session AB text escapes decode' $Hum @('graph', 'fixtures/text_escapes_decode.hum') + Assert-Json 'graph Session AB text escapes decode' $SessionABGraphEscapes + if (-not $SessionABGraphEscapes.Contains('a\\nb')) { throw 'Session AB graph output must contain the escaped newline literal' } + $SessionAAPositive = 'examples/probes/runner_replay_clock.hum' foreach ($Command in @('resolve', 'full-type-check', 'effect-check', 'ownership-check', 'resource-check', 'core-preview', 'core-lower', 'core-verify')) { $Surface = Read-NativeOutput "Session AA $Command positive" $Hum @($Command, '--format', 'json', $SessionAAPositive) diff --git a/tools/test_ci_policy.ps1 b/tools/test_ci_policy.ps1 index a58e3e1..5b03488 100644 --- a/tools/test_ci_policy.ps1 +++ b/tools/test_ci_policy.ps1 @@ -583,7 +583,7 @@ foreach($Name in @('Invoke-HumCoreCheck','Invoke-HumRuntimeProgramChecks','Invok # They are source-closure evidence, never a claim that the full corpus ran. $CompilerBodies=@{ 'Invoke-HumCompilerFrontChecks'='fa10b6d816d601b3dfc848fd68098c1a91be57a3853d88c08e5cce95f9fbcae8' - 'Invoke-HumCompilerCorpusChecks'='30c164d7d13c8cd003a59627160b743179a07c19b2c91742380b65862628038f' + 'Invoke-HumCompilerCorpusChecks'='e6ee4f3daa04fd591c34911cc6d652f94154d1c06b1629997992532c7ac63e1a' 'Invoke-HumUseAfterMoveRuntimeCheck'='a7c5db7146519cba950ec4ba2de9a0f15bf505bbfde0bc855b2e49c9a74a34f8' 'Invoke-HumUseAfterMoveProjectionCheck'='d1708df1234a0cbdf4f686d48facad32ccfb2bcc7baa834281d2246b748f41f0' } @@ -595,7 +595,8 @@ foreach($Name in $CompilerBodies.Keys){ $Hasher=[Security.Cryptography.SHA256]::Create() try { $Digest=-join($Hasher.ComputeHash([Text.Encoding]::UTF8.GetBytes($Body))|ForEach-Object{$_.ToString('x2')}) - Assert-Policy ($Digest -ceq $CompilerBodies[$Name]) "complete shared body $Name" + $ExpectedDigest = $CompilerBodies[$Name] + Assert-Policy ($Digest -ceq $ExpectedDigest) "complete shared body ${Name}: expected ${ExpectedDigest}, found ${Digest}" $First=@($Function.Body.EndBlock.Statements)[0].Extent.Text $Omitted=$Body.Replace($First,'') $BadDigest=-join($Hasher.ComputeHash([Text.Encoding]::UTF8.GetBytes($Omitted))|ForEach-Object{$_.ToString('x2')}) From 2a13d587493addaf2d7de3b5db2c5ef35565788b Mon Sep 17 00:00:00 2001 From: Ocean Bennett <204957658+undergroundrap@users.noreply.github.com> Date: Wed, 23 Sep 2026 09:51:00 +0000 Subject: [PATCH 3/4] fix: update diagnostic catalog count pin to 93 (H0638) The original Part 1b commit added H0638 to the diagnostic catalog without updating the hardcoded 92-code pin in check_all.ps1. The Fast profile failed at the catalog assertion. Updated to 93. --- tools/check_all.ps1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/check_all.ps1 b/tools/check_all.ps1 index 395367e..90ab71d 100644 --- a/tools/check_all.ps1 +++ b/tools/check_all.ps1 @@ -2804,7 +2804,7 @@ task malformed() -> UInt { Assert-Json 'diagnostic catalog JSON' $DiagnosticsJson $DiagnosticsCatalog = $DiagnosticsJson | ConvertFrom-Json $DiagnosticCodes = @($DiagnosticsCatalog.diagnostics | ForEach-Object { $_.code }) - if ($DiagnosticsCatalog.count -ne 92 -or $DiagnosticCodes.Count -ne 92 -or @($DiagnosticCodes | Sort-Object -Unique).Count -ne 92) { throw 'canonical diagnostic catalog must expose exactly 92 unique active codes' } + if ($DiagnosticsCatalog.count -ne 93 -or $DiagnosticCodes.Count -ne 93 -or @($DiagnosticCodes | Sort-Object -Unique).Count -ne 93) { throw 'canonical diagnostic catalog must expose exactly 93 unique active codes' } $H0634CatalogRows = @($DiagnosticsCatalog.diagnostics | Where-Object { $_.code -ceq 'H0634' -and $_.title -ceq 'canonical native program layout' }) if ($H0634CatalogRows.Count -ne 1) { throw 'Work Order 23 H0634 catalog projection drifted' } $H0635CatalogRows = @($DiagnosticsCatalog.diagnostics | Where-Object { $_.code -ceq 'H0635' -and $_.title -ceq 'unsupported native program feature' }) From 512e1f3b990265c84bacae1c074bca3322028f4a Mon Sep 17 00:00:00 2001 From: Ocean Bennett <204957658+undergroundrap@users.noreply.github.com> Date: Wed, 23 Sep 2026 09:51:04 +0000 Subject: [PATCH 4/4] fix: update Invoke-HumCompilerFrontChecks digest pin (93-code catalog) The 92->93 diagnostic catalog count fix modified the Invoke-HumCompilerFrontChecks function body. Updated its SHA-256 pin. --- tools/test_ci_policy.ps1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/test_ci_policy.ps1 b/tools/test_ci_policy.ps1 index 5b03488..610e90c 100644 --- a/tools/test_ci_policy.ps1 +++ b/tools/test_ci_policy.ps1 @@ -582,7 +582,7 @@ foreach($Name in @('Invoke-HumCoreCheck','Invoke-HumRuntimeProgramChecks','Invok # These pins protect the mechanically shared bodies from silent omissions. # They are source-closure evidence, never a claim that the full corpus ran. $CompilerBodies=@{ - 'Invoke-HumCompilerFrontChecks'='fa10b6d816d601b3dfc848fd68098c1a91be57a3853d88c08e5cce95f9fbcae8' + 'Invoke-HumCompilerFrontChecks'='38b665cf68b1f62b350e950ca2d4d97565e81323cf78d97a6d5b99eff5b1ad1c' 'Invoke-HumCompilerCorpusChecks'='e6ee4f3daa04fd591c34911cc6d652f94154d1c06b1629997992532c7ac63e1a' 'Invoke-HumUseAfterMoveRuntimeCheck'='a7c5db7146519cba950ec4ba2de9a0f15bf505bbfde0bc855b2e49c9a74a34f8' 'Invoke-HumUseAfterMoveProjectionCheck'='d1708df1234a0cbdf4f686d48facad32ccfb2bcc7baa834281d2246b748f41f0'