Skip to content

Release 3.3.0 - #714

Merged
kaby76 merged 14 commits into
mainfrom
trash-release-3.3.0
Sep 6, 2026
Merged

Release 3.3.0#714
kaby76 merged 14 commits into
mainfrom
trash-release-3.3.0

Conversation

@kaby76

@kaby76 kaby76 commented Sep 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

Release 3.3.0 focuses on substantially improving the performance, memory use, diagnostics, and benchmark accuracy of the interpreted ALL(*) parser used by trparse.

The release establishes repeatable parser-only performance tests, instruments prediction and runtime stages, reuses parser state across files, replaces allocation-heavy parser data structures, and builds parsing results in compact storage. It also corrects interpreted token reporting and makes OphiRust benchmark timing include its eager lexing work.

ALL(*) parser performance

  • Added parser-only DOT benchmarks, including the large grammars-v4 1864.dot input.
  • Added five-sample timing reports with mean, standard deviation, and SEM.
  • Added --no-output so parsing and tree construction can be measured without serializing output.
  • Added --parser-stats diagnostics for prediction calls, DFA activity, lookahead, closure work, prediction contexts, conflicts, full-context fallback, committed ATN walking, parse events, and retained cache storage.
  • Shared learned parser DFAs across inputs in one command, with configurable state/memory budgets and an option to disable sharing.
  • Added a conservative LL(1) bypass implementation and supporting analysis; the production bypass remains disabled because measurements did not show a reliable improvement and nullable-loop expansion caused correctness regressions.
  • Interned prediction contexts in a grammar-scoped arena using compact integer IDs.
  • Replaced recursive SortedDictionary context merging with reusable linear-merge workspaces and prediction-local symmetric pair memoization.
  • Optimized committed ATN walking with precomputed metadata and reduced repeated state/transition inspection.
  • Cached parsed/deserialized parser and lexer .interp runtime data across all files supplied to a single trparse command. This applies to ALL(*), Earley, -x, --xf, and multiple positional inputs.

Compact parsing results

  • Added CompactParseTree, which stores node kinds, relationships, token references, and rule metadata in compact indexed arrays.
  • Builds interpreted parse trees directly in compact storage instead of constructing the former allocation-heavy DOM representation.
  • Preserves the existing parsing-result interfaces and serialized output.
  • Added TokenStore, a structure-of-arrays representation for interpreted lexer tokens.
  • Changed LexerToken to an indexed value-type view while preserving standalone-token compatibility.
  • Keeps token text source-backed and serializes text directly from source spans, avoiding a string allocation per token.
  • Updated ALL(*), Earley, and tree construction paths to consume the compact token store without copying token objects.

On the large DOT workload, compact tree construction and token storage produced the largest measured improvement in this release. One measured 1864.dot run improved from 2.656 seconds/312,894 TPS to 2.104 seconds/394,883 TPS, while a subsequent corpus comparison measured AllStar at approximately 308,221 TPS versus approximately 123,000 TPS before compact tree storage.

Correctness and reporting

  • Interpreted ALL(*) and Earley runners now report the complete retained token stream, including hidden-channel tokens and EOF, matching generated-target CommonTokenStream.Size reporting.
  • Added a DOT cluster.dot regression test verifying the expected 171-token count.
  • Parsing-result serialization now supports compact node providers.
  • Added coverage for compact tree behavior, token storage, prediction-context canonicalization/merging, LL(1) analysis, runtime reuse, parser diagnostics, and output suppression.

Benchmark fairness

OphiRust CommonTokenStream::new(lexer) eagerly fills the token buffer. Its generated template previously started the parser timer afterward, excluding lexing from PT while targets with lazy token streams included lexing during the start-rule call.

The OphiRust template now measures eager token-buffer construction and adds that duration to PT/TPS. Parser construction and listener setup remain outside the measured interval.

Validation

  • Release builds complete with zero warnings and errors.
  • The broad applicable AllStar non-performance regression suite passes (282 tests in the final token-count verification run).
  • The exact DOT cluster.dot command now reports 171 tokens for ALL(*), matching the generated targets.
  • Tag: 3.3.0

Tracking issue: #713.

@kaby76
kaby76 marked this pull request as ready for review September 6, 2026 15:56
Copilot AI lite review requested due to automatic review settings September 6, 2026 15:56
@kaby76
kaby76 merged commit f7f2184 into main Sep 6, 2026
2 checks passed
Copilot AI reviewed Sep 6, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are confirmed build-breaking issues (definite assignment / API usage) and new tests that currently invoke the CLI incorrectly, so CI is expected to fail until these are fixed.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Release bump to 3.3.0 across the Trash toolchain, along with significant interpreter/performance work for trparse (ALL(*) diagnostics, shared parser DFA/prediction-context caching, compact parse-tree output) and new/expanded test coverage around those features.

Changes:

  • Bump versions/readmes/scripts/tool-manifest from 3.2.0 → 3.3.0 across many sub-tools.
  • Enhance trparse ALL(*) interpreter: --no-output, --parser-stats, shared parser-DFA caching controls, runtime caching of .interp loads, and more detailed stage timings.
  • Introduce compact storage/output paths (e.g., TokenStore, CompactParseTree via ParsingResultSet.NodeProvider) and add targeted tests/perf baselines.
File summaries
File Description
tests/AllStarParserTests/TrparseOutputTests.cs New CLI-level tests for --no-output, parser stats, shared DFA reuse (currently missing parse subcommand).
tests/AllStarParserTests/TokenStoreTests.cs Tests for TokenStore storage/view behavior and explicit-text materialization.
tests/AllStarParserTests/TestData/dot/DOTParser.interp Adds DOT parser .interp test data to support DOT perf tests.
tests/AllStarParserTests/PredictionContextArenaTests.cs Tests for prediction-context interning/canonicalization and deep chains.
tests/AllStarParserTests/Ll1DecisionAnalyzerTests.cs Tests for conservative LL(1) decision eligibility/prediction behavior.
tests/AllStarParserTests/DotParserPerformanceTests.cs Parser performance baselines for DOT inputs and caching scenarios.
tests/AllStarParserTests/DotLexerPerformanceTests.cs Updates lexer perf expectations + adds timings/runtime-cache/token-count tests.
tests/AllStarParserTests/DomBuilderTests.cs Verifies compact tree serialization parity + lazy DOM materialization.
tests/AllStarParserTests/AllStarParserTests.csproj Ensures DOT parser .interp is copied to test output.
src/trxquery/trxquery.csproj Version bump to 3.3.0.
src/trxquery/Config.cs Version bump to 3.3.0.
src/trxpath/trxpath.csproj Version bump to 3.3.0.
src/trxpath/readme.md Version bump to 3.3.0.
src/trxpath/Config.cs Version bump to 3.3.0.
src/trxml2/trxml2.csproj Version bump to 3.3.0.
src/trxml2/readme.md Version bump to 3.3.0.
src/trxml2/Config.cs Version bump to 3.3.0.
src/trxml/trxml.csproj Version bump to 3.3.0.
src/trxml/readme.md Version bump to 3.3.0.
src/trxml/Config.cs Version bump to 3.3.0.
src/trwdog/trwdog.csproj Version bump to 3.3.0.
src/trwdog/readme.md Version bump to 3.3.0.
src/trwdog/Config.cs Version bump to 3.3.0.
src/trunfoldlit/trunfoldlit.csproj Version bump to 3.3.0.
src/trunfoldlit/readme.md Version bump to 3.3.0.
src/trunfoldlit/Config.cs Version bump to 3.3.0.
src/trunfold/trunfold.csproj Version bump to 3.3.0.
src/trunfold/readme.md Version bump to 3.3.0.
src/trunfold/Config.cs Version bump to 3.3.0.
src/trull/trull.csproj Version bump to 3.3.0.
src/trull/readme.md Version bump to 3.3.0.
src/trull/Config.cs Version bump to 3.3.0.
src/trtree/trtree.csproj Version bump to 3.3.0.
src/trtree/readme.md Version bump to 3.3.0.
src/trtree/Config.cs Version bump to 3.3.0.
src/trtokens/trtokens.csproj Version bump to 3.3.0.
src/trtokens/readme.md Version bump to 3.3.0.
src/trtokens/Config.cs Version bump to 3.3.0.
src/trthompson/trthompson.csproj Version bump to 3.3.0.
src/trthompson/readme.md Version bump to 3.3.0.
src/trthompson/Config.cs Version bump to 3.3.0.
src/trtext/trtext.csproj Version bump to 3.3.0.
src/trtext/readme.md Version bump to 3.3.0.
src/trtext/Config.cs Version bump to 3.3.0.
src/trsponge/trsponge.csproj Version bump to 3.3.0.
src/trsponge/readme.md Version bump to 3.3.0.
src/trsponge/Config.cs Version bump to 3.3.0.
src/trsplit/trsplit.csproj Version bump to 3.3.0.
src/trsplit/readme.md Version bump to 3.3.0.
src/trsplit/Config.cs Version bump to 3.3.0.
src/trsort/trsort.csproj Version bump to 3.3.0.
src/trsort/readme.md Version bump to 3.3.0.
src/trsort/Config.cs Version bump to 3.3.0.
src/trsem/trsem.csproj Version bump to 3.3.0.
src/trsem/readme.md Version bump to 3.3.0.
src/trsem/Config.cs Version bump to 3.3.0.
src/trrup/trrup.csproj Version bump to 3.3.0.
src/trrup/readme.md Version bump to 3.3.0.
src/trrup/Config.cs Version bump to 3.3.0.
src/trrr/trrr.csproj Version bump to 3.3.0.
src/trrr/readme.md Version bump to 3.3.0.
src/trrr/Config.cs Version bump to 3.3.0.
src/trrename/trrename.csproj Version bump to 3.3.0.
src/trrename/readme.md Version bump to 3.3.0.
src/trrename/Config.cs Version bump to 3.3.0.
src/trpiggy/trpiggy.csproj Version bump to 3.3.0.
src/trpiggy/readme.md Version bump to 3.3.0.
src/trpiggy/Config.cs Version bump to 3.3.0.
src/trperf/trperf.csproj Version bump to 3.3.0.
src/trperf/readme.md Version bump to 3.3.0.
src/trperf/Config.cs Version bump to 3.3.0.
src/trparse/trparse.csproj Version bump to 3.3.0 (and exposes internals to test project).
src/trparse/readme.md Documents new parser stats/shared DFA/timings/no-output flags + version bump.
src/trparse/Properties/launchSettings.json Updates local debug args to exercise new ALL(*) options.
src/trparse/Grun.cs Wires new interp runtime cache, parser stats, shared DFA cache, no-output, aggregated timings.
src/trparse/EarleyAtnParser/LexerAtnSimulator.cs Switches tokenization output to TokenStore.
src/trparse/EarleyAtnParser/InterpRunner.cs Adds InterpRuntimeCache reuse; updates token counting semantics.
src/trparse/Config.cs Adds --no-output, --parser-stats, shared DFA cache flags; version bump.
src/trparse/Atn/TokenStore.cs New compact token storage abstraction.
src/trparse/Atn/LexerToken.cs Converts token to struct + store-backed view semantics.
src/trparse/Atn/CompactParseTree.cs New compact parse tree + builder and JSON node writer/materializer.
src/trparse/AllStarAtnParser/PredictionContext.cs Adds context IDs/arena, merge workspace, and arena interning logic.
src/trparse/AllStarAtnParser/ParserStatistics.cs New opt-in parser stats counters/formatting.
src/trparse/AllStarAtnParser/ParserPredictionCache.cs New shared learned parser DFA + prediction-context cache with budgets.
src/trparse/AllStarAtnParser/Ll1DecisionAnalyzer.cs New conservative LL(1) decision analyzer (currently mostly test/perf).
src/trparse/AllStarAtnParser/InterpRunTimings.cs Adds files aggregation + result construction timing.
src/trparse/AllStarAtnParser/InterpRuntimeCache.cs New command-scoped cache for parsed interp + deserialized ATNs/vocabs.
src/trparse/AllStarAtnParser/InterpRunner.cs Integrates caches/stats, builds compact tree provider, updates token counting.
src/trparse/AllStarAtnParser/CommittedAtnMetadata.cs Adds committed-ATN metadata cache for faster committed traversal.
src/trparse/AllStarAtnParser/ATNConfigSet.cs Tracks stats + uses arena/workspace merging keyed by compact IDs.
src/trparse/AllStarAtnParser/AllStarSimulator.cs Adds stats hooks + shared DFA/context caching + retention/budget logic.
src/trparse/AllStarAtnParser/AllStarParser.cs Adds recognition-only entrypoints, committed-metadata usage, cache/stats integration.
src/trnullable/trnullable.csproj Version bump to 3.3.0.
src/trnullable/readme.md Version bump to 3.3.0.
src/trnullable/Config.cs Version bump to 3.3.0.
src/trjson/trjson.csproj Version bump to 3.3.0.
src/trjson/readme.md Version bump to 3.3.0.
src/trjson/Config.cs Version bump to 3.3.0.
src/tritext/tritext.csproj Version bump to 3.3.0.
src/tritext/readme.md Version bump to 3.3.0.
src/tritext/Config.cs Version bump to 3.3.0.
src/trinterp/trinterp.csproj Version bump to 3.3.0.
src/trinterp/readme.md Version bump to 3.3.0.
src/trinterp/Config.cs Version bump to 3.3.0.
src/triconv/triconv.csproj Version bump to 3.3.0.
src/triconv/readme.md Version bump to 3.3.0.
src/triconv/Config.cs Version bump to 3.3.0.
src/trgroup/trgroup.csproj Version bump to 3.3.0.
src/trgroup/readme.md Version bump to 3.3.0.
src/trgroup/Config.cs Version bump to 3.3.0.
src/trglob/trglob.csproj Version bump to 3.3.0.
src/trglob/readme.md Version bump to 3.3.0.
src/trglob/Config.cs Version bump to 3.3.0.
src/trgenvsc/trgenvsc.csproj Version bump to 3.3.0.
src/trgenvsc/readme.md Version bump to 3.3.0.
src/trgenvsc/Config.cs Version bump to 3.3.0.
src/trgen/trgen.csproj Version bump to 3.3.0.
src/trgen/templates/OphiRust/src/st.main.rs Adjusts parse-time measurement to include eager token buffering.
src/trgen/readme.md Version bump to 3.3.0.
src/trgen/Config.cs Version bump to 3.3.0.
src/trgen/Command.cs Version bump to 3.3.0.
src/trformat/trformat.csproj Version bump to 3.3.0.
src/trformat/readme.md Version bump to 3.3.0.
src/trformat/Config.cs Version bump to 3.3.0.
src/trfoldlit/trfoldlit.csproj Version bump to 3.3.0.
src/trfoldlit/readme.md Version bump to 3.3.0.
src/trfoldlit/Config.cs Version bump to 3.3.0.
src/trfold/trfold.csproj Version bump to 3.3.0.
src/trfold/readme.md Version bump to 3.3.0.
src/trfold/Config.cs Version bump to 3.3.0.
src/trff/trff.csproj Version bump to 3.3.0.
src/trff/readme.md Version bump to 3.3.0.
src/trff/Config.cs Version bump to 3.3.0.
src/trextract/trextract.csproj Version bump to 3.3.0.
src/trextract/readme.md Version bump to 3.3.0.
src/trextract/Config.cs Version bump to 3.3.0.
src/trdot/trdot.csproj Version bump to 3.3.0.
src/trdot/readme.md Version bump to 3.3.0.
src/trdot/Config.cs Version bump to 3.3.0.
src/trcover/trcover.csproj Version bump to 3.3.0.
src/trcover/readme.md Version bump to 3.3.0.
src/trcover/Config.cs Version bump to 3.3.0.
src/trconvert/trconvert.csproj Version bump to 3.3.0.
src/trconvert/readme.md Version bump to 3.3.0.
src/trconvert/Config.cs Version bump to 3.3.0.
src/trcombine/trcombine.csproj Version bump to 3.3.0.
src/trcombine/readme.md Version bump to 3.3.0.
src/trcombine/Config.cs Version bump to 3.3.0.
src/trclonereplace/trclonereplace.csproj Version bump to 3.3.0.
src/trclonereplace/readme.md Version bump to 3.3.0.
src/trclonereplace/Config.cs Version bump to 3.3.0.
src/trcaret/trcaret.csproj Version bump to 3.3.0.
src/trcaret/readme.md Version bump to 3.3.0.
src/trcaret/Config.cs Version bump to 3.3.0.
src/trash/trash.csproj Version bump to 3.3.0.
src/trash/readme.md Version bump to 3.3.0.
src/trash/Program.cs Version bump to 3.3.0.
src/tranalyze/tranalyze.csproj Version bump to 3.3.0.
src/tranalyze/readme.md Version bump to 3.3.0.
src/tranalyze/Config.cs Version bump to 3.3.0.
src/tragl/tragl.csproj Version bump to 3.3.0.
src/tragl/readme.md Version bump to 3.3.0.
src/AntlrJson/ParsingResultSetSerializer.cs Serializes NodeProvider without forcing DOM materialization.
src/AntlrJson/ParsingResultSet.cs Adds NodeProvider + lazy Nodes materialization support.
scripts/set-version.sh Updates release version constant.
scripts/publish.sh Updates release version constant.
.config/dotnet-tools.json Updates tool manifest version to 3.3.0.
Review details
  • Files reviewed: 167/167 changed files
  • Comments generated: 5
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/trparse/AllStarAtnParser/PredictionContext.cs
Comment on lines +392 to +400
public void Clear()
{
_singletons.Clear();
Array.Clear(_singletonHot);
_arrays.Clear();
_nextId = 1;
Creations = 0;
Hits = 0;
}
Comment thread src/trparse/Grun.cs
private double _firstFileParseSeconds;
private int _fileCount;
private readonly List<BundleParse> _bundleParses = new();
private readonly AllStarAtnParser.ParserPredictionCache _parserPredictionCache;
Comment thread tests/AllStarParserTests/TrparseOutputTests.cs
Comment on lines +21 to +24
startInfo.ArgumentList.Add(trparse);
startInfo.ArgumentList.Add("--allstar");
startInfo.ArgumentList.Add("--no-output");
startInfo.ArgumentList.Add("--parser-stats");
Comment thread tests/AllStarParserTests/TrparseOutputTests.cs
Comment on lines +62 to +66
startInfo.ArgumentList.Add(typeof(Trash.Program).Assembly.Location);
startInfo.ArgumentList.Add("--allstar");
startInfo.ArgumentList.Add("--no-output");
startInfo.ArgumentList.Add("--parser-stats");
startInfo.ArgumentList.Add("--interp-timings");
Comment thread src/trparse/AllStarAtnParser/AllStarSimulator.cs
Comment on lines +651 to 657
private readonly HashSet<(int state, int alt, int context, int precedence)> _items;
private readonly int _hash;

public ConfigSetKey(ATNConfigSet configs)
{
_items = new HashSet<(int, int, PredictionContext, int)>();
_items = new HashSet<(int, int, int, int)>();
int hash = 0;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kaby76