Repository navigation
feat(model): opt-in syntax audit of original five-language benchmark references - #181
Merged
Merged
Conversation
…erated-code quality
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Before interpreting real model results, the fixed public benchmark's own reference completions should have a reproducible syntax-only validation path. Existing strict-string equality doesn't establish whether the target reference code is syntactically accepted by compilers.
Changes
model/train/audit_reference_syntax.pywith a source-free planning default and explicit--run --approve-local-compilersgate. It reads ONLY the existing 30 fixed original public references (six each Rust, C, C++, Python and TypeScript), reconstructingprefix + expected + suffix.rustcmetadata-only, C11/C++17 syntax-only, Pythonast.parseand TypeScript strict--noEmit. Never execute compiled code, model output or user source; no external model inference, GPU framework or network call. Compilers run without shell, with bounded time, ephemeral sources, restricted environment and discarded diagnostics. Missing compilers are counted asunavailableand the run fails closed rather than claiming full coverage.npm run model:reference:syntax; document purpose, manual approval and boundaries in training and evaluation docs.Important limitations
This checks benchmark reference syntax only, not actual model completions, compiler/version parity, runtime correctness, candidate code safety, training rights, editor acceptance or release readiness. No model weights or extension versions are published. Merge only after full six-check CI and VSIX packaging/install/upgrade validation.