Skip to content

feat(model): opt-in syntax audit of original five-language benchmark references - #181

Merged
sanskarIN merged 6 commits into
mainfrom
feat/model-reference-syntax-audit-20261030
Oct 30, 2026
Merged

sanskarIN merged 6 commits into
mainfrom
feat/model-reference-syntax-audit-20261030

Conversation

@sanskarIN

Copy link
Copy Markdown
Owner

Why

Before interpreting real model results, the fixed public benchmark's own reference completions should have a reproducible syntax-only validation path. Existing strict-string equality doesn't establish whether the target reference code is syntactically accepted by compilers.

Changes

  • Add model/train/audit_reference_syntax.py with a source-free planning default and explicit --run --approve-local-compilers gate. It reads ONLY the existing 30 fixed original public references (six each Rust, C, C++, Python and TypeScript), reconstructing prefix + expected + suffix.
  • For explicit runs, use local rustc metadata-only, C11/C++17 syntax-only, Python ast.parse and TypeScript strict --noEmit. Never execute compiled code, model output or user source; no external model inference, GPU framework or network call. Compilers run without shell, with bounded time, ephemeral sources, restricted environment and discarded diagnostics. Missing compilers are counted as unavailable and the run fails closed rather than claiming full coverage.
  • Produce only a small aggregate JSON with per-language passed/failed/unavailable totals and reproducible public fixture fingerprint; raw source, compiler errors and temporary paths are not in the report.
  • Add pure standard-library tests with injected fake compiler runners, no ML or real compilers required by CI. Include an approved Python AST-only CLI test.
  • Expose npm run model:reference:syntax; document purpose, manual approval and boundaries in training and evaluation docs.

Important limitations

This checks benchmark reference syntax only, not actual model completions, compiler/version parity, runtime correctness, candidate code safety, training rights, editor acceptance or release readiness. No model weights or extension versions are published. Merge only after full six-check CI and VSIX packaging/install/upgrade validation.

@sanskarIN
sanskarIN merged commit 2154bcb into main Oct 30, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant