Skip to content

feat(python): integrate the in-process Tree-sitter UAST parser - #163

Draft
infinitepwn wants to merge 1 commit into
antgroup:mainfrom
infinitepwn:codex/python-tree-sitter-migration
Draft

infinitepwn wants to merge 1 commit into
antgroup:mainfrom
infinitepwn:codex/python-tree-sitter-migration

Conversation

@infinitepwn

Copy link
Copy Markdown

Summary

Adopt the new in-process Tree-sitter Python UAST parser instead of spawning the external uast4py binary.

  • Register Python with the existing shared per-file parsing pipeline.
  • Initialize Python WASM before CLI AST export/scanning, worker parsing and direct single-file preprocessing; pass the source cache through the single-file path.
  • Build/install the local Python parser and package its grammar asset.
  • Remove Python SDK download/deployment/bundle entries and obsolete Python test SDK flags. The shared SDK option remains available for Go.
  • Adapt the existing FastAPI/Tornado tests to the asynchronous initialization lifecycle, preserving their cases and assertions.
  • Do not change Python checkers, source/sink rules or the analysis core. No C extensions or new benchmark/documentation artifacts are included.

The lockfile also restores 25 missing optional-platform entries for the already-pinned esbuild 0.27.5, required for a clean npm ci; existing dependency versions are unchanged.

Dependency / why this is a draft

Depends on antgroup/YASA-UAST#82.

The new Python parser's npm release version has not been selected in this PR. For coordinated local review, the dependency currently remains file:../uast/parser-Python, with the two repositories as siblings. This is not yet suitable for a standalone Engine checkout or the existing upstream CI.

Before marking this ready for merge:

  • Merge/release the new UAST Python npm package.
  • Replace the local file dependency with the actual published version, update the lockfile, and remove the sibling-source build requirement.
  • Run the upstream standalone CI/release checks against that published package.

No npm package or release workflow has been published/executed by this PR.

Validation

Validated locally on macOS ARM64 with the paired UAST checkout:

  • Clean npm ci --ignore-scripts; TypeScript --noEmit and compilation.
  • 32/32 existing FastAPI/Tornado checks after lifecycle adaptation and legacy SDK flag removal.
  • Two-worker scanning of the 12 existing no-init dispatch fixtures: 7 expected findings.
  • macOS ARM64 pkg executable: 12 valid ASTs and the same 7 findings.
  • xAST Python AST export: 800 source files, 758 valid ASTs and 42 zero-byte files skipped by the existing shared Engine filter; no invalid AST JSON.
  • Release YAML syntax and job dependency validation.

The full multi-platform build, all other-language benchmark suites and GitHub release execution were not run. The current zero-byte-file filtering behavior is unchanged by this PR.

Initialize Python WASM across CLI, workers and single-file analysis, reuse
the shared parser pipeline, and remove obsolete Python binary deployment.
Adapt existing Python tests without changing their expected findings.

Keep the local UAST dependency for coordinated draft review pending the
new Python parser npm release.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant