feat(extract): add R language extractor (regex, no tree-sitter grammar exists)#2065
Open
mkazbekov wants to merge 1 commit into
Open
feat(extract): add R language extractor (regex, no tree-sitter grammar exists)#2065mkazbekov wants to merge 1 commit into
mkazbekov wants to merge 1 commit into
Conversation
…r exists) .r/.R was in CODE_EXTENSIONS but had no AST extractor (Graphify-Labs#1689 warning fires on every R corpus). tree-sitter-r isn't on PyPI, so this follows apex.py's precedent: a regex extractor over a comment/string-masked copy of the source, brace-matched for correct function nesting. extract_r (extractors/r.py) emits function definitions (including backtick- quoted infix operators like `%||%`), calls, library()/require()/pkg:: imports, and source() edges — including unrolling the common `for (mod in files) source(paste0(mod, ".R"))` bootstrap loop into individual per-file edges, which a literal-string source() regex alone would miss. R has one flat namespace once every file is source()'d into .GlobalEnv, so a single-file extractor can't tell a cross-file call from a typo. extract_r resolves same-file calls and stashes a bare-name hint on the rest; _resolve_r_bare_calls (extractors/resolution.py), wired into extract() the same way _resolve_cross_file_imports is for Python, does the corpus-wide lookup afterward and leaves ambiguous (>1 candidate) names dangling rather than guessing. _symbol_safe_name moves to base.py (shared, not R-specific) because an all-symbol identifier normalizes to the empty string in make_id, which silently collapsed a `%op%` operator node onto its own file's node id and produced a self-loop `contains` edge. Validated against a 15-file, ~150-function real-world R pipeline: 97% symbol recall against a prior LLM-extracted graph baseline, 0 false-positive nodes, 0 self-loops, all 13 edges of a dynamic source() loop recovered correctly. Tests: replaced the two tests that pinned .r/.R as "no extractor exists" with a synthetic-extension fixture (so they stay valid regardless of which language gains an extractor next) and added a dedicated R extraction test covering functions, cross-file calls, imports, and sources. Full suite: same 54 pre-existing environment failures as HEAD (tree-sitter-hcl unavailable, Windows watch/repo-root path handling, shallow-clone git-log fixture) on both branches; net +1 passing test, 0 regressions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.r/.Ris inCODE_EXTENSIONSbut has no AST extractor, so R files silently contribute nothing and trip the R (.r/.R) listed in CODE_EXTENSIONS but has no AST extractor — silently drops all R files with zero warning #1689 warning on every run.tree-sitter-risn't published on PyPI, so this followsapex.py's precedent: a regex extractor over a comment/string-masked copy of the source, brace-matched so nested closures scope correctly.extract_r(graphify/extractors/r.py) emits function definitions (including backtick-quoted infix operators like`%||%`), calls,library()/require()/pkg::imports, andsource()edges — including unrolling the commonfor (mod in files) source(paste0(mod, ".R"))bootstrap loop into individual per-file edges, which a literal-stringsource()regex alone would miss.source()'d into.GlobalEnv, so a single-file extractor can't tell a cross-file call from a typo.extract_rresolves same-file calls and stashes a bare-name hint on the rest;_resolve_r_bare_calls(graphify/extractors/resolution.py), wired intoextract()the same way_resolve_cross_file_importsis for Python, does the corpus-wide lookup afterward and leaves ambiguous (>1 candidate) names dangling rather than guessing — never inventing an edge._symbol_safe_namemoves tobase.py(shared, not R-specific) because an all-symbol identifier normalizes to the empty string inmake_id, which silently collapsed a%op%operator node onto its own file's node id and produced a self-loopcontainsedge. Fixed at the shared layer since any future backtick-identifier language hits the same bug.Validation
Ran against a real 15-file, ~150-function R analysis pipeline (not a synthetic fixture):
sourcesedges — all 13 edges of a dynamicsource()loop recovered correctlycallsedge audited by hand: genuine external package calls (Biobase::,ggplot2::, base-Rstrrep/setNames/etc.), zero false negativesTwo existing tests pinned
.r/.Ras "no extractor exists" as their fixture (test_extract_warns_on_code_files_with_no_ast_extractor,test_extract_progress_final_line_uses_consistent_denominator) — closing the gap makes both fail by design. Repointed both at a synthetic.zzznolangextension so they keep testing the general no-extractor warning path without pinning it to a specific language, and addedtest_extract_r_extracts_functions_calls_and_sourcesto lock in the new behavior (functions, cross-file call resolution, imports, sources).Full suite, this branch vs. clean HEAD: identical 54 pre-existing failures on both (
tree-sitter-hclunavailable, Windows watch/repo-root path handling, a shallow-clone git-log fixture — none R-related). Net +1 passing test, 0 regressions.Test plan
pytest tests/test_extract.py -k "no_ast_extractor or extract_r_extracts or no_warning_when_all_code or progress_final_line"— all pass.rdispatch placement/_DISPATCHordering convention is acceptable (placed next to.jl, the closest thematic neighbor — bespoke single-extension scripting extractor, no interop family)