Skip to content

feat(extract): add R language extractor (regex, no tree-sitter grammar exists)#2065

Open
mkazbekov wants to merge 1 commit into
Graphify-Labs:v8from
mkazbekov:feat/r-extractor
Open

feat(extract): add R language extractor (regex, no tree-sitter grammar exists)#2065
mkazbekov wants to merge 1 commit into
Graphify-Labs:v8from
mkazbekov:feat/r-extractor

Conversation

@mkazbekov

Copy link
Copy Markdown

Summary

  • .r/.R is in CODE_EXTENSIONS but has no AST extractor, so R files silently contribute nothing and trip the R (.r/.R) listed in CODE_EXTENSIONS but has no AST extractor — silently drops all R files with zero warning #1689 warning on every run. tree-sitter-r isn't published on PyPI, so this follows apex.py's precedent: a regex extractor over a comment/string-masked copy of the source, brace-matched so nested closures scope correctly.
  • extract_r (graphify/extractors/r.py) emits function definitions (including backtick-quoted infix operators like `%||%`), calls, library()/require()/pkg:: imports, and source() edges — including unrolling the common for (mod in files) source(paste0(mod, ".R")) bootstrap loop into individual per-file edges, which a literal-string source() regex alone would miss.
  • R has one flat namespace once every file is source()'d into .GlobalEnv, so a single-file extractor can't tell a cross-file call from a typo. extract_r resolves same-file calls and stashes a bare-name hint on the rest; _resolve_r_bare_calls (graphify/extractors/resolution.py), wired into extract() the same way _resolve_cross_file_imports is for Python, does the corpus-wide lookup afterward and leaves ambiguous (>1 candidate) names dangling rather than guessing — never inventing an edge.
  • _symbol_safe_name moves to base.py (shared, not R-specific) because an all-symbol identifier normalizes to the empty string in make_id, which silently collapsed a %op% operator node onto its own file's node id and produced a self-loop contains edge. Fixed at the shared layer since any future backtick-identifier language hits the same bug.

Validation

Ran against a real 15-file, ~150-function R analysis pipeline (not a synthetic fixture):

  • 97% symbol recall (138/142) against a prior LLM-extracted graph of the same codebase, 0 false-positive nodes
  • 0 self-loops, 0 dangling sources edges — all 13 edges of a dynamic source() loop recovered correctly
  • Every remaining dangling calls edge audited by hand: genuine external package calls (Biobase::, ggplot2::, base-R strrep/setNames/etc.), zero false negatives

Two existing tests pinned .r/.R as "no extractor exists" as their fixture (test_extract_warns_on_code_files_with_no_ast_extractor, test_extract_progress_final_line_uses_consistent_denominator) — closing the gap makes both fail by design. Repointed both at a synthetic .zzznolang extension so they keep testing the general no-extractor warning path without pinning it to a specific language, and added test_extract_r_extracts_functions_calls_and_sources to lock in the new behavior (functions, cross-file call resolution, imports, sources).

Full suite, this branch vs. clean HEAD: identical 54 pre-existing failures on both (tree-sitter-hcl unavailable, Windows watch/repo-root path handling, a shallow-clone git-log fixture — none R-related). Net +1 passing test, 0 regressions.

Test plan

  • pytest tests/test_extract.py -k "no_ast_extractor or extract_r_extracts or no_warning_when_all_code or progress_final_line" — all pass
  • Full suite diffed against clean HEAD — same failure set, +1 net pass
  • Manual validation against a real-world R corpus (see above)
  • Maintainer: confirm .r dispatch placement/_DISPATCH ordering convention is acceptable (placed next to .jl, the closest thematic neighbor — bespoke single-extension scripting extractor, no interop family)

…r exists)

.r/.R was in CODE_EXTENSIONS but had no AST extractor (Graphify-Labs#1689 warning fires on
every R corpus). tree-sitter-r isn't on PyPI, so this follows apex.py's
precedent: a regex extractor over a comment/string-masked copy of the source,
brace-matched for correct function nesting.

extract_r (extractors/r.py) emits function definitions (including backtick-
quoted infix operators like `%||%`), calls, library()/require()/pkg:: imports,
and source() edges — including unrolling the common
`for (mod in files) source(paste0(mod, ".R"))` bootstrap loop into individual
per-file edges, which a literal-string source() regex alone would miss.

R has one flat namespace once every file is source()'d into .GlobalEnv, so a
single-file extractor can't tell a cross-file call from a typo. extract_r
resolves same-file calls and stashes a bare-name hint on the rest;
_resolve_r_bare_calls (extractors/resolution.py), wired into extract() the
same way _resolve_cross_file_imports is for Python, does the corpus-wide
lookup afterward and leaves ambiguous (>1 candidate) names dangling rather
than guessing.

_symbol_safe_name moves to base.py (shared, not R-specific) because an
all-symbol identifier normalizes to the empty string in make_id, which
silently collapsed a `%op%` operator node onto its own file's node id and
produced a self-loop `contains` edge.

Validated against a 15-file, ~150-function real-world R pipeline: 97% symbol
recall against a prior LLM-extracted graph baseline, 0 false-positive nodes,
0 self-loops, all 13 edges of a dynamic source() loop recovered correctly.

Tests: replaced the two tests that pinned .r/.R as "no extractor exists" with
a synthetic-extension fixture (so they stay valid regardless of which
language gains an extractor next) and added a dedicated R extraction test
covering functions, cross-file calls, imports, and sources. Full suite: same
54 pre-existing environment failures as HEAD (tree-sitter-hcl unavailable,
Windows watch/repo-root path handling, shallow-clone git-log fixture) on both
branches; net +1 passing test, 0 regressions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant