fix(scanner): index dot-directories instead of skipping them by default - #193
Open
nicknsheth-beep wants to merge 1 commit into
Open
nicknsheth-beep wants to merge 1 commit into
nicknsheth-beep wants to merge 1 commit into
Conversation
ast-grep skips hidden files and directories by default (its own "hidden" ignore rule), and codemap never overrode that default, so a project whose source lives only under a dot-directory (e.g. `.agents/`) was invisible to --deps/--importers with no config option to opt in. scanDirectory now passes `--no-ignore hidden` so ast-grep walks into dot-directories in general, and re-excludes `.git` plus every directory name in the scanner's own IgnoredDirs list (node_modules, vendor, testdata, build output, language caches, ...) via --globs, so relaxing the hidden-file default doesn't also start indexing those or slow down large repos by default. `.codemap` (codemap's own state directory) is excluded the same way. ast-grep's --globs match the literal path it walks with no awareness of where the scan root begins: given an absolute root, a bare `**/testdata/**`-style exclude matches even when "testdata" is an *ancestor* of root rather than something inside it — which is exactly how this package's own scanner tests are laid out (fixtures under ../testdata/), and running the fix against them found this the hard way (first draft returned zero files for every ../testdata/... fixture in the suite). Fixed by running the ast-grep subprocess with its cwd set to root and "." as the scanned path (cmd.Dir = root), so every walked path — and therefore every glob match — is root-relative and can never see anything above root. Added: - TestScanDirectoryInvokesAstGrepWithHiddenDirsEnabled: locks in the exact args (--no-ignore hidden, the --globs excludes, "." as the scan path) and that ast-grep's cwd is root. - TestScanDirectoryIndexesHiddenDirectories: end-to-end, a file under .agents/ is now indexed and its importer resolves correctly. - TestScanDirectoryDoesNotExcludeRootsNestedUnderAnIgnoredDirName: the regression for the ancestor-name false-exclusion bug above. - TestScanDirectoryStillExcludesKnownNeverSourceDirs: .git, node_modules, and vendor stay excluded even with hidden-dir scanning on. `go test ./...` passes, including this repo's own ../testdata/-based fixtures, which the ancestor-exclusion bug (caught before this commit) would otherwise have broken.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
codemap --deps/--importersnever index any file under a directory whose name starts with.(other than.git), with no config option to opt in. A project that keeps source under.agents/,.config/, etc. is silently invisible to codemap.Repro
The identical files in a non-dot directory index correctly.
Root cause
The scanner shells out to
ast-grep scan, which skips hidden files/dirs by default (--no-ignore hiddenis ast-grep's own opt-out flag). codemap never passed that flag and had no config knob for it.Fix
scanner/astgrep.go: pass--no-ignore hiddentoast-grep, paired with explicit--globsexcludes for codemap's existingIgnoredDirs(.git,node_modules,vendor,testdata,build,dist,target,.venv,.idea,.next, ...) plus.codemapitself — so relaxing the hidden-file default doesn't start indexing.gitinternals or slow down large repos.Second-order bug I found while testing this:
--globsmatch the literal path handed toast-grepwith no notion of where the scan root begins, so a bare**/testdata/**exclude also matches whentestdatais an ancestor of the scan root — exactly how this repo's own scanner tests are laid out (fixtures under../testdata/...). Fixed by runningast-grepwithcmd.Dir = rootand"."as the scan path, so every walked path is root-relative. The existingfilepath.Rel(root, m.File)fallback logic inastgrep.goalready handles ast-grep now reporting root-relative paths, so nothing else needed to change.Tests
scanner/astgrep_test.go:TestScanDirectoryInvokesAstGrepWithHiddenDirsEnabled— locks in the exact args andcmd.Dir, via a fakesgscriptTestScanDirectoryIndexesHiddenDirectories— end-to-end: a file under.agents/indexes and its importer resolvesTestScanDirectoryDoesNotExcludeRootsNestedUnderAnIgnoredDirName— regression test for the ancestor-name false-exclusion bug aboveTestScanDirectoryStillExcludesKnownNeverSourceDirs—.git,node_modules,vendorstay excluded with hidden-dir scanning ongo test ./...passes (all 17 packages),gofmt -l .andgo vet ./...are clean.Related: filing alongside two other independent fixes found during the same investigation (
-Cscoping, and coverage-honesty for unindexed files) — happy to merge in any order, this one has no dependency on the others.