Skip to content

fix(scanner): index dot-directories instead of skipping them by default - #193

Open
nicknsheth-beep wants to merge 1 commit into
JordanCoin:mainfrom
nicknsheth-beep:fix/hidden-dot-directories
Open

nicknsheth-beep wants to merge 1 commit into
JordanCoin:mainfrom
nicknsheth-beep:fix/hidden-dot-directories

Conversation

@nicknsheth-beep

Copy link
Copy Markdown

Problem

codemap --deps / --importers never index any file under a directory whose name starts with . (other than .git), with no config option to opt in. A project that keeps source under .agents/, .config/, etc. is silently invisible to codemap.

Repro

repro/
├── .git/
└── .agents/
    ├── a.js   # const b = require('./b');
    └── b.js
codemap --deps .
# before: "No source files found."

The identical files in a non-dot directory index correctly.

Root cause

The scanner shells out to ast-grep scan, which skips hidden files/dirs by default (--no-ignore hidden is ast-grep's own opt-out flag). codemap never passed that flag and had no config knob for it.

Fix

scanner/astgrep.go: pass --no-ignore hidden to ast-grep, paired with explicit --globs excludes for codemap's existing IgnoredDirs (.git, node_modules, vendor, testdata, build, dist, target, .venv, .idea, .next, ...) plus .codemap itself — so relaxing the hidden-file default doesn't start indexing .git internals or slow down large repos.

Second-order bug I found while testing this: --globs match the literal path handed to ast-grep with no notion of where the scan root begins, so a bare **/testdata/** exclude also matches when testdata is an ancestor of the scan root — exactly how this repo's own scanner tests are laid out (fixtures under ../testdata/...). Fixed by running ast-grep with cmd.Dir = root and "." as the scan path, so every walked path is root-relative. The existing filepath.Rel(root, m.File) fallback logic in astgrep.go already handles ast-grep now reporting root-relative paths, so nothing else needed to change.

Tests

scanner/astgrep_test.go:

  • TestScanDirectoryInvokesAstGrepWithHiddenDirsEnabled — locks in the exact args and cmd.Dir, via a fake sg script
  • TestScanDirectoryIndexesHiddenDirectories — end-to-end: a file under .agents/ indexes and its importer resolves
  • TestScanDirectoryDoesNotExcludeRootsNestedUnderAnIgnoredDirName — regression test for the ancestor-name false-exclusion bug above
  • TestScanDirectoryStillExcludesKnownNeverSourceDirs — .git, node_modules, vendor stay excluded with hidden-dir scanning on

go test ./... passes (all 17 packages), gofmt -l . and go vet ./... are clean.

Related: filing alongside two other independent fixes found during the same investigation (-C scoping, and coverage-honesty for unindexed files) — happy to merge in any order, this one has no dependency on the others.

ast-grep skips hidden files and directories by default (its own
"hidden" ignore rule), and codemap never overrode that default, so a
project whose source lives only under a dot-directory (e.g. `.agents/`)
was invisible to --deps/--importers with no config option to opt in.

scanDirectory now passes `--no-ignore hidden` so ast-grep walks into
dot-directories in general, and re-excludes `.git` plus every
directory name in the scanner's own IgnoredDirs list (node_modules,
vendor, testdata, build output, language caches, ...) via --globs, so
relaxing the hidden-file default doesn't also start indexing those or
slow down large repos by default. `.codemap` (codemap's own state
directory) is excluded the same way.

ast-grep's --globs match the literal path it walks with no awareness
of where the scan root begins: given an absolute root, a bare
`**/testdata/**`-style exclude matches even when "testdata" is an
*ancestor* of root rather than something inside it — which is exactly
how this package's own scanner tests are laid out (fixtures under
../testdata/), and running the fix against them found this the hard
way (first draft returned zero files for every ../testdata/... fixture
in the suite). Fixed by running the ast-grep subprocess with its cwd
set to root and "." as the scanned path (cmd.Dir = root), so every
walked path — and therefore every glob match — is root-relative and
can never see anything above root.

Added:
- TestScanDirectoryInvokesAstGrepWithHiddenDirsEnabled: locks in the
  exact args (--no-ignore hidden, the --globs excludes, "." as the
  scan path) and that ast-grep's cwd is root.
- TestScanDirectoryIndexesHiddenDirectories: end-to-end, a file under
  .agents/ is now indexed and its importer resolves correctly.
- TestScanDirectoryDoesNotExcludeRootsNestedUnderAnIgnoredDirName: the
  regression for the ancestor-name false-exclusion bug above.
- TestScanDirectoryStillExcludesKnownNeverSourceDirs: .git,
  node_modules, and vendor stay excluded even with hidden-dir scanning
  on.

`go test ./...` passes, including this repo's own ../testdata/-based
fixtures, which the ancestor-exclusion bug (caught before this commit)
would otherwise have broken.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant