Skip to content

Add reviewed Linguist corpus conventions - #2

Merged
andrew merged 1 commit into
mainfrom
corpus/linguist-conventions
Sep 14, 2026
Merged

andrew merged 1 commit into
mainfrom
corpus/linguist-conventions

Conversation

@andrew

@andrew andrew commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Add 52 rules adapted from GitHub Linguist, bringing the corpus to 254 rules. New coverage includes third-party directory spellings, CocoaPods and Yarn dependency trees, citation/manual documents, named minified files and source maps, .NET designer code, build wrappers and configuration files. The 128 new corpus examples cover nested packages, mixed case, inherited roles and misleading names.

The source audit used Linguist revision 16cac170f73c97eb4e4329d66fcb341ef25ef4f0: vendor patterns, documentation patterns and generated-file detection, including their callers and relevant tests. NOTICE records the revision, adaptations and MIT attribution.

Linguist groups several conventions differently because its categories control file filtering. This import assigns roles by repository purpose: Pods/ is vendor, app.min.js is generated, gradlew is build, and .gitattributes is configuration. testdata/ retains test/fixture roles. Vendor regexes are narrowed to complete components; Carthage/Checkouts and selected Yarn trees avoid classifying all adjacent project metadata as vendor. Documentation directories work inside monorepo packages, with document extensions restricted to avoid matching commands such as bare lowercase install.

Content-dependent generator detection, attribute overrides and broad library basename patterns remain outside this import. A filename such as jquery.js, a plain .d.ts, or packaged data under inst/extdata/ provides insufficient evidence for the added roles. Arbitrary .map files and generator headers need bounded content inspection with separate spoofing and truncation cases. All manifest and lockfile names continue to come from git-pkgs/manifests.

Add case-insensitive suffix matching for designer code and named source maps while retaining literal case rules for existing suffixes. Index directory-path rules by their final component to limit the cost of corpus growth. On the same machine, the million-path benchmark measured 253–258 ms before the additions, 307–308 ms with the initial implementation, and 270–277 ms after indexing, with zero allocations for label matching. These measurements exclude filesystem and blob I/O.

Allow CI on PRs targeting branches other than main so this stacked change receives its own checks. README examples and corpus documentation describe the added conventions and their limits.

@andrew
andrew added this pull request to stack #3 September 14, 2026 15:07
Base automatically changed from corpus/gitignore-conventions to main September 14, 2026 15:16
@andrew
andrew force-pushed the corpus/linguist-conventions branch from 6f1300a to 49c359e Compare September 14, 2026 15:16
@andrew
andrew merged commit 0aa607c into main Sep 14, 2026
5 checks passed
@andrew
andrew deleted the corpus/linguist-conventions branch September 14, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant