Skip to content

feat(file): add ReadFileSplit / ReadFileWithReaderSplit (#719)#742

Open
ChrisJr404 wants to merge 1 commit intoprojectdiscovery:mainfrom
ChrisJr404:feature/read-file-split
Open

feat(file): add ReadFileSplit / ReadFileWithReaderSplit (#719)#742
ChrisJr404 wants to merge 1 commit intoprojectdiscovery:mainfrom
ChrisJr404:feature/read-file-split

Conversation

@ChrisJr404
Copy link
Copy Markdown

Closes #719.

What

Two new helpers in `file/file.go` for streaming values from a list-style file where each line may contain one entry or multiple separator-delimited entries:

```go
// Read each line, then split on every rune in separators, trim whitespace,
// drop empties, stream the result.
ReadFileSplit(filename string, separators ...rune) (chan string, error)
ReadFileWithReaderSplit(r io.Reader, separators ...rune) (chan string, error)
```

When no separators are passed, behaviour reduces to the existing `ReadFile` / `ReadFileWithReader` plus `TrimSpace`. So consumers in httpx / nuclei / similar can call the splitting variant unconditionally and get the strictly-line-based reading for free when they don't pass `','`.

Why

Per @dogancanbakir on httpx#2351:

The resolver file format (one resolver per line) is common to all tools. If we want to support comma separated resolvers on the same line within file, maybe we should move this to utils.

Yes — having it once in `projectdiscovery/utils` lets every PD tool consume the same parsing without copy-pasting the split-and-trim logic each time.

Tests

8 new unit tests in `file/file_split_test.go` covering:

  • `splitLineByRunes` with no separators (line-only behaviour)
  • comma + multiple separators
  • `ReadFileWithReaderSplit` with no separators / with a comma
  • `ReadFileSplit` round-trip against a real `t.TempDir()` file mixing single and CSV lines
  • missing-file error path
  • `ReadFileSplit` with no separator equals `ReadFile` semantics

All new tests pass; existing `file/` package tests (including the `FuzzSafeOpen` fuzzer) still pass:

```
$ go test ./file/...
ok github.com/projectdiscovery/utils/file 20.629s
```

Notes

  • Pure-additive: no existing function signature changes; `ReadFile` etc. behave identically.
  • The split is rune-based via `strings.FieldsFunc` so callers can pass any of `',', ';', '\\t'` etc.; the typical caller will pass just `','`.
  • Whitespace is trimmed and empty pieces are dropped so `"a, ,b"` yields exactly `["a","b"]` without empty noise.

…very#719)

When a tool reads a list of values from a file (resolvers, wordlists,
etc.) it's common for users to mix one-per-line and comma-separated
forms on the same line. The two new helpers stream non-empty values from
the file/reader, splitting each scanned line on the supplied runes and
trimming whitespace; passing no separators reduces to the existing
ReadFile/ReadFileWithReader behaviour with TrimSpace applied.

Closes projectdiscovery#719
@dogancanbakir dogancanbakir requested a review from Mzack9999 May 6, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support comma-separated lines

1 participant