Skip to content

Preserve single-column values during delimiter detection - #1211

Merged
quinnj merged 2 commits into
mainfrom
fix/timestamp-delimiter-20260928
Oct 15, 2026
Merged

quinnj merged 2 commits into
mainfrom
fix/timestamp-delimiter-20260928

Conversation

@quinnj

@quinnj quinnj commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Delimiter detection could elect a candidate that does not appear in the header row, so a one-column file of times or timestamps was split on : and kept only the hour (with long-row problems):

Input CSV 0.10.16 CSV 1.1.0 This PR
ts\n2026-09-28T01:02:03.123456789\n 2026-09-28T01 2026-09-28T01 the timestamp
event time\n10:30:00\n11:45:00\n Time values [10, 11] Time values
Created Date\n2026-09-28T01:02:03\n2026-09-28T01:02:04\n DateTime values 2026-09-28T01 twice timestamps

The first row passed the byte-count "count divisible by the line count" tier: two colons over two lines. The other two passed the data-only fallback, which 1.0 added, so they are 1.0 regressions.

Change: a delimiter that does not split the first surviving row would leave the header one column wide, so detection keeps , in that case. The divisibility tier could only elect such candidates, so it is removed. An explicit delim is unaffected. The existing test that pinned : for "header text\n1:2\n3:4\n" now expects ,, which is what CSV 0.10 returns.

Validation:

  • Regression tests: one, two, and ten rows of times and timestamps under a plain (ts) and a spaced (event time) header, both scanners, other punctuation, every reader, and colon-delimited controls with and without a header.
  • Full suite, Julia 1.12.6, bounds checks, four threads: 331,778 passed.

Co-authored by Codex

AI disclosure: This work was prepared with assistance from OpenAI Codex and Claude Code.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.79%. Comparing base (a0335ff) to head (f5abfe9).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1211      +/-   ##
==========================================
+ Coverage   95.75%   95.79%   +0.04%     
==========================================
  Files           9        9              
  Lines        5489     5566      +77     
==========================================
+ Hits         5256     5332      +76     
- Misses        233      234       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@quinnj quinnj left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I refreshed the source review against base a0335ff5bcbd3abe41a2f6ee9cb1b00187503937 and ran the complete package suite on Julia 1.12.6 with bounds checks and four threads: all 331,670 checks pass. The shared detector handles the single-column timestamp reproducer and the existing reader-mode/colon-delimiter regressions without an additional source finding.

All 16 executed Actions jobs pass on this exact head; performance was intentionally skipped. The current codecov/project check reports a small coverage drop (95.73% versus 95.75%), while patch coverage passes. That check and the required independent approval remain outstanding, so this is not yet ready to merge.

AI disclosure: This work was prepared with assistance from OpenAI Codex.

A one-column file whose header has a space still elected `:` when its
values were times or timestamps ("event time" over "10:30:00" rows), and
returned only the hour. Two fallbacks could elect a candidate that does
not appear in the first row: the byte-count divisibility tier and the
consistency scorer's data-only pick. Such a delimiter leaves the header
one column wide, so detection now falls back to ',' in that case, and
the divisibility tier, which could only elect such candidates, is gone.
CSV 0.10 read "header text\n1:2\n3:4\n" as one column; the test that
pinned ':' for it now expects ','.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quinnj
quinnj merged commit 2f84f42 into main Oct 15, 2026
19 checks passed
@quinnj
quinnj deleted the fix/timestamp-delimiter-20260928 branch October 15, 2026 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant