[codex] fix: date type inference and zoneless CSV timestamps

[youhaowei]

Fixes date-column type inference and display in data previews and Insight results. Date columns now use Arrow schema metadata when available, and CSV calendar timestamps no longer shift to the next day when imported west of UTC.

Closes #79.

Issue #78 (numeric aggregate display noise) is explicitly deferred. This PR contains no generic numeric-formatting behavior or test changes.

Changes

• Source DataFrame column types from the Arrow schema, with full-sample widening only as a fallback for schema-less results.
• Format numeric epoch values in declared date columns as UTC calendar dates, avoiding west-of-UTC shifts for Arrow date values.
• Normalize strict zoneless T and space-separated CSV date-times to UTC at the CSV adapter boundary before Arrow persistence.
• Preserve existing date-only, Z, and explicit-offset timestamp behavior.
• Keep the shared engine parser and REST connector semantics unchanged; deterministic negative-timezone coverage guards that boundary.

Runtime evidence

Final behavioral QA used a fresh CSV import and a real Insight preview at commit a80d28f8.

Column	Row 1	Row 2
zoneless_t	2024-01-18	2024-07-18
day_only	2024-01-18	2024-07-18
space_datetime	2024-01-18	2024-07-18
utc_z	2024-01-18	2024-07-18
explicit_offset (-07:00)	2024-01-18	2024-07-19

The explicit-offset control proves the offset instant is honored rather than stripped. Browser console contained only development informational messages; no page errors were observed.

Screenshot is stored on the existing pr-320-screenshots prerelease, not in git.

Verification

• bun run check: PASS — all four summary checks, 85/85 package tasks.
• bun run format:check: PASS.
• git diff --check: PASS.
• Focused CSV, engine, REST, app, and UI date tests passed, including randomized CSV order and named negative-timezone REST coverage.
• Independent full-diff review: SHIP, no findings.
• Final runtime QA: PASS against the exact pushed commit, with the rendered values above and screenshot evidence attached.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• Bug Fixes
  • Improved data type detection using declared schema information and comprehensive value validation.
  • Correctly distinguishes valid dates, mixed date-like text, numeric values, and empty columns.
  • Formats numeric values consistently, including trimming floating-point noise.
  • Formats valid epoch-millisecond values as UTC dates for date-typed columns.
  • Rejects invalid timestamps and malformed date values.

Walkthrough

The change adds Arrow schema-aware column typing, broader value inference, numeric display formatting, and epoch-millisecond date formatting. Tests cover numeric, date, mixed, invalid, and string values.

Changes

Data preview formatting

Layer / File(s)	Summary
Arrow schema and column inference packages/app/src/hooks/useDataFrameData.ts, packages/app/src/hooks/useDataFrameData.test.tsx	extractColumns now prefers Arrow field types and otherwise infers types from all non-null values. Date validation, mixed-type widening, and schema-aware tests were added.
Numeric cell formatting packages/app/src/lib/cell-formatter.ts, packages/app/src/lib/cell-formatter.test.ts, packages/app/src/app/insights/.../DataPreviewSection.tsx	Number-typed values now use formatNumeric. Comments and tests describe and verify trimmed floating-point output.
Date cell formatting packages/ui/src/lib/format-virtual-table-value.ts, packages/ui/src/lib/format-virtual-table-value.test.ts	Date columns now format valid numeric epoch timestamps in UTC and return null for invalid numeric or incompatible values.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant QueryResult
  participant useDataFrameData
  participant DataPreview
  participant formatCellValue
  participant formatDate
  QueryResult->>useDataFrameData: provide rows and Arrow schema fields
  useDataFrameData->>useDataFrameData: extract declared or inferred column types
  useDataFrameData->>DataPreview: provide typed columns and values
  DataPreview->>formatCellValue: format numeric values
  DataPreview->>formatDate: format date-typed epoch values
  formatCellValue-->>DataPreview: return formatted numeric text
  formatDate-->>DataPreview: return UTC date text

Loading

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description includes summary, changes, verification, and screenshot evidence, but it omits the required Screenshots section and contradicts the numeric-formatting changes.	Add a dedicated ## Screenshots section and correct the statement that numeric-formatting changes are deferred.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address numeric display noise in #78 and date typing and formatting requirements in #79.
Out of Scope Changes check	✅ Passed	The code, tests, comments, and screenshots support the linked issue objectives without unrelated changes.
Title check	✅ Passed	The title clearly identifies the main date inference and zoneless CSV timestamp changes.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/app/src/hooks/useDataFrameData.ts`:
- Around line 37-57: Update isDateString in
packages/app/src/hooks/useDataFrameData.ts (lines 37-57) to require an explicit
UTC offset before classifying T-separated timestamps as dates, preventing
zone-less timestamps from shifting calendar dates; preserve date-only and
supported existing cases. Add a regression test in
packages/app/src/hooks/useDataFrameData.test.tsx (lines 639-642) for a zone-less
T-separated timestamp under a positive-offset time zone, documenting that
existing date-only and space-separated cases do not exercise this path.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: f16658d6-70ea-485a-a205-4cf1ae00de38

📥 Commits

Reviewing files that changed from the base of the PR and between 97e39d4 and 9b3e4ca.

📒 Files selected for processing (7)

• packages/app/src/app/insights/[insightId]/_components/sections/DataPreviewSection.tsx
• packages/app/src/hooks/useDataFrameData.test.tsx
• packages/app/src/hooks/useDataFrameData.ts
• packages/app/src/lib/cell-formatter.test.ts
• packages/app/src/lib/cell-formatter.ts
• packages/ui/src/lib/format-virtual-table-value.test.ts
• packages/ui/src/lib/format-virtual-table-value.ts

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9b3e4ca911

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Stale automated review on 9b3e4ca. The date finding is addressed and its thread resolved at a80d28f with CSV-boundary normalization, deterministic timezone tests, independent review, and real Insight runtime QA.