Skip to content

Harden io.arrow: validate sample-column ordering on load; mmap / partitioned reading (ADR-013 §4.5b, §8) #199

Description

@Polichinel

Two hardening items ADR-013 (views-postprocessing) records as future views-frames work — filed now so the intent stops dangling (§8 named 'a hardening issue to be filed at execution'; consumer execution has started at views-faoapi #100):

  1. Ordering validation on load. arrow.save writes sample = np.tile(np.arange(S, dtype=np.int32), N) (io/arrow.py:47) but arrow.load never reads it — it reshapes positionally (:79-98). A reordered/truncated table yields plausible floats in wrong sample slots, silently. ADR-013 §4.5(b) makes consumers check the column against the tile formula on a separate raw-table read before trusting the load; io.arrow doing (or offering) this itself would harden every consumer at once.
  2. mmap / partitioned arrow. load is read-all-to-RAM (~1.6 GB transient per full-S month shard at global reference). Per-month sharding is the contract's mitigation; mmap or partitioned reading remains the long-term fix. Explicitly NOT a contract dependency (ADR-013 §8).

Refs: views-postprocessing docs/ADRs/013_sampled_forecast_wire_contract.md §4.5(b)/§8; views-faoapi #100 (consumer epic).

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions