ros2-poc: the FicTrac record in the bag — one /arena/frame message per frame with 10 columns (LAB-303)

docs/development/telemetry-logging-reference.md

@@ -57,6 +57,17 @@ must skip it (rule 2 in the hand-off §7.2). After #198 merges, all readers belo
   `readRunlogText`). **v0.78 guarantee:** the final ring drain and the `trial_quality` event are written **before** the
   export, on normal and on fault runs (v0.77 files can have controller rows after `trial_quality` — see §7).
 
+
+### 1.x The MCAP bag (ROS 2 bridge, LAB-303) and the FicTrac columns it keeps
+
+On the ROS 2 graph (`ros2-poc/`, PR #254) the record of truth is an MCAP bag per run and this
+NDJSON is DERIVED from it by `ros2-poc/nodes/mcap_convert.py`; every row and event above is
+unchanged. The bag's FicTrac record is the per-frame `/arena/frame` message: the behavior row's
+five columns plus FicTrac's error score (col 5), lab-frame delta rotation (cols 6–8) and sequence
+counter (col 23), `float64`, one message per frame — the other 15 columns and the verbatim line are
+not recorded by default (`--record-fictrac-raw` opts in; the `full` level needs it). Why those ten
+and what it costs (≈ 2.2–2.5× this file's `.gz`): `fictrac-columns-in-the-bag.md`.
+
 ## 2. Row catalogue (arrays)
 
 All examples are verbatim lines from `arena-log-20260913-120611-713.jsonl` unless noted. "Clock" names the owner of the

ros2-poc/README.md

@@ -187,10 +187,10 @@ without further changes; **"ROS Studio" = that Studio served from the `/ros/` Pa
 
 | item | where | what |
 |---|---|---|
-| MCAP record | `nodes/recorder.py`, `bridge_node.py` | one bag per run (opened with the run's log, closed at `log_control off` / `log_export` / shutdown) holding every device topic as received (raw CDR bytes), `/arena/frame` (the closed-loop evaluation with the node's **own sample time** `t_ms`/`ms` as fields, and as MCAP `publish_time`), `/arena/browser` (**every** browser message, verbatim, with the receive stamp), `/arena/event` (bias_config, heading_tare, connector_status, source_lost/resumed, warn), `/arena/session`. Bridge-side records carry the writer's emission sequence number `seq`. Two writer profiles: `crashsafe` (no chunking, no cache; the default) and rosbag2's `default`. |
+| MCAP record | `nodes/recorder.py`, `bridge_node.py` | one bag per run (opened with the run's log, closed at `log_control off` / `log_export` / shutdown) holding every device topic as received (raw CDR bytes) — except the FicTrac topics, which since 2026-10-18 are recorded only with `--record-fictrac-raw` ([docs/development/fictrac-columns-in-the-bag.md](../docs/development/fictrac-columns-in-the-bag.md)) — `/arena/frame` (the closed-loop evaluation with the node's **own sample time** `t_ms`/`ms` as fields, and as MCAP `publish_time`, plus the 10 FicTrac columns the lab keeps: the behavior row's 5 and `err`, `dr_lab_x/y/z`, `ft_seq`, `ft_ns`; an untracked `FT_BADFR` frame is a flagged record), `/arena/browser` (**every** browser message, verbatim, with the receive stamp), `/arena/event` (bias_config, heading_tare, connector_status, source_lost/resumed, warn), `/arena/session`. Bridge-side records carry the writer's emission sequence number `seq`. Two writer profiles: `crashsafe` (no chunking, no cache; the default) and rosbag2's `default`. |
 | converter | `nodes/mcap_convert.py` (`pixi run convert`) | bag → NDJSON, pure Python (`mcap` + a 60-line CDR decoder driven by the schema text rosbag2 embeds): no ROS process needed, reads a torn bag with `--recover`. Reuses bridge.py's own pure encoders (`compact_arena_command`, schema line). Levels are a converter option (`--level behavior_v2|behavior_v1|full`); the run's acked level is the default. `--order seq` reproduces the live writer's order; `--order time` is what a separate closed-loop process would have to do. ONE ms base per run (the pipeline's t0 is set from the log's open stamp — bridge 3.4 takes two samples there, comparison.md appendix 4). `log_export` returns the converted log; the live NDJSON stays as a shadow file during the spike and `converted == live` is checked and printed at every export. |
 | typed messages | `interfaces/arena_interfaces/` (`FicTracFrame`, `ArenaFrame`, `ConnectorReply`) | built **by `pixi install -e typed`** (pixi-build preview + the RoboStack `pixi-build-ros` backend; the opt-in `typed` environment depends on the package by path and carries the ament/rosidl toolchain). The **default environment stays String-only** so a rig PC without a compiler can always install and run the graph; the nodes choose typed topics only when the package imports AND serializes (`_common.probe_interfaces`). `pixi run build-interfaces` is the colcon alternative. `ArenaFrame` and `FicTracFrame` are in use; `ArenaFrame` also carries two DERIVED channels for live viewing — `turning_dps` (heading rate, deg/s) and `forward_mms` (forward velocity, mm/s; ball diameter `--ball-mm`, 9 mm default) — computed in the bridge node from consecutive frames the way `js/kinematics.js` does (two-point difference on the FicTrac camera clock, NaN on a run's first frame); the hub message to the Studio and the behavior_v1 columns are unchanged. `ConnectorReply` is defined and built but the BIAS reply stays JSON-in-String (its shape is BIAS's, not ours — typing it would make the derived connector event a rebuild instead of a copy). |
-| FicTrac as a typed topic | `fictrac_udp_adapter.py --typed`, `bridge_node.py --typed` | the adapter publishes `/fictrac/frame` (parsed columns + FT tag + receive stamps + the verbatim line) beside `/fictrac/raw`; the bridge subscribes to the typed topic when the package imports (`auto`). The mapping math still parses the verbatim line, so it stays byte-for-byte bridge 3.4's. |
+| FicTrac as a typed topic | `fictrac_udp_adapter.py --typed`, `bridge_node.py --typed` | the adapter publishes `/fictrac/frame` (parsed columns + FT tag + receive stamps + the verbatim line) beside `/fictrac/raw`; the bridge subscribes to the typed topic when the package imports (`auto`). The mapping math still parses the verbatim line, so it stays byte-for-byte bridge 3.4's. The typed FicTrac topic is a LIVE topic; the bag records it only with `--record-fictrac-raw` (2026-10-18). |
 | harness | `tests/test_ros2_poc.py` | the 32 PoC checks now judged against the **converted** log, plus bag presence per topic, fidelity vs the shadow, levels, ordering, typed schemas, and a crash phase (bridge node SIGKILLed mid-run, both profiles): **46 / 46 in 46 s** on the Mac. |
 | Pages path | `scripts/tiers/stamp.py --ros REF`, `deploy-pages.yml` input `ros`, `pixi run ros-preview`, `js/build-channel.js` channel `ros` | a third, optional path `/ros/` beside Production and Next: this branch's tree, stamped `channel: "ros"`, magenta badge/banner "ROS STUDIO", tier-suffixed window names, `run_metadata.channel = "ros"`. Off by default; a problem there never blocks the other tiers; the production tree check skips it. Production and Next are untouched (tiers test 109 / 109). **Enabling it on the live site needs this tooling commit on `main`** (the deploy workflow runs from `main`): a tooling-only PR, then `gh workflow run deploy-pages.yml -f ros=<spike sha>`. Until then: `pixi run ros-preview` + `python -m http.server -d _site`. |
 
@@ -206,7 +206,7 @@ without further changes; **"ROS Studio" = that Studio served from the `/ros/` Pa
 | recording overhead on the closed loop (adapter → bridge, `connector_status`) | FicTrac p50 0.87 / p99 1.65 ms, Kinefly p50 0.82 / p99 1.51 ms — the same as the PoC without recording (0.89 / 1.43) |
 | recorder receive time vs the closed-loop node's own sample time (`/arena/frame` `log_time − publish_time`) | p50 0.53 ms, p99 1.07 ms, max 1.2 ms (the sample time is a whole ms). The record carries both, so `idx` reconstruction uses the node's clock, never the recorder's. |
 | conversion | 962 lines in 50 ms (smoke run); 3949 lines (harness run, 197 kB NDJSON) from a 4.1 MB bag of 4889 records while open. Open, the bag is ~20× the NDJSON: it holds the raw inputs (every FicTrac line twice — typed fields + verbatim — every Kinefly envelope) and the outputs, uncompressed so a crash loses nothing; after the close-time re-index (zstd) it is ~3× the NDJSON (bench: 6.76 MB → 2.33 MB vs a 954 kB log). |
-| **fidelity, sequenced lines** (frames, browser lines, bridge events, session/schema): converted vs the live writer | **byte-identical and in the same order** in every run (smoke, harness, both levels); `bag → behavior_v1` equals `bridge.py --convert` of the live v2 byte for byte; `full` carries the 25 FicTrac columns for every frame |
+| **fidelity, sequenced lines** (frames, browser lines, bridge events, session/schema): converted vs the live writer | **byte-identical and in the same order** in every run (smoke, harness, both levels); `bag → behavior_v1` equals `bridge.py --convert` of the live v2 byte for byte; `full` carries the 25 FicTrac columns for every frame (bags recorded with the raw FicTrac topics; since 2026-10-18 that needs `--record-fictrac-raw`, otherwise `full` is refused) |
 | fidelity, DERIVED lines (2204 Kinefly rows + 31 BIAS reply/status events): content | identical apart from `ms`; `\|Δms\|` **p50 0 / p99 1 ms**, worst 2 ms in the typed run and 12 ms once (one row of 2236) in a String-topic run while the harness was killing and restarting the adapter (recorder receive vs the live writer's processing time) |
 | fidelity, derived lines: placement | 113 of 2235 (5 %) land after a different neighbouring line than in the live file (within the same 1–2 ms). The file as a whole is therefore **not** byte-identical when Kinefly/BIAS are present; it is when they are absent. |
 | `--order time` vs `--order seq` | same multiset of lines; **17 of 3949** positions differ (12 of 962 in the smoke run), all same-millisecond ties (`log_control` vs the first frame, runner `fictracApply` vs `bias_config`, `config` vs `heading_tare`); `closed-loop-report.py` output identical for both orders |

ros2-poc/interfaces/arena_interfaces/msg/ArenaFrame.msg

@@ -2,9 +2,14 @@
 # frame (LAB-303 Plan B, /arena/frame). THE field the record of truth needs is t_ms — the
 # closed-loop node's OWN sample time, the instant idx, the bias phase and a pending tare were
 # evaluated. A recorder's receive time lags it; offline reconstruction must use t_ms / ms.
+#
+# Since 2026-10-18 this is also THE FicTrac record in the bag (docs/development/
+# fictrac-columns-in-the-bag.md): the raw FicTrac topics are no longer recorded by default, and
+# the 10 FicTrac columns the lab keeps ride here — cols 1, 15, 16, 17, 22 as before, plus
+# cols 5, 6-8 and 23 below. One message per frame; one type for the live topic and the bag.
 int64 t_ms            # the node's wall clock at evaluation (ms since the Unix epoch)
 int32 ms              # the same instant relative to the run's t0 (behavior_v1 column `ms`)
-int64 fc              # FicTrac frame counter (col 1)
+int64 fc              # FicTrac frame counter (col 1); 0 when an untracked frame carried none
 int32 idx             # displayed frame index (0-based)
 bool has_ft           # false when FicTrac's col-22 timestamp was absent
 float64 ft            # FicTrac camera timestamp relative to the first frame, ms (behavior_v1 `ft`)
@@ -18,6 +23,16 @@ uint64 seq            # the bridge writer's emission order within the run (0 whe
 # to the Studio and the behavior_v1 columns do not carry them. Two-point differences over
 # consecutive frames on the FicTrac camera clock `ft` (falling back to the bridge clock when
 # ft is absent), the way js/kinematics.js derives the same channels; NaN on the first frame
-# of a run, when the clock does not advance, or when FicTrac restarted.
+# of a run, when the clock does not advance, or when FicTrac restarted. In the BAG these two
+# are always NaN (live-only since 2026-10-18): exactly recomputable offline, and 16 B per frame
+# of incompressible doubles otherwise — a fifth of the record.
 float64 turning_dps   # heading rate, deg/s (positive = increasing FicTrac heading)
 float64 forward_mms   # forward velocity, mm/s (x/y step projected onto the heading × ball radius; --ball-mm, 9 mm default)
+# The FicTrac columns kept beyond the behavior row (NaN / 0 when the line did not carry them).
+float64 err           # col 5: delta-rotation error score (FicTrac's optimizer residual; its only per-frame quality signal)
+float64 dr_lab_x      # cols 6-8: delta rotation vector in laboratory coordinates, radians/frame —
+float64 dr_lab_y      #   the primary measurement everything else is integrated from
+float64 dr_lab_z      #   (FicTrac: heading -= dr_lab_z; velx = dr_lab_y; vely = -dr_lab_x)
+int64 ft_seq          # col 23: sequence counter — equals fc until FicTrac's tracking is reset, then restarts at 1
+int64 ft_ns           # col 22 as received, truncated to an integer (a ns camera clock on our rigs; FicTrac's doc says ms)
+uint8 flags           # bit 0: untracked frame (an FT_BADFR line — no columns; recorded so the bag says "untracked", not "missing"; no run-log row)

ros2-poc/nodes/_rows.py

@@ -1,14 +1,90 @@
-"""_rows.py — pure row encoders shared by the bridge node (live shadow file) and the converter.
+"""_rows.py — pure row/record helpers shared by the bridge node (live shadow file) and the converter.
 
 ONE definition on purpose: the converter's fidelity promise (`converted == live`) depends on both
-writers encoding a Kinefly record identically. No ROS imports here — mcap_convert.py must stay
-runnable without a ROS process.
+writers encoding a record identically. No ROS imports here — mcap_convert.py must stay runnable
+without a ROS process.
+
+  * Kinefly KF/1 → the compact ["kf", …] row (kf_row)
+  * a FicTrac line → (tag, fields)                                   (parse_fictrac_line)
+  * the FicTrac columns kept in the ArenaFrame record (docs/development/fictrac-columns-in-the-bag.md,
+    decided 2026-10-18): cols 5, 6-8, 22 native, 23                  (fictrac_extras)
+  * an ArenaFrame record (typed fields or the hub message) → the behavior_v1 row
+    [ms, fc, idx, ft, x, y, hd], or None for a flagged (untracked) frame (arena_frame_to_row)
 """
 
 from __future__ import annotations
 
+import math
+
 KF_TAG = "kf"
 
+NAN = float("nan")
+
+# ArenaFrame.flags
+FLAG_BAD_FRAME = 1  # bit 0: an FT_BADFR line — FicTrac could not track; no columns, no run-log row
+
+# The FicTrac columns (1-based) kept beyond the behavior row, and the ArenaFrame field each lands in.
+FICTRAC_EXTRA_COLS = {"err": 5, "dr_lab_x": 6, "dr_lab_y": 7, "dr_lab_z": 8, "ft_ns": 22, "ft_seq": 23}
+BEHAVIOR_ROW_COLS = ("ms", "fc", "idx", "ft", "x", "y", "hd")  # == bridge.py BEHAVIOR_V1_COLS (never widened)
+
+
+def parse_fictrac_line(raw: str) -> tuple[str, list[float]]:
+    """(tag, fields) the way bridge.py's Pipeline parses: a leading FT* token is the tag
+    ('' when absent, as in offline .dat lines and the simulator); fields = the floats after
+    it, or [] when the line is not numeric (a bad frame has no usable data)."""
+    parts = raw.strip().split(",")
+    tag = ""
+    if parts and parts[0].strip()[:2].upper() == "FT":
+        tag = parts[0].strip()
+        parts = parts[1:]
+    try:
+        return tag, [float(p) for p in parts]
+    except ValueError:
+        return tag, []
+
+
+def is_bad_frame_tag(tag: str) -> bool:
+    """FicTrac's socket output tags every record: "FT" = tracked, anything else starting with
+    FT (seen: "FT_BADFR") = a frame it could not track."""
+    return tag[:2].upper() == "FT" and tag != "FT"
+
+
+def fictrac_extras(fields: list[float] | None) -> dict:
+    """The kept FicTrac columns beyond the behavior row, as ArenaFrame fields. PURE. A short or
+    absent line gives NaN for the floats and 0 for the integers (the typed message cannot carry
+    null); `ft_ns` is col 22 truncated to an integer — on our rigs it already is one (ns)."""
+    f = fields or []
+
+    def fl(col: int) -> float:
+        return float(f[col - 1]) if len(f) >= col else NAN
+
+    def it(col: int) -> int:
+        if len(f) < col:
+            return 0
+        v = f[col - 1]
+        return int(v) if math.isfinite(v) else 0
+
+    return {"err": fl(5), "dr_lab_x": fl(6), "dr_lab_y": fl(7), "dr_lab_z": fl(8), "ft_seq": it(23), "ft_ns": it(22)}
+
+
+def frame_flags(fr: dict) -> int:
+    v = fr.get("flags", 0)
+    return int(v) if isinstance(v, (int, float)) else 0
+
+
+def arena_frame_to_row(fr: dict) -> list | None:
+    """An /arena/frame record → the behavior_v1 row `[ms, fc, idx, ft, x, y, hd]` the live writer
+    emits (bridge.py write_frame), or None when the record is a flagged untracked frame (the
+    live writer wrote no row). Accepts the typed ArenaFrame fields (`has_ft`) or the hub message
+    the String path records (`ft` None when absent)."""
+    if frame_flags(fr) & FLAG_BAD_FRAME:
+        return None
+    if "has_ft" in fr:  # typed ArenaFrame
+        ft = float(fr["ft"]) if fr["has_ft"] else None
+        return [int(fr["ms"]), int(fr["fc"]), int(fr["idx"]), ft, float(fr["x"]), float(fr["y"]), float(fr["hd"])]
+    return [fr.get(c) for c in BEHAVIOR_ROW_COLS]
+
+
 def kf_row(rel_ms: int, pkt: dict, seq: int) -> list:
     """KF/1 record → compact tagged row. PURE. Angles are the FIRST (strongest-edge)
     angle of each part when the part is valid, else null — Kinefly's bridge plan: the