A hands-free infrared pan-temperature monitor and cooking coach. A thermal camera watches your pan from a mount above the stove; a bright touchscreen shows the real surface temperature, predicts overshoot before it happens, and tells you exactly when to turn the heat down, flip, or add the next batch — no probes, no instrumented cookware, no phone.
Status: roadmap complete — M0-M21 (live control hardware-gated). This README grows one section at a time as each milestone lands. Sections below marked (coming in M#) aren't built yet.
🖼️ UI images are synthesized placeholders — rendered by the LVGL simulator (
python scripts/screenshots.py), not photos of a real panel. They show the actual on-device layout and will be swapped for device photos once the hardware is in hand. New screens appear here as each milestone lands.
PanPilot points a MLX90640 32×24 thermal array at your pan and turns what it sees into plain guidance. Because it measures the whole pan (not a single spot) it can tell a pan from a hand from an empty burner, follow the temperature as you cook, and — the flagship trick — predict the peak a ramp is heading for and warn you before you blow past your target.
It runs in two forms from one firmware:
- Stovetop Advisor — mounted over any stove (gas, electric, induction). It coaches: “TURN TO MEDIUM”, “FLIP”, “READY”, with screen flashes and buzzer patterns, and even checks that you actually turned the knob. (guidance: M3+)
- Griddle Autopilot — with the optional PanPilot SSR box inline to an electric griddle, it holds the temperature itself. (Phase 3: M14+)
No accounts, no cloud, works fully offline. Wi-Fi (M8+) is a convenience layer.
The as-built bench unit — three parts, no soldering:
| # | Part | What it does | Link |
|---|---|---|---|
| 1 | Elecrow CrowPanel Advance 3.5″ HMI — ESP32-S3-WROOM-1-N16R8, 480×320 IPS, GT911 capacitive touch, onboard buzzer, 16 MB flash / 8 MB PSRAM | The whole computer + screen | amazon |
| 2 | Waveshare MLX90640-D110 thermal array — 32×24 px, 110°×75° FOV, I²C @ 0x33, 28 × 16 mm | Sees the pan | amazon |
| 3 | HY2.0-4P → 2.54 mm DuPont female, 20 cm (optional but easiest) | Plugs the sensor into the board's I²C-OUT connector without soldering | amazon |
Plus: a USB-C cable, and something to hold the head above the cooktop (see the enclosure).
Wiring — MLX90640 → CrowPanel I2C-OUT (HY2.0-4P):
| Sensor pin | Board pin |
|---|---|
| VCC | 3V3 |
| GND | GND |
| SDA | IO15 |
| SCL | IO16 |
That connector shares the I²C bus with the GT911 touch controller; the firmware
serialises the two with a mutex and re-grabs the bus pins before every sensor
transaction. The UART1-OUT connector (IO17/IO18) is deliberately left free —
IO17 is the candidate direct SSR trigger.
Lens choice — D110 (this build) vs D55. Both work; the spec allows either and the firmware assumes neither (FOV is
SENSOR_FOV_*_DEGinapp_config.h, used for geometry hints only). They differ only in mounting: the D110 (110°×75°) sees the whole cooktop from a low mount (~30 cm → ~86 × 46 cm, 2–3 burners), while the D55 (55°×35°) is the one to pick for a high hood mount (~60 cm), where the wide lens leaves too few pixels on each pan. Same 28 × 16 mm board and I²C address — a swap is a lens, one config line, andmlx_fovin the enclosure model. Full coverage/pixel table:hardware/enclosure_concept.md.
The 3.5″ basic (ESP32-WROVER) board is a secondary target; its pins are
unverified in this build. CrowPanel Advance 5″ variants are planned targets
(their pin maps are captured in include/board_pins.h).
A parametric OpenSCAD wedge — screen tilted to the cook, sensor pod canted down
at the burners — lives in
hardware/enclosure/panpilot_enclosure.scad
(part = "shell" | "lid" | "base"). Every dimension is measured from Elecrow's
factory STEP model and the Waveshare drawing, not guessed. Print in PETG or
ASA — it lives over a cooktop and PLA sags. Fit is pending a test print
(HARDWARE_TEST "ENCLOSURE").
Easiest — web flasher (Chrome/Edge): 👉 https://jamesdavid.github.io/PanPilot/ — plug the board in over USB-C, pick your board variant, click Install. The page shows the firmware version.
From source (PlatformIO):
pio run -e crowpanel35_advance -t upload # Advance 3.5" (ESP32-S3)
pio device monitor # serial console @115200Updates (M10): once on Wi-Fi, update over the air from a browser at http://panpilot.local/update — no cable. PanPilot uses a dual-app partition and auto-reverts to the previous firmware if a new image boot-loops (3 failed boots), so a bad update can't brick it.
⚠️ Safety: PanPilot is a cooking aid, not a substitute for attention. Never leave a hot stove unattended. Sections on poultry/ground-meat internal temperatures and the SSR box carry safety callouts you should read fully.
The very first boot runs a short setup wizard: a welcome, your temperature unit (°F/°C), where to mount and aim the sensor, and a "you're all set" screen. It appears once — a saved flag skips it on every boot after — and you can change anything later in Settings.
After setup, PanPilot shows the live thermal view — exactly what the sensor sees, false-colored (hot = white/yellow, cool = dark). This replaces a laser dot: aim by moving the sensor head until the pan sits under the center crosshair.
- The green circle is the region of interest (ROI) locked onto the pan; the number is the pan-surface temperature (75th-percentile of the pan interior, so a stray flame lick or hot spot doesn't spike the reading).
- The bottom hint reads “Center the pan in view” → “Good aim” once the pan blob is under the crosshair. “No pan in view” means nothing pan-like is detected.
- Tap the pan to lock the ROI onto it — on a two-burner scene this pins tracking to the pan you tapped even if the other is larger. The Auto button (top-left, shown while locked) returns to auto-follow; the ROI ring turns amber while locked.
Synthesized simulator image; a device photo replaces it once the MLX90640 is wired and aimed at a real pan.
Confidence: every screen shows a confidence indicator. Bare stainless reads low and reflective, so PanPilot caps confidence and leans on the temperature trend rather than the absolute number (add oil/water for a true reading).
The home screen shows the smoothed pan-surface temperature (big numeral), the rate of change with a trend arrow, and — the M3 addition — a target with an ETA and a color-coded action bar telling you what to do.
- Set a target with the
–/+buttons (5 °F steps, remembered across reboots). PanPilot then guides you to it. - Smoothed, not jumpy: the big number is exponentially smoothed (~2 s); the rate is a least-squares fit over 10 s (estimating… until it has data). ETA shows ready in m:ss while heating.
- Action bar (bottom) is the at-a-glance instruction, color-coded: Heat more / Hold (blue) → Turn down soon (amber) → TURN DOWN NOW (orange) → READY (green) → TOO HOT (red). It also shows No pan, Check aim, Cooling.
- °F / °C toggle (top-right). Tap the big temperature for the thermal view.
Overshoot prediction (the flagship trick): PanPilot projects where a fast ramp is heading and calls TURN DOWN NOW before you overshoot — not after.
Full-screen alerts take over the screen for the loud states so you can read them across the kitchen, and clear themselves when the condition passes:
Synthesized simulator images; device photos to follow.
Tap the preset name (top-left) to pick a built-in target band instead of dialing
one in. Each preset sets the ready window and the overheat threshold for that
food; nudging –/+ afterward makes it a custom target.
| Preset | Ready band | Notes |
|---|---|---|
| Eggs | 270–300 °F | gentle |
| Pancakes | 350–375 °F | recovery-monitored (M6) |
| Stainless | 400–450 °F | shows the bare stainless reads low banner |
| Sear | 475–550 °F | high heat, warn at 650 °F |
| Tortillas | 400–450 °F | |
| Generic | 340–360 °F | fully adjustable |
Synthesized simulator image; device photo to follow.
Make your own. The preset grid scrolls, and the last tile is + New —
tap it to define a custom preset: name it on the on-screen keyboard, set the
low/high °F band with the steppers, and flag it as a stainless pan if needed.
Custom presets get their own ✎ edit button (with a Delete), sit alongside the
built-ins, and are saved to flash. Up to eight; the warn threshold is derived
automatically (band top + 100 °F, capped at the 650 °F ceiling).
For batch cooking (pancakes, smash burgers), recovery-monitored presets watch for the temperature drop when food hits the pan, then track the climb back into the band and flash “ADD NEXT BATCH” with a chime when it's ready — so every batch starts at the same temperature. If the pan is climbing back too slowly it says “Recovery slow — raise heat?”; too fast, “watch heat.”
Every cook is logged (to on-device flash): max temperature, time in range, overheat seconds, food-added count, plus a 1 Hz temperature trace. The Last Cook screen (from the preset picker) draws the trace as a sparkline:
The web interface browses the full history and downloads any cook as CSV for a spreadsheet.
The array sees two adjacent burners, so PanPilot tracks both pans with their own independent targets and guidance — e.g. eggs at 300 °F on one and a sear at 500 °F on the other, each with its own READY alert. When a second pan appears the home screen splits down the middle into two columns — each with its own temperature, target, and color-coded action bar (tap a column to set that pan's target). Each pan stays pinned to its burner frame-to-frame.
Each pan can run its own cook. Tap a column → Cook a food and that pan gets its own food timer — so one side counts down "FLIP in 0:48" for the eggs while the other holds a sear. Both timers auto-start on their pan's food-added drop and cue independently.
Presets say how hot; the built-in cook database says how long. Pick a food (“Cook a food” on the preset picker) and PanPilot runs a per-side timer that starts itself, cues flips, and — the differentiator — compensates for the actual pan temperature.
- Auto-start on pour: when PanPilot sees food hit the pan (the temperature drop), the side-1 timer starts. A countdown arc wraps the temperature; a line shows Side 1/2, the batch number, and FLIP / REMOVE in m:ss.
- Temperature-compensated (not a wall clock): the countdown is a doneness accumulator — a cold pan visibly stretches the remaining time (with a banner), a hot pan shortens it. A phone timer can't do this.
- Flip hints: each food carries a real cue, e.g. “flip when bubbles pop and edges set.”
- Food-safety (non-negotiable): poultry, ground meat, pork and fish show “Surface timing only — verify NNN °F internal” — thermal surface data can never claim internal doneness, and the note can't be dismissed.
The 28-entry seed database (src/core/foodlib/foodlib_seed.h) covers breakfast,
burgers, steak, poultry, pork, seafood, melts and vegetarian — authored from
standard references, pending human review before release.
Add your own foods. Drop a /foods.json on the device filesystem and
PanPilot merges it over the seed at boot: a new name + variant adds a food to
the picker, while a matching pair overrides the seed values (times, temps,
flip hint) — so you can retune "Pancakes / 4-inch" to your griddle without
touching firmware. See docs/foods.example.json for
the schema. The safeInternalF field still forces the verify-internal-temp
note and must never be zeroed to quiet the UI.
When a cook finishes, PanPilot asks how it turned out — Undercooked / Perfect / Overcooked. Each answer nudges that food's timer by ±8% and remembers it, so the seed times drift toward your burner and pans over a few cooks. The adjustment is per food and variant (4-inch vs 6-inch pancakes are tracked separately), bounded to 0.6×–1.5× of the original so it can't run away, and saved to flash.
The food picker's blue Recipe program row runs a multi-step cook program (the built-in is Smash Burgers x4; the Recipe Creator adds your own). While a program runs, its name replaces the preset in the top-left, and the bottom bar becomes the step display — and the button:
- Action steps ("Add 2 patties + smash", "Flip + cheese") show "— tap when done ✓": tap the bar to tell PanPilot you did it. Adding food is also auto-detected from the temperature drop, so usually the program advances by itself. Action steps nag at L2 (beep + strobe) until satisfied.
- Passive steps ("Searing side 1", preheat) show a live countdown
(
Searing side 1 1:23) and chirp once on entry — no nagging while the pan is just doing its thing. - The program's hold temperature stays in force between steps — the pan is still meant to sit at 450 °F while you're searing, and the overheat threshold follows the program (clamped by any fat's smoke point).
Every cue — from a gentle trend tick to a loud alarm — routes through one attention system with four escalation levels, so the device can reach you whether you're standing over the pan or across the kitchen:
| Level | Screen | Sound | Used for |
|---|---|---|---|
| L0 Passive | status text / bar color | silent | trend, HOLD, ETA ticks |
| L1 Notify | bar pulse | single chirp | READY, food-added ack, recovery done |
| L2 Act now | full-screen card + backlight strobe | double-beep every 5 s | TURN DOWN NOW, ADD BATCH, flip cue |
| L3 Alarm | full-screen red + strobe | urgent, repeats until cleared | TOO HOT, PLUG ME IN, interlock trips |
- Mute silences L0–L2 but never L3 (a too-hot pan always alarms).
- Compliance verification (Stovetop Advisor): after a “TURN DOWN” cue, PanPilot watches whether the pan's rate of change actually responds — if you turned the knob, it chirps a confirmation; if not, it escalates. The device knows whether you actually did the thing it asked.
- The backlight strobe stays under 3 Hz (photosensitivity) and restores your brightness afterward.
Different pans and burners overshoot by different amounts. Learn Pan Mode (open it from the preset picker) watches a pan heat up for 30 s, measures how fast it climbs, and stores a learned thermal lag so the overshoot prediction is tuned to your pan — not a generic guess.
Tap Start with an empty pan on medium heat, leave it, and Save the result. From then on, guidance uses that pan's lag. (Synthesized simulator image.)
Keep up to 8 pans. Each Save adds a profile (cast iron, nonstick, carbon steel…); open My Pans to switch the active one or delete it. The active profile's learned lag drives overshoot prediction, so a heavy cast-iron and a thin nonstick each get guidance tuned to how they hold and shed heat.
Each pan row also carries an SS chip (tap to mark the pan stainless — the active pan's material drives the stainless guidance behavior for the whole cook).
Map Burner (per-pan knob calibration). "Turn down" cues normally suggest a knob setting from a generic temperature table. Tap Map burner on My Pans to calibrate the active pan on your actual burner instead (~5 minutes): the wizard walks you through the five knob positions (LOW → HIGH), measuring the heating rate at each, then one burner-off window to measure how fast the pan sheds heat. From those six numbers it predicts the hold temperature of every knob setting and shows them before you save. From then on, cues like "aim knob at MED-LOW" come from your burner's map, not the generic table. Maps are per-pan — a thin nonstick and a cast iron on the same burner get different maps.
Open Settings from the preset picker's bottom row to change the things that aren't part of a cook. Tap any row to change it:
- Temperature — switch between °F and °C.
- Sound — mute or unmute all chimes and alarms.
- Brightness — cycle the backlight between Low / Medium / High. On battery the level is capped so a bright setting still saves power.
- Time zone — pick your zone (US Eastern, Central Europe, India, Japan…). On the Wi-Fi build PanPilot syncs the clock over NTP and shows the time on the home screen; the zones carry full DST rules, so it springs forward and falls back on its own.
- Wi-Fi — shows the connection at a glance: "tap to set up" when
unprovisioned, "join AP PanPilot-XXXX" while the setup hotspot is open,
and " — panpilot.local" once connected (that address is the
web interface). Tapping the row reopens the setup hotspot for 3 minutes: join
PanPilot-XXXXfrom your phone, and the captive portal asks for your Wi-Fi password (plus the optional MQTT broker and Web PIN).
Every choice is saved to flash and restored on the next boot. The settings list scrolls, and also holds entries for Autopilot and PID autotune.
PanPilot hosts a local web dashboard — no cloud, no account. Join it to your
Wi-Fi once (it opens a PanPilot-XXXX setup hotspot the first time), then
open http://panpilot.local/ from any phone or laptop on the same network:
-
Live dashboard — current temperature, rate, ETA, target, and the color-coded action bar, pushed over a WebSocket at 2 Hz.
-
Live thermal view in the browser — the 32×24 array streamed and rendered to a canvas with the same ironbow palette as the device, so you can aim and watch the pan from your phone.
-
Settings mirror at http://panpilot.local/settings — change the temperature unit, sound, brightness, and time zone from your phone, exactly like the on-device Settings screen (with the clock in the header). Set an optional Web PIN during Wi-Fi setup and edits require it. Changes are applied on the device's main loop (never from the web task), so they're safe and show up on the panel instantly.
-
Cable-free debugging — the entire serial log is mirrored to a RAM ring served at
/log(the last few minutes of everything the firmware printed), and/api/healthreports firmware version, uptime, free heap, last reset reason, and Wi-Fi signal. With browser OTA at/update(ortools/ota_flash.ps1from the repo), a stove-mounted PanPilot can be diagnosed, fixed, and reflashed without ever touching the USB port.
Everything cooking-related keeps working with Wi-Fi off — the web UI is a convenience mirror. (Browser screenshot added from a live device; the page is served from the ESP32 so it can't be rendered by the simulator.)
At http://panpilot.local/creator you build repeatable cook programs in the browser: pick a food to auto-generate the steps (preheat → add → per-side timers with flip cues → remove with the safety note), tweak them, validate, and save to the device. The firmware validator is the source of truth — it rejects a 700 °F hold, adding butter before a 500 °F sear, or a bad loop. Saved programs appear on the device under the blue Recipe programs card (they're re-validated on load) and run through the same sequencer as the built-in one. It works fully offline.
The same page has a New food form: name, variant, temp band, per-side
times, the flip hint, and the safe-internal-temp — it saves straight into the
device's /foods.json and shows up in the food picker immediately (an entry
matching a built-in name+variant overrides it instead, and can never lower the
USDA internal-temp minimum). So all three kinds are user-creatable: presets
on the device ("+ New" card), foods and programs here in the browser. (Compile-verified; browser flow is bench-tested — HARDWARE_TEST M20.)
Fats are watched, not just timed. A recipe's PREP step runs a fat
monitor: it waits for the pan to reach the fat's add window, tells you when
it's ready (butter foamed/melted after a dwell, oil equalized when the climb
flattens, water immediately), and warns if the pan is too hot to add it yet.
Once a fat is in the pan, a fat-state clamp caps the overheat threshold at
that fat's smoke point for the rest of the cook — so Autopilot (or a TURN DOWN
cue) can't push the pan into burning it, unless the program is explicitly stamped
browning on purpose. This logic is unit-tested (test_prep, test_recipe).
Enter your MQTT broker address during Wi-Fi setup (optional field) and PanPilot appears in Home Assistant automatically via MQTT discovery — no YAML:
- Sensors: pan temperature, rate, guidance state, and the current Alert cue (mirrored the instant PanPilot escalates — flash the lights on "Too hot").
- Binary sensor: pan present.
- Controls: mute (switch), target (number), active preset (select) — all commandable from HA.
- Availability via an MQTT LWT, so HA shows PanPilot offline when it's off.
Example automation: flash the kitchen lights when guidance = “Too hot.” Leave the broker field blank to keep MQTT off. (Compile-verified; broker behavior is bench-tested — see HARDWARE_TEST M9.)
Point PanPilot at a $40 electric griddle with the PanPilot SSR box inline and the same firmware stops advising and starts doing — it holds the temperature itself. Nothing about the perception, guidance, preset, food or recipe layers changes; the actuator is the only variable.
⚠️ Supervised use only. Mains + heat. Autopilot is for an attended cook on an electric griddle/hot-plate through the SSR box. Never leave it unattended; never used for gas. The box switches line voltage — build it exactly perhardware/panpilot_ssr_box.yamland §3.1.1 (name-brand 40 A zero-cross SSR, heatsink, thermal fuse, line fuse, griddle thermostat at MAX as a backstop).
How it stays safe — the interlocks. Control authority is a privilege the perception system must continuously earn. Before every control tick, eleven interlocks (S1–S11) run and can only lower power — the PID can never override them. All fail safe to power off:
| Trips when | ||
|---|---|---|
| S1 | confidence < 60 for 5 s | S7 |
| S2 | pan removed | S8 |
| S3 | obstruction > 10 s | S9 |
| S4 | runaway (duty high, not heating) | S10 |
| S5 | over warn+25 °F or 650 °F | S11 |
| S6 | sensor fault / frame gap |
Arming ceremony: ASSIST is available only when a watchdog-capable actuator is discovered and armed through an explicit confirm screen naming it — with no actuator present, every control element is hidden and there is no code path to energize anything. Open it from Settings → Autopilot: the screen names the actuator, spells out that you're still the cook, and keeps the ARM button disabled until you flip the "I'll stay nearby" switch. The box's own hardware watchdog turns the SSR off within 15 s if PanPilot ever goes quiet.
Once armed, the home screen carries a persistent red STOP bar showing the commanded duty (or the interlock reason when power is being held) — tap it to cut power instantly (that's interlock S9).
PID autotune. Every griddle heats differently, so PanPilot can tune its own PID gains. From Settings → PID autotune (with Autopilot armed and an empty pan on the heat), it runs a relay autotuner — briefly pulsing the burner to induce a controlled oscillation — measures the ultimate gain and period, and derives Ziegler-Nichols gains. It shows the result to Save or Discard; saved gains persist and drive every later hold.
The control logic — interlocks (S1–S11), bang-bang & PID against a simulated
plant, and the relay autotuner — is unit-tested (test_interlocks,
test_controller, test_autotune); the SSR box firmware is
hardware/panpilot_ssr_box.yaml. Live control is bench-gated on building the
box — see HARDWARE_TEST M14–M18.
- Specs (authoritative, read-only):
specs/panpilot-firmware-spec.md(M0–M6),specs/panpilot-phase2-to-ultimate-spec.md(M7+). Working agreements:CLAUDE.md. - Layout:
include/(board pins, config, LVGL config),src/hal(display, touch, buzzer),src/ui(LVGL screens),src/core(hardware-free, unit-tested logic — grows M1+),src/sensor(thermal pipeline — M1+),test/(native Unity tests),sim/(LVGL SDL simulator — M0+),scripts/,web/(flasher). - Build & test:
Native tests run locally (MinGW-w64 GCC) and in CI; CI additionally renders the simulator screenshots.
pio run -e crowpanel35_advance # firmware pio test -e native # host unit tests (runs in CI)
- Hardware bring-up checklist:
HARDWARE_TEST.md.