feat(plugins): request_on_demand() / end_on_demand() -- plugins ask for the screen in-process

CHANGELOG.md

@@ -19,6 +19,30 @@ accepts both, but the store flags the old spelling as deprecated
 
 ## Unreleased
 
+### Plugins ask for the screen in-process: `request_on_demand()` / `end_on_demand()`
+
+The in-process way in that stage 5 of the control socket needed
+(`docs/IPC_CONTROL_SOCKET.md`, "Plugins in the display process").
+
+- **`BasePlugin.request_on_demand(mode=None, duration=None, pinned=False)`**
+  shows the plugin now, and **`BasePlugin.end_on_demand()`** gives the
+  screen back. Both are safe from any thread (an MQTT callback, a timer
+  thread): `PluginManager.request_on_demand()` / `end_on_demand()` hand the
+  request to `DisplayController.submit_plugin_on_demand()`, which only
+  queues it (at most 32) and wakes the render thread through the control
+  socket's flag (`ControlServer.wake()`). The render thread applies it with
+  the socket's commands, through the same handler as a web on-demand
+  request, so it lands within a frame rather than on the mailbox's
+  once-a-second look. Both return the request id, or `None` when no display
+  runs in the process (the web interface, `scripts/check_plugin.py`) or the
+  queue is full.
+- **A plugin's stop ends only its own session.** A mailbox stop still ends
+  any session, whoever started it.
+- **Older cores.** Plugins detect the methods with `hasattr` and write the
+  `display_on_demand_request` mailbox when they are missing or answer
+  `None`; the pattern is in `docs/PLUGIN_API_REFERENCE.md` ("On-demand
+  display"). The display still reads the mailbox for plugins that write it.
+
 ### Web UI: Schedule and General are ES-module pages (stage 3)
 
 - The Schedule and General tabs follow stage 2 (#727): their inline

docs/IPC_CONTROL_SOCKET.md

@@ -489,7 +489,7 @@ restart banner, as before.
 
 | Mailbox | Written by | Read by the display | While the socket is up |
 |---|---|---|---|
-| `display_on_demand_request` | the web interface, only on fallback; four plugins directly (birdnet-go, mqtt-notifications, on-air, pomodoro-timer) | the render thread, `_poll_on_demand_requests()` | looked at every 1 s (`MAILBOX_POLL_INTERVAL_WITH_SOCKET`), 0.25 s without a socket |
+| `display_on_demand_request` | the web interface, only on fallback; plugins that predate `BasePlugin.request_on_demand()`, or run on a core without it | the render thread, `_poll_on_demand_requests()` | looked at every 1 s (`MAILBOX_POLL_INTERVAL_WITH_SOCKET`), 0.25 s without a socket |
 | `plugin_error_clear_request` | the web interface, only on fallback | the error publisher's thread, every 5 s tick | unchanged rate |
 
 A look is one `stat()` of the mailbox file (`CacheManager.file_signature`):
@@ -502,8 +502,25 @@ the mailbox instead of being re-read until it expires.
 
 A request that comes through the on-demand mailbox while the socket is up
 is logged once per writer (`came through the file mailbox although the
-control socket is up`), which names the plugins that still need an
-in-process way in before the mailbox is removed.
+control socket is up`), which names the plugins that still write it.
+
+### Plugins in the display process
+
+A plugin asks for the screen with `BasePlugin.request_on_demand()` and gives
+it back with `end_on_demand()` (see "On-demand display" in
+[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md)). Neither goes through
+the socket or a file: `PluginManager` hands the mailbox-shaped request,
+marked `source: 'plugin'`, to `DisplayController.submit_plugin_on_demand`,
+which queues it in memory (at most `PLUGIN_ON_DEMAND_QUEUE_SIZE`, 32) from
+whatever thread the plugin called on, and wakes the render thread through
+the socket's queue flag (`ControlServer.wake()`). The render thread applies
+it in `_drain_control_commands`, after the socket's commands, through the
+same `_handle_on_demand_request`, so it lands within a frame like a socket
+command. Without a socket it lands on the next pending-changes pass (typically
+within 0.25 s). A plugin's stop ends only a session that plugin owns. The four
+plugins that wrote the mailbox (birdnet-go, mqtt-notifications, on-air,
+pomodoro-timer) use it where the core has it and write the mailbox
+otherwise.
 
 ## Robustness
 
@@ -659,9 +676,11 @@ device never touches the live display.
      running display (their routes say so); they are not mailboxes.
 5. **Remove the mailboxes (next release).** Once every device has run a
    display with stage 4, the web interface stops writing both mailboxes and
-   the display stops reading them. The four plugins that write
-   `display_on_demand_request` need an in-process way to ask for the screen
-   first. The display also stops writing `display_current_state`,
+   the display stops reading them. The four plugins that wrote
+   `display_on_demand_request` now have an in-process way to ask for the
+   screen (`BasePlugin.request_on_demand()` / `end_on_demand()`, see
+   "Plugins in the display process"); they keep the mailbox write only as
+   their fallback on older cores. The display also stops writing `display_current_state`,
    `display_on_demand_state` and `plugin_runtime_snapshot` once the web
    interface no longer falls back to them.
 

docs/PLUGIN_API_REFERENCE.md

@@ -488,6 +488,78 @@ working for the plugin itself. `get_vegas_segment_width()` read the
 `vegas_panel_count` config value, which has never affected Vegas — a card's
 width comes from `get_vegas_content()` and `vegas_width_pct`.
 
+### On-demand display
+
+A plugin that reacts to something outside the rotation (an MQTT message, a
+timer, a detection) can take the screen for it, and give it back. Both
+methods are safe from any thread, including an MQTT callback: they only
+queue the request, and the display applies it on its render thread within a
+frame or so, exactly like an on-demand start or stop from the web interface.
+
+#### `request_on_demand(mode=None, duration=None, pinned=False) -> Optional[str]`
+
+Show this plugin now.
+
+- `mode`: one of the plugin's display modes; `None` for its first.
+- `duration`: seconds before the rotation resumes; `None` (or `0`) for no
+  limit, until `end_on_demand()` or the user stops it.
+- `pinned`: stay on `mode` instead of cycling through the plugin's other
+  modes.
+
+Returns the request id once the display has queued it, or `None` when
+there is no display in this process to ask (the web interface's plugin
+manager, `scripts/check_plugin.py`) or its queue is full. A bad argument
+(a `mode` that is not a string, a `duration` that is not a number) raises
+`ValueError`.
+
+#### `end_on_demand() -> Optional[str]`
+
+Give the screen back. Ends only a session this plugin owns: a session the
+user started for another plugin, or one that already ended, is left alone.
+Returns the request id once queued, or `None` as above.
+
+#### Older cores: feature detection
+
+These methods are new after core 3.8.0 (see `CHANGELOG.md`). Before them,
+plugins wrote the `display_on_demand_request` cache key (the "mailbox")
+themselves. The display reads it only once a second while the control
+socket is up, and it will be removed in a future release (see
+[IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md), stage 5). A plugin that
+must keep working on older cores checks for the method, and writes the
+mailbox only when the method is missing or answers `None`:
+
+```python
+import time, uuid
+
+def _show_alert(self):
+    if hasattr(self, "request_on_demand") and self.request_on_demand(
+            mode="my_alert", duration=15):
+        return
+    # Older core, or no display in this process: the mailbox, as before.
+    self.cache_manager.set("display_on_demand_request", {
+        "request_id": str(uuid.uuid4()), "action": "start",
+        "plugin_id": self.plugin_id, "mode": "my_alert",
+        "duration": 15, "pinned": False, "timestamp": time.time(),
+    })
+
+def _release(self):
+    if hasattr(self, "end_on_demand") and self.end_on_demand():
+        return
+    self.cache_manager.set("display_on_demand_request", {
+        "request_id": str(uuid.uuid4()), "action": "stop",
+        "plugin_id": self.plugin_id, "timestamp": time.time(),
+    })
+```
+
+Keep `ledmatrix_min_version` where it is: the fallback is what keeps the
+plugin working on older cores. A mailbox stop ends any on-demand session,
+whoever started it; `end_on_demand()` ends only the plugin's own.
+
+Both methods answer a request id only when the plugin manager returned a
+string, so a test that gives the plugin a `MagicMock()` plugin manager gets
+`None` and exercises the mailbox path. To test the new path, set
+`plugin_manager.request_on_demand.return_value = "some-id"`.
+
 > The full source for `BasePlugin` lives in
 > `src/plugin_system/base_plugin.py`. If a method here disagrees with the
 > source, the source wins — please open an issue or PR to fix the doc.
@@ -966,6 +1038,14 @@ if info:
     self.logger.info(f"Plugin: {info['name']}, Version: {info.get('version')}")
 ```
 
+#### `request_on_demand(plugin_id, mode=None, duration=None, pinned=False)` / `end_on_demand(plugin_id)`
+
+What `BasePlugin.request_on_demand()` and `end_on_demand()` call, with the
+plugin's own id. Call those instead; see
+[On-demand display](#on-demand-display). The display controller routes them
+to itself with `set_on_demand_handler()`; a plugin manager without a
+display behind it answers `None`.
+
 #### `get_all_plugin_info() -> List[Dict[str, Any]]`
 
 Get information for all plugins.

src/display_controller.py

@@ -394,6 +394,12 @@ def _follower_gated_update():
         plugin_time = time.time()
         self.plugin_manager = None
         self._plugin_runtime_publisher = None
+        # On-demand requests plugins make in this process (BasePlugin.
+        # request_on_demand / end_on_demand), from any thread; the render
+        # thread drains them with the socket's commands. Created before the
+        # plugins load, because a plugin may ask from its first thread.
+        self._plugin_on_demand: deque = deque()
+        self._plugin_on_demand_lock = threading.Lock()
         self.plugin_modes = {}  # mode -> plugin_instance mapping for plugin-first dispatch
         self.mode_to_plugin_id: Dict[str, str] = {}
         self.plugin_display_modes: Dict[str, List[str]] = {}
@@ -508,6 +514,12 @@ def _follower_gated_update():
                 cache_manager=self.cache_manager,
                 font_manager=self.font_manager
             )
+            # BasePlugin.request_on_demand() / end_on_demand() land here.
+            # Before any plugin loads: a plugin may ask from its first thread.
+            # getattr: tests and golden traces stand in simpler managers.
+            set_handler = getattr(self.plugin_manager, 'set_on_demand_handler', None)
+            if callable(set_handler):
+                set_handler(self.submit_plugin_on_demand)
 
             # The web UI's loaded / state / error_info for each plugin read
             # what this publishes. Started before loading, so the loads that
@@ -1882,6 +1894,10 @@ def _show_on_demand_step(self, nxt: ArbiterState) -> None:
     _on_demand_mailbox: Optional[MailboxWatch] = None
     #: Writers whose mailbox requests have been logged (_note_mailbox_request).
     _mailbox_writers_logged: FrozenSet[str] = frozenset()
+    #: Most plugin on-demand requests waiting for the render thread at once.
+    #: A plugin that asks faster than the display drains (four times a
+    #: second at worst) is refused, not queued without end.
+    PLUGIN_ON_DEMAND_QUEUE_SIZE = 32
 
     def _service_pending_changes(self) -> None:
         """Apply changes made elsewhere while the display thread is busy.
@@ -1906,7 +1922,7 @@ def _service_pending_changes(self) -> None:
         # A command queued on the control socket skips the floor: it is in
         # memory, so applying it now costs no disk read.
         if (last is not None and now - last < self.PENDING_CHANGES_INTERVAL
-                and not (self._control_server and self._control_server.has_pending)):
+                and not self._control_command_pending()):
             return
         self._last_pending_service = now
 
@@ -2105,11 +2121,15 @@ def _drain_control_commands(self) -> None:
         here. A plugin reload waits for the top of the next loop pass, where
         no plugin is on the stack (_apply_pending_plugin_reloads); until
         then the current screen ends early (_plugin_reload_pending).
+
+        Plugins' own on-demand requests (submit_plugin_on_demand) are
+        applied here too, after the socket's, with or without a socket.
         """
         server = self._control_server
-        if server is None or not server.has_pending:
-            return
-        for command in server.drain():
+        # drain() clears the wake flag before the plugin queue is read below,
+        # so a plugin request queued from here on wakes the next wait.
+        commands = server.drain() if server is not None and server.has_pending else []
+        for command in commands:
             try:
                 if command.cmd == ControlCommand.BRIGHTNESS_SET:
                     self._apply_control_brightness(command)
@@ -2121,24 +2141,78 @@ def _drain_control_commands(self) -> None:
                 logger.exception("Failed to apply control socket command %s",
                                  command.request_id)
                 command.fail(ControlErrorCode.INTERNAL, 'the display failed to apply it')
+        self._drain_plugin_on_demand()
+
+    # -- plugins' in-process on-demand requests ---------------------------------
+
+    def submit_plugin_on_demand(self, request: Dict[str, Any]) -> bool:
+        """Queue a plugin's on-demand request for the render thread. Any thread.
+
+        ``PluginManager.request_on_demand`` / ``end_on_demand`` (which
+        BasePlugin's methods of the same names call) build ``request``: the
+        mailbox's shape, with ``source: 'plugin'`` and the asking plugin's
+        id. Nothing here touches the panel or the on-demand state; the render
+        thread applies the request where it applies a socket command
+        (_drain_control_commands), through _handle_on_demand_request, and is
+        woken for it when the control socket is up. True when it was queued;
+        False (logged) when the queue is full.
+        """
+        pending = self.__dict__.get('_plugin_on_demand')
+        lock = self.__dict__.get('_plugin_on_demand_lock')
+        if pending is None or lock is None:
+            return False   # a controller built without __init__ (tests)
+        with lock:
+            if len(pending) >= self.PLUGIN_ON_DEMAND_QUEUE_SIZE:
+                logger.warning("Plugin on-demand queue full; refusing %s %s from %s",
+                               request.get('action'), request.get('request_id'),
+                               request.get('plugin_id'))
+                return False
+            pending.append(dict(request))
+        server = self._control_server
+        if server is not None:
+            server.wake()
+        return True
+
+    def _plugin_on_demand_pending(self) -> bool:
+        """A plugin's on-demand request is waiting. A length check: no lock."""
+        return bool(self.__dict__.get('_plugin_on_demand'))
+
+    def _drain_plugin_on_demand(self) -> None:
+        """Apply the plugins' queued on-demand requests, oldest first. Render thread."""
+        pending = self.__dict__.get('_plugin_on_demand')
+        while pending:
+            try:
+                request = pending.popleft()
+            except IndexError:
+                break
+            try:
+                self._handle_on_demand_request(request)
+            except Exception:  # pylint: disable=broad-except
+                logger.exception("Failed to apply on-demand request %s from plugin %s",
+                                 request.get('request_id'), request.get('plugin_id'))
 
     def _wait_for_control(self, timeout: float) -> bool:
         """Sleep up to ``timeout``, waking early for a control socket command.
 
         True when a command is waiting. Without a socket (Windows, switched
-        off, tests) this is the plain sleep it replaces.
+        off, tests) this is the plain sleep it replaces, unless a plugin's
+        on-demand request is already waiting.
         """
         server = self._control_server
         wait = getattr(server, 'wait_for_command', None) if server is not None else None
         if wait is None:
+            if self._plugin_on_demand_pending():
+                return True
             time.sleep(timeout)
             return False
         return bool(wait(timeout))
 
     def _control_command_pending(self) -> bool:
-        """A socket command is queued: Vegas checks this every frame."""
+        """A socket command or a plugin's on-demand request is queued: Vegas
+        checks this every frame."""
         server = self._control_server
-        return bool(server is not None and server.has_pending)
+        return bool((server is not None and server.has_pending)
+                    or self._plugin_on_demand_pending())
 
     def _wait_frame_interval(self, interval: float, screen: Screen) -> Optional[ScreenPlan]:
         """The static screen's sleep between frames, woken by socket commands.
@@ -2450,21 +2524,36 @@ def _note_mailbox_request(self, request: Dict[str, Any]) -> None:
     def _handle_on_demand_request(self, request: Dict[str, Any]) -> None:
         """Process one on-demand request, from the mailbox or the control socket.
 
-        A socket command carries ``source: 'socket'``. Only a mailbox request
-        is removed from the mailbox afterwards: a socket command never put
-        anything there, so that would be a disk read and maybe a delete for
-        nothing.
+        A socket command carries ``source: 'socket'``, and a plugin's own
+        request (submit_plugin_on_demand) ``source: 'plugin'``. Only a
+        mailbox request is removed from the mailbox afterwards: the others
+        never put anything there, so that would be a disk read and maybe a
+        delete for nothing.
+
+        A plugin's stop ends only that plugin's own session: a plugin
+        releasing the screen must not end one the user started for
+        another plugin. (A stop through the mailbox ends any session, as it
+        always has.)
         """
         request_id = request.get('request_id')
         if not request_id:
             return
-        from_mailbox = request.get('source') != 'socket'
+        source = request.get('source')
+        from_mailbox = source not in ('socket', 'plugin')
 
         action = request.get('action')
 
         # For stop requests, always process them (don't check processed_id)
         # This allows stopping even if the same stop request was sent before
         if action == 'stop':
+            if source == 'plugin' and not (
+                    self.on_demand_active
+                    and self.on_demand_plugin_id == request.get('plugin_id')):
+                logger.debug("On-demand stop %s from plugin %s ignored: it does not own "
+                             "the screen (on-demand %s, plugin %s)", request_id,
+                             request.get('plugin_id'), self.on_demand_status,
+                             self.on_demand_plugin_id)
+                return
             logger.info("Received on-demand stop request %s", request_id)
             # Always process stop requests, even if same request_id (user might click multiple times)
             if self.on_demand_active:
@@ -2507,8 +2596,9 @@ def _handle_on_demand_request(self, request: Dict[str, Any]) -> None:
                 self._consume_on_demand_request(request_id)
             return
 
-        logger.info("Received on-demand request %s: %s (plugin_id=%s, mode=%s)",
-                   request_id, action, request.get('plugin_id'), request.get('mode'))
+        logger.info("Received on-demand request %s: %s (plugin_id=%s, mode=%s, via %s)",
+                   request_id, action, request.get('plugin_id'), request.get('mode'),
+                   'mailbox' if from_mailbox else source)
 
         # Mark as processed BEFORE processing (to prevent duplicate processing)
         self.cache_manager.set('display_on_demand_processed_id', request_id, ttl=3600)