Add per-message input callbacks

[gadenbuie]

Closes #375

Summary

Add transform_user_input() callbacks for automatic chat clients in R and Python.
Callbacks transform outbound contents in registration order while the chat UI and last submitted input keep the user's original message.
Slash commands are excluded: a command's handler owns the transformation for its own submissions.
Document the RAG and per-message-state use case in both getting-started guides.

Example

library(shiny)
library(shinychat)
library(ellmer)

ui <- page_fillable(chat_ui("chat"))

server <- function(input, output, session) {
  client <- chat_openai()
  chat <- chat_server("chat", client)

  chat$transform_user_input(function(contents) {
    context <- "The product is available in red and blue."
    c(list(context), contents)
  })
}

shinyApp(ui, server)

When a user asks a question, the model receives the context followed by the submitted contents. The chat UI and chat$last_input() still show only the user's message.

[cpsievert]

This is a Python-specific concern, but I'm not loving the name on_user_input for 2 reasons:

1. It's very similar to on_user_submit
2. For a while we supported transform_user_input (essentially the same thing), but it was removed in #245. That made sense in a world where we expect an explicit on_user_submit, but obviously now with Chat(client=client), it's needed again.

All that said, I think I'd prefer that we "resurrect" transform_user_input. That also aligns with the still supported transform_assistant_response.

Also, I don't think we need to worry about storing the transformed user input in the way we did before #245 since .messages() now only tracks the UI message state (not LLM facing state)

[gadenbuie]

Done — renamed in both R and Python. We looked at a few ways to align the two APIs — including having Python's on_user_submit() do double duty — and decided transform_user_input() is the best way to have similar handlers on both the R and Python side that authentically advertise what they do and can be used the same way in both languages.

Along the way I removed the TypeError tombstone left over from #245.

And agreed on storage: nothing new is stored — the transform only changes the contents passed to the client's stream() / stream_async(), while last_input() / user_input() and the chat UI keep the user's original message.

[nbenn]

Thanks a lot for the quick response. Works for us.

Minor thing to be aware of rather than a problem: a message containing a slash command doesn't go through the callback, since the handler does its own client$stream(). Easy enough to handle there, and in our case they all funnel through one handler anyway.

[gadenbuie]

@nbenn Agreed, and that is now the documented contract: slash command submissions do not go through transform_user_input(). The command's handler owns the transformation for its own submissions — both getting-started guides call this out.

[cpsievert]

This feature doesn't feel worth putting ahead of basically all the other stuff that follows in this document. In fact, I might prefer it's not mentioned here at all.

Furthermore, if I'm gonna put on my skeptical hat, the idea of passing user input directly into a retrieval step (i.e., traditional RAG) is outdated now. Instead you should be equipping the agent with a retrieval tool so it can generate the retrieval query.

@nbenn outside of RAG, it wasn't totally clear to me what you'd need this feature for. Would you be able to elaborate on that? One of the more compelling workflows I've come across is a situation where you want to explain to the LLM the surrounding environment that the user is operating in (which can change over time). For that sort of use case, it feels like you'd want this pattern as well as tidyverse/ellmer#369?

[nbenn]

@cpsievert Our use case is an app where we have state that can be edited by a user via the browser. We've been integrating an assistant to help with such edits.

Instead you should be equipping the agent with a retrieval tool so it can generate the retrieval query.

For us, I feel this is not an "either-or". Ideally we have both: We give the assistant tools to query certain aspects of the app state, but we'd also like to push a summary on each turn. The naive approach was to update the system prompt on each turn. But this comes with its own down-sides: it kills prefix caching and it makes for an inconsistent chat transcript. It might even introduce outright contradictions.

My hope with such a callback was that we can basically inject into each user-turn an update on what has changed since the last assistant-turn. Without re-writing the system prompt up top.

Going tool-only comes at a cost: The model will first have to run a tool to get a summary of the latest updates. Then it might need to drill down a bit with follow-up tool calls. If we have the chance to inject some context into the prompt (and are smart about it), we can avoid these round-trips all together and make the whole thing more responsive for the user.

Does that make sense? I'm happy to share more details, but trying to keep it as generic as possible here.

[nbenn]

(If this is only about where in the docs this should go, I have no opinion. For me the only thing that counts is that the capability is available. Finding things that are not well documented also has gotten a lot easier recently 😃)

[gadenbuie]

I agree with you @cpsievert on placement. I removed the section from both getting-started docs in cb222ed. The Python docstring already had a full example, and I added the same one to the chat_server() docs in R, so the API reference now carries the feature on its own.

I think this feature would play nicely with tidyverse/ellmer#369, especially for injected per-turn context, and if you're okay with the cache prefix lagging by a turn.

Speaking of ellmer features though, ellmer v0.5.0 just landed with a new chat$on_request_start() hook tidyverse/ellmer#1052, which technically provides another entry point into rewriting or appending content in the latest turn. That said, I still feel like this is a reasonable feature for us to provide in shinychat. It's always bugged me a little bit that we don't have a developer-facing hook here.

[cpsievert]

I'd prefer the implementation here to be closer to how transform_assistant_response looks:

1. Overloads so that both @transform_user_input and @transform_user_input() is supported.
2. Stronger types in the input values. I think I'll also prefer that the decorated function signature looks like on_user_submit, meaning the 1st param is str and 2nd is list[Attachment] (see UserSubmitFunction2)

[gadenbuie]

Point 1 is mechanical and easy (a995419), but I'm seeing the role of transform_user_input() differently from on_user_submit() and I think it's worth talking through it.

My take is that on_user_submit and transform_user_input are intended to solve different problems, and matching their signatures might not be good for chatlas + shinychat users.

If you're using shinychat without chatlas, or if you're handling all the server logic yourself, on_user_submit is the "roll your own" approach: the callback receives the user-facing view of the input (text, Attachment objects — the same things the UI shows) and owns everything that happens next, including building the contents and calling stream_async() itself.

transform_user_input exists precisely because the client= path takes that ownership away: Chat(client=...) registers its own on_user_submit that converts (text, attachments) into chatlas contents and calls stream_async(*contents). The whole value of the transform hook is letting the user reach into the chatlas content layer — prepending retrieved context chunks, inserting images or structured content — without having to opt out of the automatic client wiring. If the transform took (text, attachments), the user would be back to string concatenation, which is exactly what a chatlas user is trying to avoid; and the return would have to be either a loose list[Any] anyway (so you gain nothing) or a (text, attachments) tuple (so you lose content injection entirely).

[gadenbuie]

I have to say neither option is satisfying. Solving this right for both Python and R is trickier than I anticipated and the more I think about it the more it feels like something that should be solved with client callbacks, not shinychat callbacks.

ellmer's on_request_start() callback is very close, but it also isn't the best approach either because it doesn't differentiate between user-initiated and within-tool-loop requests. Maybe the request should really be to have a way to register an on_request_start() callback that only fires for user-initiated turns. Or an on_chat_start() callback in both languages. I didn't make that callback in ellmer because an ellmer developer is calling $chat() or $stream() or similar directly, so I reasoned that it was the job of whatever software was wrapping ellmer to provide an appropriate lifecycle callback. But this example could be evidence to rethink that.

[gadenbuie]

ellmer's on_request_start() callback is very close, but it also isn't the best approach either because it doesn't differentiate between user-initiated and within-tool-loop requests

I should point out that that it is possible to differentiate but the dev has to do some extra work: the test is that pure user input turns don't contain tool results.