diff --git a/demos/structured-outputs/README.md b/demos/structured-outputs/README.md
new file mode 100644
index 0000000..e02e100
--- /dev/null
+++ b/demos/structured-outputs/README.md
@@ -0,0 +1,97 @@
+# Structured Outputs
+
+A working demonstration of `llm:chat-with-schema`, `llm:chat-json`, and `llm:get` —
+constraining a model's reply to a JSON Schema and reading the fields as real NetLogo
+values instead of parsing a sentence.
+
+## The Problem
+
+`llm:chat` returns free text. A model that needs a number has to hope the phrasing
+stays stable and pick it out of the string:
+
+```netlogo
+let reply llm:chat "How confident are you, 0 to 1?"
+;=> "I'd say about 0.8, though it depends on the situation."
+; now what? substring? position? what if it says "eighty percent"?
+```
+
+That works until the model words things differently, and then it fails quietly.
+
+## The Fix
+
+```netlogo
+let schema "{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\",\"enum\":[\"eat\",\"explore\",\"rest\"]},\"confidence\":{\"type\":\"number\"}},\"required\":[\"action\",\"confidence\"]}"
+
+let reply llm:chat-with-schema "A turtle sees food. What now?" schema
+;=> [[action eat] [confidence 0.9]]
+
+let act llm:get reply "action" ;=> "eat" constrained to the enum
+let conf llm:get reply "confidence" ;=> 0.9 a NUMBER, not text
+```
+
+## What the demo shows
+
+Six turtles forage. Each tick every turtle sends its energy and whether it is
+standing on food, and gets back a reply matching the schema above.
+
+Two consequences are visible on screen:
+
+- **`action` is enum-constrained**, so `act-on` compares it directly. No fuzzy
+ matching, no fallback branch for unexpected wording.
+- **`confidence` is a number**, so `recolor` writes `confidence >= 0.7`. Turtles
+ above the threshold turn lime, the rest orange. That comparison is only possible
+ because the value arrives typed.
+
+Press **show decisions** for each turtle's parsed fields, or **raw JSON (no schema)**
+to see what `llm:chat-json` gives instead — valid JSON, but still a string.
+
+## How to Run
+
+1. `cp config.txt.example config.txt` and add your key. `config.txt` is gitignored.
+2. Open `structured-outputs.nlogox` in NetLogo 7.0.3.
+3. Press **setup**, then **go**.
+
+Structured output needs a provider that supports constrained decoding. Verified
+against Groq; OpenAI, Anthropic, and Gemini also support it. Ollama depends on the
+local model.
+
+## How JSON maps into NetLogo
+
+NetLogo has no dictionary type, so objects become `[key value]` pair lists:
+
+| JSON | NetLogo |
+|---|---|
+| `{"a": 1}` | `[[a 1]]` |
+| `[1, 2]` | `[1 2]` |
+| `"text"` | string |
+| `10` | number |
+| `true` | boolean |
+| `null` | `""` (NetLogo has no null) |
+
+Nested objects are just more pair lists, so `llm:get` chains:
+
+```netlogo
+let stats llm:get reply "stats" ;=> [[alive true] [speed 10]]
+let speed llm:get stats "speed" ;=> 10
+```
+
+## Gotchas
+
+- **The schema is a JSON string, not a NetLogo list.** The escaped quotes are
+ unavoidable. Passing a list raises an error naming the problem.
+- **A missing key raises**, listing the keys that were available. Wrap `llm:get` in
+ `carefully` when a field is genuinely optional.
+- **Key matching is exact and case-sensitive**, because JSON keys are.
+- **The schema constrains shape, not truth.** A well-formed reply can still be a bad
+ decision — this removes parsing failures, not model error.
+
+## Verification
+
+Run headless against live Groq: 12 schema-constrained calls across 2 ticks, 0
+failures. Turtles standing on food chose `eat` at 0.9 confidence while others chose
+`explore` at 0.8 — the replies track state rather than repeating a default.
+
+## Related
+
+- API reference: `docs/API-REFERENCE.md` → Structured Output
+- Issue [#22](https://github.com/NetLogo/Netlogo-LLM-Extension/issues/22)
diff --git a/demos/structured-outputs/config.txt.example b/demos/structured-outputs/config.txt.example
new file mode 100644
index 0000000..a9675e7
--- /dev/null
+++ b/demos/structured-outputs/config.txt.example
@@ -0,0 +1,27 @@
+# Structured outputs demo — provider settings
+#
+# Copy this file to config.txt and fill in a key. config.txt is gitignored,
+# so your key stays out of version control.
+#
+# Structured output needs a provider that supports constrained decoding.
+# Verified working: Groq, OpenAI, Anthropic, Gemini. Ollama support depends
+# on the local model.
+
+# --- Groq (free tier, fast; key from https://console.groq.com/keys) ---
+provider=groq
+groq_api_key=gsk_REPLACE_WITH_YOUR_KEY
+model=openai/gpt-oss-20b
+
+# --- OpenAI ---
+#provider=openai
+#openai_api_key=sk-REPLACE_WITH_YOUR_KEY
+#model=gpt-4o-mini
+
+# --- Local Ollama, no key needed ---
+#provider=ollama
+#model=llama3.2:3b
+#base_url=http://localhost:11434
+
+temperature=0.0
+max_tokens=300
+timeout_seconds=60
diff --git a/demos/structured-outputs/structured-outputs.nlogox b/demos/structured-outputs/structured-outputs.nlogox
new file mode 100644
index 0000000..bb102bb
--- /dev/null
+++ b/demos/structured-outputs/structured-outputs.nlogox
@@ -0,0 +1,270 @@
+
+
+ = 0.7) [lime] [orange]
+end
+
+;; Prints each turtle's most recent typed decision.
+to show-decisions
+ clear-output
+ output-print "action confidence colour"
+ ask turtles [
+ output-print (word
+ last-action
+ " " precision confidence 2
+ " " (ifelse-value (confidence >= 0.7) ["lime"] ["orange"]))
+ ]
+ output-print ""
+ output-print (word "last parsed reply: " last-reply)
+end
+
+;; The no-schema path: valid JSON, returned as raw text.
+to show-raw-json
+ clear-output
+ carefully
+ [ let raw llm:chat-json
+ "Name two colours a turtle could be, as a JSON object with key colours."
+ output-print "llm:chat-json returns a STRING:"
+ output-print raw
+ output-print (word "is-string? " is-string? raw) ]
+ [ output-print (word "failed: " error-message) ]
+end
+]]>
+
+
+
+
+
+
+
+ parsed-count
+ failed-count
+
+ Lime turtles reported confidence of 0.7 or more, orange ones less. That comparison works because confidence arrives as a number, not as text to be parsed.
+
+ [[action eat] [confidence 0.9] [reason found food here]]
+
+ let act llm:get reply "action" ;=> "eat" a string
+ let conf llm:get reply "confidence" ;=> 0.9 a NUMBER
+
+Two things follow, and both are visible on screen:
+
+- `action` is constrained by the schema's `enum`, so `act-on` compares it directly
+ with no fuzzy matching or fallback branch.
+- `confidence` arrives as a number, so `recolor` can write `confidence >= 0.7`.
+ Turtles above the threshold are lime, the rest orange.
+
+## HOW TO USE IT
+
+1. Put your provider settings in `config.txt` beside this model.
+2. Press **setup**, then **go**.
+3. Press **show decisions** to see each turtle's typed fields.
+4. Press **raw JSON (no schema)** to see what `llm:chat-json` returns instead —
+ valid JSON, but as a string you would still have to handle yourself.
+
+## THINGS TO NOTICE
+
+The schema is a JSON **string**, not a NetLogo list. The escaped quotes in `setup`
+are unavoidable, and passing a list instead raises an error.
+
+Nesting has no dotted-path syntax. A nested object is another `[[key value] ...]`
+list, so you call `llm:get` again on the result.
+
+A missing key raises rather than reporting a default, and names the keys that were
+available. Wrap `llm:get` in `carefully` when a field is genuinely optional.
+
+`null` in JSON becomes an empty string, because NetLogo has no null.
+
+## EXTENDING THE MODEL
+
+Add a field to `decision-schema` — say a `target` object with `xcor` and `ycor` —
+and read it with a second `llm:get`. Nothing else has to change.
+
+The **failed** monitor counts replies that did not parse. Schema constraint is
+enforced by the provider, so this counts network and model failures rather than
+malformed JSON; a run that only ever increments **parsed** is the expected case.
+
+## RELATED PRIMITIVES
+
+`llm:chat-with-schema` — schema-constrained reply, parsed into nested lists
+`llm:chat-json` — valid JSON with no schema, returned as a string
+`llm:get` — read a key from a `[[key value] ...]` list
+]]>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/API-REFERENCE.md b/docs/API-REFERENCE.md
index fad341f..64b1399 100644
--- a/docs/API-REFERENCE.md
+++ b/docs/API-REFERENCE.md
@@ -13,6 +13,9 @@ The NetLogo Multi-LLM Extension provides a unified interface for multiple Large
| `llm:chat-with-template file vars` | Chat | Send templated prompt with variable substitution |
| `llm:chat-with-thinking text` | Chat | Returns `[answer thinking]` for reasoning-capable models |
| `llm:choose prompt choices` | Chat | Force selection from provided options |
+| `llm:chat-with-schema prompt schema` | Structured | Schema-constrained chat, returns parsed nested lists |
+| `llm:chat-json prompt` | Structured | Force valid JSON output, returns the raw JSON string |
+| `llm:get parsed key` | Structured | Look up a key in a `[[key value] ...]` list |
| `llm:compile-error code` | Validation | `""` if valid, else compiler error; optional disallowed list |
| `llm:set-thinking bool` | Reasoning | Enable/disable reasoning mode for current provider |
| `llm:set-reasoning-effort level` | Reasoning | Set effort: `"low"`, `"medium"`, `"high"` |
@@ -235,6 +238,117 @@ set color read-from-string color-choice
- Forces LLM to return exactly one of the provided choices
- Useful for agent decision-making in models
- Maintains conversation context
+- Where the provider supports constrained decoding, the option list is sent as a
+ schema constraint so the model cannot reply with anything else. Providers that
+ do not support it fall back to the prompt wording, and the reply is still
+ matched case-insensitively against the list — so behaviour is unchanged, just
+ more reliable where the constraint is available.
+- Still raises an error if the reply matches no option. Wrap in `carefully` if a
+ model should tolerate that.
+
+## Structured Output Primitives
+
+These constrain the *shape* of a reply at the API level rather than by asking
+nicely in the prompt. See [Structured Output](#structured-output-details) below
+for the JSON-to-NetLogo mapping and provider support.
+
+### llm:chat-with-schema
+
+**Syntax**: `llm:chat-with-schema prompt schema`
+
+**Description**: Sends a prompt with the reply constrained to a JSON Schema, and
+reports the parsed result as nested NetLogo lists.
+
+**Parameters**:
+
+- `prompt` (string): The question or context
+- `schema` (string): A JSON Schema object, as a string
+
+**Returns**: List — the parsed JSON. Objects arrive as `[[key value] ...]` pairs;
+use `llm:get` to read fields.
+
+**Example**:
+
+```netlogo
+let schema "{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\"},\"confidence\":{\"type\":\"number\"}}}"
+let result llm:chat-with-schema "What should the agent do?" schema
+let action llm:get result "action" ; "explore"
+let conf llm:get result "confidence" ; 0.85
+```
+
+**Notes**:
+
+- The schema's structure is checked before any request is sent — that it is a
+ JSON object, that every level declares a supported `type`, that objects have
+ non-empty `properties`, that arrays declare `items`, and that any `enum` is a
+ non-empty array. A problem in any of those fails immediately with a message
+ naming the offending path, rather than as an opaque provider error after a
+ network round-trip.
+- Other JSON Schema keywords are passed through to the provider unchecked. The
+ four providers accept different dialects, so validating every keyword here
+ would reject schemas a given provider would have honoured; a keyword one
+ provider rejects still surfaces as that provider's error.
+- Object schemas are normalized to strict mode (`additionalProperties: false`,
+ every property listed in `required`) because OpenAI and Anthropic require it.
+ The same normalization is applied for every provider so one schema behaves
+ the same everywhere.
+- If the reply is not valid JSON, this raises an error and conversation history
+ is left unchanged — a failed call never records a bogus exchange.
+
+### llm:chat-json
+
+**Syntax**: `llm:chat-json prompt`
+
+**Description**: Sends a prompt with the reply constrained to valid JSON, with no
+schema, and reports the raw JSON string.
+
+**Returns**: String — the JSON text
+
+**Example**:
+
+```netlogo
+let json-str llm:chat-json "List three actions as a JSON array"
+```
+
+**Notes**:
+
+- Use `llm:chat-with-schema` when the shape matters and you want NetLogo values
+ back; use this when you want the JSON text itself.
+- The reply is checked to be valid JSON before it is reported. Not every
+ provider can enforce schemaless JSON natively — Anthropic has no such mode —
+ so a model that replies with prose raises an error rather than returning it,
+ and conversation history is left unchanged.
+- Any JSON value is accepted, not just objects: an array or a scalar is still
+ valid JSON. The text is reported exactly as received, not reformatted.
+
+### llm:get
+
+**Syntax**: `llm:get parsed key`
+
+**Description**: Looks up a key in a list of `[key value]` pairs — the shape
+`llm:chat-with-schema` reports.
+
+**Parameters**:
+
+- `parsed` (list): A list of `[key value]` pairs
+- `key` (string): The key to find
+
+**Returns**: The value at that key — a string, number, boolean, or nested list
+
+**Example**:
+
+```netlogo
+let result llm:chat-with-schema "Describe a turtle" schema
+let name llm:get result "name"
+let city llm:get (llm:get result "address") "city" ; nested lookup
+```
+
+**Notes**:
+
+- Matching is exact and case-sensitive, because JSON keys are.
+- A missing key raises an error listing the available keys. It does not report a
+ blank — a silent `""` would be indistinguishable from a JSON null and would let
+ a typo travel through a run as data.
### llm:chat-with-template
@@ -744,6 +858,85 @@ print llm:list-models ; Shows all providers, with Anthropic marked as ACTIVE
- Custom models added via `models-override.yaml` are marked with `[custom]`
- The currently active provider and model are marked with `[ACTIVE]`
+## Structured Output Details
+
+### Why JSON becomes nested lists
+
+NetLogo's type system has strings, numbers, booleans, lists, and agents — there
+is no map or dictionary type. A JSON object therefore arrives as a list of
+`[key value]` pairs, which is the association-list shape NetLogo modelers
+already use and which `llm:get` can search.
+
+| JSON | NetLogo | Example |
+| ----------- | --------------------------- | ------------------------------ |
+| Object `{}` | List of `[key value]` pairs | `[["name" "Alice"] ["age" 30]]` |
+| Array `[]` | List | `["a" "b" "c"]` |
+| String | String | `"hello"` |
+| Number | Number | `42`, `3.14` |
+| Boolean | Boolean | `true`, `false` |
+| Null | Empty string | `""` |
+
+Null maps to `""` because NetLogo has no null, and `""` is a value a model can
+compare against without a runtime error.
+
+### Provider support
+
+Each provider expresses the same constraint in its own request field:
+
+| Provider | Field | Schemaless JSON mode |
+| ----------------------------------- | -------------------------------------- | --------------------------- |
+| OpenAI, Groq, Together, OpenRouter | `response_format.json_schema.schema` | `response_format.json_object` |
+| Anthropic (Claude) | `output_config.format.schema` | Prompt instruction only |
+| Gemini | `generationConfig.responseJsonSchema` | `responseMimeType` only |
+| Ollama | `format` | `format: "json"` |
+
+Notes on the two asymmetries:
+
+- **Anthropic has no schemaless JSON mode.** For `llm:chat-json`, no
+ `output_config.format` is sent — inventing one would be rejected by the API —
+ so the JSON instruction is carried in the prompt instead. `llm:chat-with-schema`
+ uses the native schema field.
+- **Claude's `output_config` is shared.** Thinking depth (`effort`) and response
+ format (`format`) are siblings under one key, so both are merged into the same
+ object. Setting a schema does not disturb an existing `effort`, and vice versa.
+
+Whether a *particular model* honours the constraint is decided by the provider,
+not this extension. Where a model ignores it, the reply is still returned and
+`llm:choose` still matches it against the option list — nothing fails merely
+because constrained decoding was unavailable.
+
+### Scope of this release
+
+`llm:set-response-format` and `llm:clear-response-format` — a persistent schema
+applied to ordinary `llm:chat` calls — are **not** included. Two reasons:
+
+1. `llm:chat` declares a `StringType` return in its syntax. A persistent schema
+ would make it report JSON-as-a-string with no signal at the type level, so
+ model code could not tell which shape it was about to get.
+2. The format would live in global config while conversation history is
+ per-agent, so a schema set for one agent would silently apply to every other
+ agent's calls mid-run.
+
+The one-shot primitives (`llm:chat-with-schema`, `llm:chat-json`) express the
+same capability without either problem: the constraint and the return type are
+visible at the call site. If a persistent form is added later, per-agent scoping
+and a distinct return type are the open design questions.
+
+Generalizing `llm:choose` to non-string results (numbers, coordinates, ordered
+plans) is likewise not a new set of primitives here — those are schema shapes for
+`llm:chat-with-schema`. For example, a multi-step plan:
+
+```netlogo
+let schema "{\"type\":\"object\",\"properties\":{\"steps\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}}}}"
+let plan llm:get (llm:chat-with-schema "Plan three moves" schema) "steps"
+foreach plan [ s -> print s ]
+```
+
+If the steps are meant to be executed as NetLogo code rather than printed, check
+each one with `llm:compile-error` first and keep `carefully` around the `run` —
+a schema constrains the *shape* of a reply, not whether its contents are safe or
+runnable.
+
## Usage Patterns
### Basic Chat Bot
diff --git a/src/main/LLMExtension.scala b/src/main/LLMExtension.scala
index 450131c..54f2cd5 100644
--- a/src/main/LLMExtension.scala
+++ b/src/main/LLMExtension.scala
@@ -4,7 +4,8 @@ import org.nlogo.api._
import org.nlogo.core.{LogoList, Syntax}
import org.nlogo.extensions.llm.config.{ConfigLoader, ConfigStore}
import org.nlogo.extensions.llm.providers.{LLMProvider, ProviderDescriptor, ProviderFactory, ProviderRegistry, ProviderRegistrations, ModelRegistry, OllamaProvider, ReadinessCheck, RetryPolicy}
-import org.nlogo.extensions.llm.models.{ChatMessage, ChatResponse}
+import org.nlogo.extensions.llm.models.{ChatMessage, ChatResponse, EnumFormat, JsonObjectFormat, ResponseFormat}
+import org.nlogo.extensions.llm.utils.JsonToNetLogo
import scala.collection.mutable.{ArrayBuffer, WeakHashMap}
import scala.concurrent.{Await, ExecutionContext, Future}
import scala.concurrent.duration._
@@ -102,6 +103,11 @@ class LLMExtension extends DefaultClassManager {
manager.addPrimitive("chat-with-thinking", ChatWithThinkingReporter)
manager.addPrimitive("choose", ChooseReporter)
+ // Structured output primitives
+ manager.addPrimitive("chat-with-schema", ChatWithSchemaReporter)
+ manager.addPrimitive("chat-json", ChatJsonReporter)
+ manager.addPrimitive("get", GetReporter)
+
// Code validation primitive
manager.addPrimitive("compile-error", CompileErrorReporter)
@@ -621,8 +627,10 @@ class LLMExtension extends DefaultClassManager {
tempHistory.prepend(ChatMessage.system(systemPrompt))
tempHistory += ChatMessage.user(userPrompt)
- // Use chatWithFullResponse to access thinking field for thinking models
- val responseFuture = provider.chatWithFullResponse(tempHistory.toSeq)
+ // Constrain the reply to the choice list where the provider supports it.
+ // The prompt still spells out the options, so a provider that ignores
+ // the constraint behaves exactly as it did before.
+ val responseFuture = provider.chatWithFormat(tempHistory.toSeq, EnumFormat(choices))
val response = Await.result(responseFuture, getAwaitTimeout)
// Extract text: prefer content, fall back to thinking field
@@ -631,11 +639,16 @@ class LLMExtension extends DefaultClassManager {
.getOrElse("")
.trim
+ // A constrained reply arrives as {"choice": "..."} while an unconstrained
+ // one is the bare option text. Accept both: which shape comes back
+ // depends on provider support, and the modeler asked for neither.
+ val candidate = extractEnumChoice(text).getOrElse(text)
+
// Exact match only (case-insensitive)
- val chosenOption = choices.find(_.equalsIgnoreCase(text))
+ val chosenOption = choices.find(_.equalsIgnoreCase(candidate))
.getOrElse {
throw new ExtensionException(
- s"llm:choose: response '$text' did not match any choice. " +
+ s"llm:choose: response '$candidate' did not match any choice. " +
s"Choices: ${choices.mkString(", ")}"
)
}
@@ -652,6 +665,194 @@ class LLMExtension extends DefaultClassManager {
}
}
+ /**
+ * Read the selected option out of an enum-constrained reply.
+ *
+ * A provider enforcing the constraint returns `{"choice": "north"}`; one that
+ * ignores it returns `north`. Returns None for anything that is not the
+ * constrained shape, so the caller can fall back to the raw text.
+ */
+ private def extractEnumChoice(text: String): Option[String] =
+ scala.util.Try {
+ ujson.read(text)(EnumFormat.ChoiceKey).str
+ }.toOption
+
+ // Structured Output Primitives
+
+ /**
+ * Chat with the reply constrained to a JSON Schema, reported as nested lists.
+ *
+ * llm:chat-with-schema prompt schema-string
+ *
+ * NetLogo has no map type, so the parsed JSON object arrives as a list of
+ * `[key value]` pairs — use `llm:get` to read fields out of it.
+ *
+ * The schema is validated before any request is sent: an unusable schema
+ * otherwise surfaces as an opaque provider 400 after a network round-trip,
+ * which a modeler cannot act on.
+ */
+ object ChatWithSchemaReporter extends Reporter {
+ override def getSyntax: Syntax = Syntax.reporterSyntax(
+ right = List(Syntax.StringType, Syntax.StringType),
+ ret = Syntax.ListType
+ )
+
+ override def report(args: Array[Argument], context: Context): AnyRef = {
+ val inputText = args(0).getString
+ val schemaText = args(1).getString
+ val agent = context.getAgent
+
+ // Validate before touching the provider so a bad schema costs nothing.
+ val format =
+ try ResponseFormat.parseSchema(schemaText)
+ catch {
+ case e: IllegalArgumentException =>
+ throw new ExtensionException(s"llm:chat-with-schema: ${e.getMessage}")
+ }
+
+ // This reporter's syntax promises a list, and only a JSON object converts
+ // to the [key value] pairs llm:get reads. A top-level scalar or array
+ // schema would report a bare string or a flat list instead, breaking that
+ // promise, so it is rejected here rather than at the provider.
+ val topLevelType = format.schema.value.get("type").collect { case ujson.Str(s) => s }
+ if (!topLevelType.contains("object")) {
+ throw new ExtensionException(
+ s"llm:chat-with-schema: schema must have type 'object' at the top level, but got " +
+ s"'${topLevelType.getOrElse("none")}'. llm:chat-with-schema reports a list of " +
+ "[key value] pairs, so the reply has to be a JSON object. Wrap it, e.g. " +
+ """{"type":"object","properties":{"value":{"type":"string"}}}"""
+ )
+ }
+
+ try {
+ val provider = ensureProvider()
+
+ val userMessage = ChatMessage.user(inputText)
+
+ val responseFuture = provider.chatWithFormat(snapshotHistory(agent) :+ userMessage, format)
+ val response = Await.result(responseFuture, getAwaitTimeout)
+
+ val content = response.firstContent.getOrElse("")
+
+ // Parse before committing: a reply that is not JSON means the call did
+ // not deliver what was asked for, so history must not record it as a
+ // successful exchange.
+ val parsed =
+ try JsonToNetLogo.parseObject(content)
+ catch {
+ case e: IllegalArgumentException =>
+ throw new ExtensionException(s"llm:chat-with-schema: ${e.getMessage}")
+ }
+
+ commitExchange(agent, userMessage, ChatMessage.assistant(content))
+
+ parsed
+
+ } catch {
+ case e: Exception if !e.isInstanceOf[ExtensionException] =>
+ throw new ExtensionException(s"llm:chat-with-schema failed: ${e.getMessage}")
+ }
+ }
+ }
+
+ /**
+ * Chat with the reply constrained to valid JSON, reported as a raw string.
+ *
+ * llm:chat-json prompt
+ *
+ * Reports the JSON text rather than parsed lists, for modelers who want to
+ * hand it to another tool or inspect it directly. Use `llm:chat-with-schema`
+ * when the shape matters and you want NetLogo values back.
+ */
+ object ChatJsonReporter extends Reporter {
+ override def getSyntax: Syntax = Syntax.reporterSyntax(
+ right = List(Syntax.StringType),
+ ret = Syntax.StringType
+ )
+
+ override def report(args: Array[Argument], context: Context): AnyRef = {
+ val inputText = args(0).getString
+ val agent = context.getAgent
+
+ try {
+ val provider = ensureProvider()
+
+ // Anthropic has no schemaless JSON mode and Gemini's mime type alone is
+ // only a hint, so the instruction is also stated in the prompt. Providers
+ // that do enforce JSON natively are unaffected by the extra sentence.
+ val jsonInstruction = ChatMessage.system(
+ "Respond with valid JSON only. No prose, no markdown code fences."
+ )
+ val userMessage = ChatMessage.user(inputText)
+
+ val tempHistory = ArrayBuffer.from(snapshotHistory(agent))
+ tempHistory.prepend(jsonInstruction)
+ tempHistory += userMessage
+
+ val responseFuture = provider.chatWithFormat(tempHistory.toSeq, JsonObjectFormat)
+ val response = Await.result(responseFuture, getAwaitTimeout)
+
+ val content = response.firstContent.getOrElse("")
+
+ // Not every provider can enforce schemaless JSON natively — Anthropic
+ // has no such mode at all — so the reply is verified here rather than
+ // trusted. Checking before the commit keeps a failed call out of
+ // history, and stops prose being reported from a primitive whose whole
+ // contract is that the result parses as JSON. Any JSON value is
+ // accepted: an array or scalar is still valid JSON.
+ try ujson.read(content)
+ catch {
+ case e: Exception =>
+ throw new ExtensionException(
+ s"llm:chat-json: Response was not valid JSON: ${e.getMessage}. Response text: $content"
+ )
+ }
+
+ // Store the clean exchange only — the JSON instruction is prompt
+ // scaffolding, not part of the conversation.
+ commitExchange(agent, userMessage, ChatMessage.assistant(content))
+
+ content
+
+ } catch {
+ case e: Exception if !e.isInstanceOf[ExtensionException] =>
+ throw new ExtensionException(s"llm:chat-json failed: ${e.getMessage}")
+ }
+ }
+ }
+
+ /**
+ * Look up a key in a list of `[key value]` pairs.
+ *
+ * llm:get parsed-result "key"
+ *
+ * Throws on a missing key rather than reporting a sentinel: a silent "" would
+ * be indistinguishable from a JSON null and would let a typo propagate as data
+ * through the rest of a run.
+ */
+ object GetReporter extends Reporter {
+ override def getSyntax: Syntax = Syntax.reporterSyntax(
+ right = List(Syntax.ListType, Syntax.StringType),
+ ret = Syntax.WildcardType
+ )
+
+ override def report(args: Array[Argument], context: Context): AnyRef = {
+ val list = args(0).getList
+ val key = args(1).getString
+
+ JsonToNetLogo.lookup(list, key).getOrElse {
+ val available = list.toVector.collect {
+ case pair: LogoList if pair.size == 2 => pair(0).toString
+ }
+ val detail =
+ if (list.size == 0) "The list is empty."
+ else if (available.isEmpty) "Available keys: (none - the list is not a list of [key value] pairs)"
+ else s"Available keys: ${available.mkString(", ")}"
+ throw new ExtensionException(s"""llm:get: key "$key" not found. $detail""")
+ }
+ }
+ }
+
// Thinking/Reasoning Primitives
object ChatWithThinkingReporter extends Reporter {
diff --git a/src/main/models/ChatRequest.scala b/src/main/models/ChatRequest.scala
index dad09c9..bed533b 100644
--- a/src/main/models/ChatRequest.scala
+++ b/src/main/models/ChatRequest.scala
@@ -9,13 +9,18 @@ import upickle.default.{ReadWriter => RW, macroRW}
* @param messages The conversation history as a sequence of messages
* @param maxTokens Optional maximum number of tokens to generate
* @param temperature Optional temperature for response randomness (0.0-2.0)
+ * @param thinkingConfig Optional reasoning/thinking configuration
+ * @param responseFormat Optional constraint on the shape of the reply. Each
+ * provider wraps this in its own request field; None
+ * leaves the request byte-for-byte as before.
*/
case class ChatRequest(
model: String,
messages: Seq[ChatMessage],
maxTokens: Option[Int] = None,
temperature: Option[Double] = None,
- thinkingConfig: Option[ThinkingConfig] = None
+ thinkingConfig: Option[ThinkingConfig] = None,
+ responseFormat: Option[ResponseFormat] = None
)
object ChatRequest {
diff --git a/src/main/models/ResponseFormat.scala b/src/main/models/ResponseFormat.scala
new file mode 100644
index 0000000..09e37f3
--- /dev/null
+++ b/src/main/models/ResponseFormat.scala
@@ -0,0 +1,268 @@
+// ABOUTME: Response-format model describing how an LLM's output should be constrained
+// ABOUTME: Parses and normalizes modeler-supplied JSON Schema strings into a provider-neutral form
+package org.nlogo.extensions.llm.models
+
+/**
+ * How a provider should constrain the shape of its reply.
+ *
+ * Provider-neutral by design: each provider decides how to wrap these in its own
+ * request body (see the `applyResponseFormat` hooks in the providers package).
+ */
+sealed trait ResponseFormat
+
+/**
+ * Constrain output to a JSON Schema.
+ *
+ * `schema` is always a JSON object, validated at construction by
+ * [[ResponseFormat.parseSchema]]. Providers that require strict-mode schemas
+ * pass it through [[ResponseFormat.strictSchema]] first.
+ */
+case class JsonSchemaFormat(schema: ujson.Obj, name: String = ResponseFormat.DefaultSchemaName)
+ extends ResponseFormat
+
+/**
+ * Constrain output to exactly one of a fixed list of strings.
+ *
+ * Rendered as a one-property object schema rather than a bare `{"enum": [...]}`,
+ * because the providers that support constrained decoding all accept an object
+ * schema, while a top-level enum is accepted only by some. One shape works
+ * everywhere, so `llm:choose` behaves the same regardless of provider.
+ */
+case class EnumFormat(choices: List[String]) extends ResponseFormat {
+ require(choices.nonEmpty, "Choice list cannot be empty")
+
+ /** The enum rendered as an object schema with a single constrained property. */
+ def schema: ujson.Obj =
+ ujson.Obj(
+ "type" -> "object",
+ "properties" -> ujson.Obj(
+ EnumFormat.ChoiceKey -> ujson.Obj(
+ "type" -> "string",
+ "enum" -> ujson.Arr(choices.map(c => ujson.Str(c))*)
+ )
+ ),
+ "required" -> ujson.Arr(ujson.Str(EnumFormat.ChoiceKey)),
+ "additionalProperties" -> false
+ )
+}
+
+object EnumFormat {
+ /** Property name holding the selected option in an enum-constrained reply. */
+ val ChoiceKey: String = "choice"
+}
+
+/** Constrain output to syntactically valid JSON, with no schema. */
+case object JsonObjectFormat extends ResponseFormat
+
+object ResponseFormat {
+
+ /** Schema name sent to providers that require one (OpenAI). */
+ val DefaultSchemaName: String = "netlogo_schema"
+
+ /**
+ * Serialization for the sealed hierarchy.
+ *
+ * Hand-written rather than derived: ujson.Obj has no upickle ReadWriter, and
+ * ChatRequest's derived instance needs one for this field to exist at all.
+ */
+ implicit val rw: upickle.default.ReadWriter[ResponseFormat] =
+ upickle.default.readwriter[ujson.Value].bimap[ResponseFormat](
+ {
+ case JsonSchemaFormat(schema, name) =>
+ ujson.Obj("kind" -> "json_schema", "name" -> name, "schema" -> schema)
+ case EnumFormat(choices) =>
+ ujson.Obj("kind" -> "enum", "choices" -> ujson.Arr(choices.map(ujson.Str(_))*))
+ case JsonObjectFormat =>
+ ujson.Obj("kind" -> "json_object")
+ },
+ json =>
+ json("kind").str match {
+ case "json_schema" =>
+ JsonSchemaFormat(ujson.Obj.from(json("schema").obj), json("name").str)
+ case "enum" =>
+ EnumFormat(json("choices").arr.map(_.str).toList)
+ case "json_object" =>
+ JsonObjectFormat
+ case other =>
+ throw new IllegalArgumentException(s"Unknown response format kind: $other")
+ }
+ )
+
+ /**
+ * Parse and validate a modeler-supplied JSON Schema string.
+ *
+ * Validation is deliberately front-loaded: an unusable schema otherwise fails
+ * as an opaque provider 400 after a network round-trip, which a modeler cannot
+ * act on. Errors here name what is wrong with the schema they wrote.
+ *
+ * @throws IllegalArgumentException if the string is not a usable object schema
+ */
+ def parseSchema(schemaText: String): JsonSchemaFormat = {
+ if (schemaText == null || schemaText.trim.isEmpty) {
+ throw new IllegalArgumentException(
+ "Schema cannot be empty. Provide a JSON Schema object, e.g. " +
+ """{"type":"object","properties":{"action":{"type":"string"}}}"""
+ )
+ }
+
+ val parsed =
+ try ujson.read(schemaText)
+ catch {
+ case e: Exception =>
+ throw new IllegalArgumentException(
+ s"Schema is not valid JSON: ${e.getMessage}"
+ )
+ }
+
+ val obj = parsed match {
+ case o: ujson.Obj => o
+ case other =>
+ throw new IllegalArgumentException(
+ s"Schema must be a JSON object, but got ${typeName(other)}. " +
+ """Example: {"type":"object","properties":{"action":{"type":"string"}}}"""
+ )
+ }
+
+ validate(obj, path = "schema")
+ JsonSchemaFormat(obj)
+ }
+
+ /**
+ * Check a schema is one a provider can actually enforce.
+ *
+ * Only the constraints every supported provider shares are checked. Anything
+ * provider-specific stays out, so a schema accepted here is not silently
+ * rejected by a different provider later.
+ */
+ private def validate(schema: ujson.Obj, path: String): Unit = {
+ val typeValue = schema.value.get("type").map {
+ case ujson.Str(s) => s
+ case other =>
+ throw new IllegalArgumentException(
+ s"$path: 'type' must be a string, but got ${typeName(other)}"
+ )
+ }.getOrElse {
+ throw new IllegalArgumentException(
+ s"$path is missing a 'type' field. Every schema needs one, e.g. " +
+ """{"type":"object","properties":{...}}"""
+ )
+ }
+
+ // `enum` is checked because every provider requires a non-empty array here
+ // and `llm:choose` builds one itself. Other keywords are passed through
+ // unchecked on purpose: the four providers accept different JSON Schema
+ // dialects, so validating them all here would reject schemas a provider
+ // would have honoured.
+ schema.value.get("enum").foreach {
+ case arr: ujson.Arr =>
+ if (arr.value.isEmpty) {
+ throw new IllegalArgumentException(s"$path: 'enum' must list at least one value")
+ }
+ case other =>
+ throw new IllegalArgumentException(
+ s"$path: 'enum' must be an array of values, but got ${typeName(other)}. " +
+ """Example: {"type":"string","enum":["north","south"]}"""
+ )
+ }
+
+ typeValue match {
+ case "object" =>
+ val props = schema.value.get("properties") match {
+ case Some(o: ujson.Obj) => o
+ case Some(other) =>
+ throw new IllegalArgumentException(
+ s"$path: 'properties' must be a JSON object, but got ${typeName(other)}"
+ )
+ case None =>
+ throw new IllegalArgumentException(
+ s"$path of type 'object' is missing 'properties'. An object schema with no " +
+ "properties constrains nothing, so no provider can enforce it."
+ )
+ }
+ if (props.value.isEmpty) {
+ throw new IllegalArgumentException(
+ s"$path: 'properties' is empty. An object schema must declare at least one property."
+ )
+ }
+ props.value.foreach { case (key, value) =>
+ value match {
+ case o: ujson.Obj => validate(o, s"$path.properties.$key")
+ case other =>
+ throw new IllegalArgumentException(
+ s"$path.properties.$key must be a schema object, but got ${typeName(other)}"
+ )
+ }
+ }
+
+ case "array" =>
+ schema.value.get("items") match {
+ case Some(o: ujson.Obj) => validate(o, s"$path.items")
+ case Some(other) =>
+ throw new IllegalArgumentException(
+ s"$path.items must be a schema object, but got ${typeName(other)}"
+ )
+ case None =>
+ throw new IllegalArgumentException(
+ s"$path of type 'array' is missing 'items'. Declare the element schema, e.g. " +
+ """{"type":"array","items":{"type":"string"}}"""
+ )
+ }
+
+ case "string" | "number" | "integer" | "boolean" | "null" => ()
+
+ case other =>
+ throw new IllegalArgumentException(
+ s"$path: unsupported 'type' value '$other'. Supported: " +
+ "object, array, string, number, integer, boolean, null"
+ )
+ }
+ }
+
+ /**
+ * Return a strict-mode copy of a schema.
+ *
+ * OpenAI and Anthropic both reject schemas that omit `additionalProperties:
+ * false` or leave properties out of `required`. Applying the same
+ * normalization for every provider means one schema behaves identically
+ * everywhere, rather than a modeler's schema working on Ollama and 400ing on
+ * OpenAI.
+ *
+ * The input is never mutated — ujson values are mutable, and the caller's
+ * schema is reused across requests.
+ */
+ def strictSchema(schema: ujson.Value): ujson.Value = schema match {
+ case obj: ujson.Obj =>
+ val out = ujson.Obj()
+ obj.value.foreach {
+ case ("properties", props: ujson.Obj) =>
+ val strictProps = ujson.Obj()
+ props.value.foreach { case (k, v) => strictProps(k) = strictSchema(v) }
+ out("properties") = strictProps
+ case ("items", items) =>
+ out("items") = strictSchema(items)
+ case (k, v) =>
+ out(k) = v
+ }
+
+ if (obj.value.get("type").exists(_ == ujson.Str("object"))) {
+ out("additionalProperties") = false
+ obj.value.get("properties") match {
+ case Some(props: ujson.Obj) =>
+ out("required") = ujson.Arr(props.value.keys.map(k => ujson.Str(k)).toSeq*)
+ case _ => ()
+ }
+ }
+ out
+
+ case other => other
+ }
+
+ private def typeName(value: ujson.Value): String = value match {
+ case _: ujson.Obj => "an object"
+ case _: ujson.Arr => "an array"
+ case _: ujson.Str => "a string"
+ case _: ujson.Num => "a number"
+ case _: ujson.Bool => "a boolean"
+ case ujson.Null => "null"
+ }
+}
diff --git a/src/main/providers/BaseHttpProvider.scala b/src/main/providers/BaseHttpProvider.scala
index 1c80073..8997cdb 100644
--- a/src/main/providers/BaseHttpProvider.scala
+++ b/src/main/providers/BaseHttpProvider.scala
@@ -2,7 +2,7 @@
// ABOUTME: Reduces boilerplate by providing shared implementation of config, validation, and HTTP request handling
package org.nlogo.extensions.llm.providers
-import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse}
+import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, ResponseFormat}
import org.nlogo.extensions.llm.config.ConfigStore
import sttp.client4._
import sttp.client4.httpclient.HttpClientFutureBackend
@@ -160,7 +160,10 @@ abstract class BaseHttpProvider(implicit ec: ExecutionContext) extends LLMProvid
/**
* Build a ChatRequest from current config and messages, resolving ThinkingConfig
*/
- protected def buildRequest(messages: Seq[ChatMessage]): ChatRequest = {
+ protected def buildRequest(
+ messages: Seq[ChatMessage],
+ responseFormat: Option[ResponseFormat] = None
+ ): ChatRequest = {
val model = configStore.getOrElse(ConfigStore.MODEL, defaultModel)
val temperature = configStore.get(ConfigStore.TEMPERATURE).map(_.toDouble)
val maxTokens = configStore.get(ConfigStore.MAX_TOKENS).map(_.toInt)
@@ -171,7 +174,8 @@ abstract class BaseHttpProvider(implicit ec: ExecutionContext) extends LLMProvid
messages = messages,
maxTokens = maxTokens,
temperature = temperature,
- thinkingConfig = thinkingConfig
+ thinkingConfig = thinkingConfig,
+ responseFormat = responseFormat
)
}
@@ -194,6 +198,20 @@ abstract class BaseHttpProvider(implicit ec: ExecutionContext) extends LLMProvid
chat(request)
}
+ /**
+ * Chat with the reply constrained to a response format.
+ *
+ * The format travels on the ChatRequest; each provider's createProviderRequest
+ * decides how to express it on the wire.
+ */
+ override def chatWithFormat(
+ messages: Seq[ChatMessage],
+ format: ResponseFormat
+ ): Future[ChatResponse] = {
+ val request = buildRequest(messages, Some(format))
+ chat(request)
+ }
+
/**
* Send the actual HTTP request
*/
diff --git a/src/main/providers/ClaudeProvider.scala b/src/main/providers/ClaudeProvider.scala
index 1db0180..1302d1b 100644
--- a/src/main/providers/ClaudeProvider.scala
+++ b/src/main/providers/ClaudeProvider.scala
@@ -3,7 +3,7 @@
package org.nlogo.extensions.llm.providers
-import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse}
+import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, EnumFormat, JsonObjectFormat, JsonSchemaFormat, ResponseFormat}
import org.nlogo.extensions.llm.config.ConfigStore
import sttp.client4._
import sttp.model.Uri
@@ -115,7 +115,7 @@ class ClaudeProvider(implicit ec: ExecutionContext) extends BaseHttpProvider {
ClaudeModelCapabilities
.effortValue(request.thinkingConfig.flatMap(_.reasoningEffort))
.foreach { effort =>
- baseRequest("output_config") = ujson.Obj("effort" -> effort)
+ outputConfig(baseRequest)("effort") = effort
}
}
} else if (allowsSampling) {
@@ -124,9 +124,52 @@ class ClaudeProvider(implicit ec: ExecutionContext) extends BaseHttpProvider {
}
}
+ applyResponseFormat(baseRequest, request)
+
baseRequest
}
+ /**
+ * Get (creating if needed) the request's `output_config` object.
+ *
+ * `effort` and `format` are siblings under one key, so both writers must
+ * extend the same object. Assigning a fresh `ujson.Obj` from either would
+ * silently drop whatever the other had already set.
+ */
+ private def outputConfig(baseRequest: ujson.Obj): ujson.Obj =
+ baseRequest.value.get("output_config") match {
+ case Some(existing: ujson.Obj) => existing
+ case _ =>
+ val created = ujson.Obj()
+ baseRequest("output_config") = created
+ created
+ }
+
+ /**
+ * Add Anthropic's native `output_config.format`.
+ *
+ * Only schema-bearing formats are sent. Anthropic has no schemaless JSON mode,
+ * so [[JsonObjectFormat]] adds nothing here — inventing a `json_object` type
+ * would be rejected by the API. The prompt-level instruction added by the
+ * extension carries that case instead.
+ */
+ private def applyResponseFormat(baseRequest: ujson.Obj, request: ChatRequest): Unit =
+ request.responseFormat.foreach {
+ case JsonSchemaFormat(schema, _) =>
+ outputConfig(baseRequest)("format") = ujson.Obj(
+ "type" -> "json_schema",
+ "schema" -> ResponseFormat.strictSchema(schema)
+ )
+
+ case enumFormat: EnumFormat =>
+ outputConfig(baseRequest)("format") = ujson.Obj(
+ "type" -> "json_schema",
+ "schema" -> enumFormat.schema
+ )
+
+ case JsonObjectFormat => ()
+ }
+
override protected def parseProviderResponse(responseBody: String, model: String): ChatResponse = {
try {
val parsed = ujson.read(responseBody)
diff --git a/src/main/providers/GeminiProvider.scala b/src/main/providers/GeminiProvider.scala
index d44df27..3ff489d 100644
--- a/src/main/providers/GeminiProvider.scala
+++ b/src/main/providers/GeminiProvider.scala
@@ -3,7 +3,7 @@
package org.nlogo.extensions.llm.providers
-import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse}
+import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, EnumFormat, JsonObjectFormat, JsonSchemaFormat, ResponseFormat}
import org.nlogo.extensions.llm.config.ConfigStore
import sttp.client4._
import sttp.model.Uri
@@ -131,6 +131,22 @@ class GeminiProvider(implicit ec: ExecutionContext) extends BaseHttpProvider {
hasConfig = true
}
+ // Structured output lives in generationConfig alongside thinking, so both
+ // can be set on one request. responseMimeType is mandatory whenever JSON is
+ // wanted; responseJsonSchema is the newer standard-JSON-Schema field, which
+ // takes lowercase types exactly as the modeler wrote them.
+ request.responseFormat.foreach { format =>
+ generationConfig("responseMimeType") = "application/json"
+ hasConfig = true
+ format match {
+ case JsonSchemaFormat(schema, _) =>
+ generationConfig("responseJsonSchema") = ResponseFormat.strictSchema(schema)
+ case enumFormat: EnumFormat =>
+ generationConfig("responseJsonSchema") = enumFormat.schema
+ case JsonObjectFormat => ()
+ }
+ }
+
if (hasConfig) {
baseRequest("generationConfig") = generationConfig
}
diff --git a/src/main/providers/LLMProvider.scala b/src/main/providers/LLMProvider.scala
index 44b1770..5c7e2c3 100644
--- a/src/main/providers/LLMProvider.scala
+++ b/src/main/providers/LLMProvider.scala
@@ -1,6 +1,6 @@
package org.nlogo.extensions.llm.providers
-import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse}
+import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, ResponseFormat}
import scala.concurrent.Future
import scala.util.Try
@@ -36,6 +36,20 @@ trait LLMProvider {
*/
def chatWithFullResponse(messages: Seq[ChatMessage]): Future[ChatResponse]
+ /**
+ * Chat with the reply constrained to a response format.
+ *
+ * Defaults to an unconstrained request so existing providers keep compiling
+ * and behave exactly as before; providers that can enforce a format override
+ * this. A provider that ignores the format still returns a usable answer,
+ * which is the graceful-degradation behaviour the extension relies on.
+ *
+ * @param messages The conversation history
+ * @param format How the reply should be shaped
+ */
+ def chatWithFormat(messages: Seq[ChatMessage], format: ResponseFormat): Future[ChatResponse] =
+ chatWithFullResponse(messages)
+
/**
* Set a configuration parameter for this provider
*
diff --git a/src/main/providers/OllamaProvider.scala b/src/main/providers/OllamaProvider.scala
index 5e7ad54..06a995f 100644
--- a/src/main/providers/OllamaProvider.scala
+++ b/src/main/providers/OllamaProvider.scala
@@ -3,7 +3,7 @@
package org.nlogo.extensions.llm.providers
-import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse}
+import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, EnumFormat, JsonObjectFormat, JsonSchemaFormat, ResponseFormat}
import org.nlogo.extensions.llm.config.ConfigStore
import sttp.client4._
import sttp.model.Uri
@@ -55,6 +55,15 @@ class OllamaProvider(implicit ec: ExecutionContext) extends BaseHttpProvider {
baseRequest("think") = true
}
+ // Ollama takes the raw schema in `format`, or the bare string "json" for
+ // schemaless JSON mode. Enforcement is grammar-level at the sampling layer,
+ // so it is independent of `think`.
+ request.responseFormat.foreach {
+ case JsonSchemaFormat(schema, _) => baseRequest("format") = ResponseFormat.strictSchema(schema)
+ case enumFormat: EnumFormat => baseRequest("format") = enumFormat.schema
+ case JsonObjectFormat => baseRequest("format") = "json"
+ }
+
// Add options if parameters are specified
val options = ujson.Obj()
var hasOptions = false
diff --git a/src/main/providers/OpenAICompatibleProvider.scala b/src/main/providers/OpenAICompatibleProvider.scala
index dbf18f9..113fadd 100644
--- a/src/main/providers/OpenAICompatibleProvider.scala
+++ b/src/main/providers/OpenAICompatibleProvider.scala
@@ -2,7 +2,7 @@
// ABOUTME: Shared by OpenAI, OpenRouter, and Together AI — subclasses override hooks for headers, reasoning, and thinking
package org.nlogo.extensions.llm.providers
-import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, Choice}
+import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, Choice, EnumFormat, JsonObjectFormat, JsonSchemaFormat, ResponseFormat}
import org.nlogo.extensions.llm.config.ConfigStore
import sttp.client4._
import sttp.model.Uri
@@ -87,9 +87,42 @@ abstract class OpenAICompatibleProvider(implicit ec: ExecutionContext) extends B
}
}
+ applyResponseFormat(baseRequest, request)
+
baseRequest
}
+ /**
+ * Add the OpenAI `response_format` field.
+ *
+ * A schema is normalized to strict mode first: OpenAI rejects a schema that
+ * omits `additionalProperties: false` or leaves any property out of
+ * `required`, and applying the same normalization for every provider keeps one
+ * modeler-written schema working across all of them.
+ */
+ protected def applyResponseFormat(baseObj: ujson.Obj, request: ChatRequest): Unit =
+ request.responseFormat.foreach {
+ case JsonSchemaFormat(schema, name) =>
+ baseObj("response_format") = jsonSchemaField(name, ResponseFormat.strictSchema(schema))
+
+ case enumFormat: EnumFormat =>
+ baseObj("response_format") =
+ jsonSchemaField(ResponseFormat.DefaultSchemaName, enumFormat.schema)
+
+ case JsonObjectFormat =>
+ baseObj("response_format") = ujson.Obj("type" -> "json_object")
+ }
+
+ private def jsonSchemaField(name: String, schema: ujson.Value): ujson.Obj =
+ ujson.Obj(
+ "type" -> "json_schema",
+ "json_schema" -> ujson.Obj(
+ "name" -> name,
+ "strict" -> true,
+ "schema" -> schema
+ )
+ )
+
override protected def parseProviderResponse(responseBody: String, model: String): ChatResponse = {
try {
val parsed = ujson.read(responseBody)
diff --git a/src/main/utils/JsonToNetLogo.scala b/src/main/utils/JsonToNetLogo.scala
new file mode 100644
index 0000000..a925f96
--- /dev/null
+++ b/src/main/utils/JsonToNetLogo.scala
@@ -0,0 +1,105 @@
+// ABOUTME: Converts parsed JSON into NetLogo values, mapping objects to [key value] pair lists
+// ABOUTME: Also provides the key lookup that backs the llm:get primitive
+package org.nlogo.extensions.llm.utils
+
+import org.nlogo.core.LogoList
+
+/**
+ * Bridges JSON structures into NetLogo's type system.
+ *
+ * NetLogo has no map/dictionary type — only strings, numbers, booleans, lists,
+ * and agents. A JSON object therefore becomes a list of `[key value]` pairs,
+ * which is the shape NetLogo modelers already use for association lists and
+ * which [[lookup]] can search.
+ */
+object JsonToNetLogo {
+
+ /**
+ * Parse a JSON string and convert it to a NetLogo value.
+ *
+ * @throws IllegalArgumentException if the text is not valid JSON
+ */
+ def parse(jsonText: String): AnyRef = {
+ val parsed =
+ try ujson.read(jsonText)
+ catch {
+ case e: Exception =>
+ throw new IllegalArgumentException(
+ s"Response was not valid JSON: ${e.getMessage}. Response text: $jsonText"
+ )
+ }
+ convert(parsed)
+ }
+
+ /**
+ * Parse a JSON string that must be an object, and convert it.
+ *
+ * Callers that report a NetLogo list need the `[key value]` pair shape, which
+ * only a JSON object produces. A model that ignores its schema and replies
+ * with a bare scalar would otherwise yield a value of the wrong NetLogo type.
+ *
+ * @throws IllegalArgumentException if the text is not valid JSON or not an object
+ */
+ def parseObject(jsonText: String): LogoList = {
+ val parsed =
+ try ujson.read(jsonText)
+ catch {
+ case e: Exception =>
+ throw new IllegalArgumentException(
+ s"Response was not valid JSON: ${e.getMessage}. Response text: $jsonText"
+ )
+ }
+
+ parsed match {
+ case obj: ujson.Obj => convert(obj).asInstanceOf[LogoList]
+ case other =>
+ throw new IllegalArgumentException(
+ s"expected a JSON object in the reply, but got ${typeName(other)}. Response text: $jsonText"
+ )
+ }
+ }
+
+ private def typeName(value: ujson.Value): String = value match {
+ case _: ujson.Obj => "an object"
+ case _: ujson.Arr => "an array"
+ case _: ujson.Str => "a string"
+ case _: ujson.Num => "a number"
+ case _: ujson.Bool => "a boolean"
+ case ujson.Null => "null"
+ }
+
+ /**
+ * Recursively convert a JSON value to its NetLogo equivalent.
+ *
+ * Numbers box to Double because NetLogo has a single numeric type; null maps
+ * to the empty string because NetLogo has no null and an empty string is a
+ * value modelers can compare against without a runtime error.
+ */
+ def convert(value: ujson.Value): AnyRef = value match {
+ case obj: ujson.Obj =>
+ LogoList.fromIterator(
+ obj.value.iterator.map { case (key, v) => LogoList(key, convert(v)) }
+ )
+ case arr: ujson.Arr =>
+ LogoList.fromIterator(arr.value.iterator.map(convert))
+ case ujson.Str(s) => s
+ case ujson.Num(n) => Double.box(n)
+ case ujson.Bool(b) => Boolean.box(b)
+ case ujson.Null => ""
+ }
+
+ /**
+ * Find the value for `key` in a list of `[key value]` pairs.
+ *
+ * Matching is exact and case-sensitive, because JSON keys are. Entries that
+ * are not two-element lists are skipped rather than matched positionally, so
+ * asking a plain list such as `["a" "b"]` for key "a" misses instead of
+ * returning a neighbouring element.
+ *
+ * @return the value, or None if no pair has that key
+ */
+ def lookup(list: LogoList, key: String): Option[AnyRef] =
+ list.toVector.collectFirst {
+ case pair: LogoList if pair.size == 2 && pair(0) == key => pair(1)
+ }
+}
diff --git a/src/test/DeterministicTestProvider.scala b/src/test/DeterministicTestProvider.scala
index 263fcde..cca0ecb 100644
--- a/src/test/DeterministicTestProvider.scala
+++ b/src/test/DeterministicTestProvider.scala
@@ -1,7 +1,7 @@
package org.nlogo.extensions.llm.providers
import org.nlogo.extensions.llm.config.ConfigStore
-import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, Choice}
+import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, Choice, EnumFormat, JsonObjectFormat, JsonSchemaFormat, ResponseFormat}
import scala.concurrent.{ExecutionContext, Future}
import scala.util.{Success, Try}
@@ -18,6 +18,8 @@ class DeterministicTestProvider(implicit ec: ExecutionContext) extends LLMProvid
private val testRespondRegex = """__TEST_RESPOND:(.+)""".r
private val testThinkingRegex = """__TEST_THINKING:(.+)""".r
private val testDelayRegex = """__TEST_DELAY:(\d+):(.*)""".r.unanchored
+ // Captures to end of line so a raw choice may contain spaces, braces, or quotes.
+ private val testRawChoiceRegex = """__TEST_RAW_CHOICE:(.*)""".r
override def chat(request: ChatRequest): Future[ChatResponse] = {
chat(request.messages).map { message =>
@@ -58,6 +60,80 @@ class DeterministicTestProvider(implicit ec: ExecutionContext) extends LLMProvid
}
}
+ /**
+ * Stands in for a provider that honours the requested format.
+ *
+ * An explicit `__TEST_RESPOND:` still wins, so a test can supply the exact
+ * reply text — including a deliberately malformed one. Otherwise an enum
+ * request answers in the constrained `{"choice": ...}` shape, which is what
+ * a real provider enforcing the constraint returns.
+ */
+ override def chatWithFormat(messages: Seq[ChatMessage], format: ResponseFormat): Future[ChatResponse] = {
+ val lastUserMessage = messages.reverseIterator
+ .find(_.role == "user")
+ .map(_.content)
+ .getOrElse("")
+
+ if (lastUserMessage.contains("__TEST_RESPOND:") || lastUserMessage.contains("__TEST_FAIL")) {
+ return chatWithFullResponse(messages)
+ }
+
+ // __TEST_ECHO_FORMAT* markers report which format actually arrived, so a
+ // primitive that stopped constraining its request fails a test instead of
+ // silently degrading. The enum echo answers with the LAST choice because
+ // the unconstrained prompt-only path answers with the first.
+ if (lastUserMessage.contains("__TEST_ECHO_FORMAT_ENUM")) {
+ return Future.successful(structured(format match {
+ case EnumFormat(choices) => choices.last
+ case _ => "__NO_ENUM_FORMAT_RECEIVED__"
+ }))
+ }
+
+ if (lastUserMessage.contains("__TEST_ECHO_FORMAT")) {
+ return Future.successful(structured(format match {
+ case JsonObjectFormat => ujson.Obj("format" -> "json_object").toString
+ case JsonSchemaFormat(schema, name) =>
+ // Echo the schema exactly as handed over. Normalizing here would make
+ // the assertion test this stub rather than the primitive, and it would
+ // stay green even if every real provider stopped normalizing.
+ ujson.Obj(
+ "format" -> "json_schema",
+ "name" -> name,
+ "topLevelType" -> schema.value.get("type").getOrElse(ujson.Null)
+ ).toString
+ case EnumFormat(_) => ujson.Obj("format" -> "enum").toString
+ }))
+ }
+
+ // __TEST_IGNORE_FORMAT simulates a provider that cannot enforce the
+ // requested format and answers in free text anyway — the graceful
+ // degradation path every caller has to survive. __TEST_RAW_CHOICE:
+ // pins that free-text reply so a specific shape can be asserted.
+ if (lastUserMessage.contains("__TEST_IGNORE_FORMAT")) {
+ val raw = testRawChoiceRegex.findFirstMatchIn(lastUserMessage)
+ .map(_.group(1).trim)
+ .getOrElse(respond(lastUserMessage).content)
+ return Future.successful(structured(raw))
+ }
+
+ format match {
+ case EnumFormat(choices) =>
+ val body = ujson.Obj(EnumFormat.ChoiceKey -> choices.head).toString
+ Future.successful(structured(body))
+ case JsonSchemaFormat(_, _) =>
+ Future.successful(structured("{}"))
+ case JsonObjectFormat =>
+ Future.successful(structured("{}"))
+ }
+ }
+
+ private def structured(content: String): ChatResponse =
+ ChatResponse.simple(
+ id = "deterministic-test-response",
+ model = "deterministic-model",
+ message = ChatMessage.assistant(content)
+ )
+
override def chat(messages: Seq[ChatMessage]): Future[ChatMessage] = {
val lastUserMessage = messages.reverseIterator
.find(_.role == "user")
diff --git a/src/test/JsonToNetLogoSpec.scala b/src/test/JsonToNetLogoSpec.scala
new file mode 100644
index 0000000..755b102
--- /dev/null
+++ b/src/test/JsonToNetLogoSpec.scala
@@ -0,0 +1,142 @@
+// ABOUTME: Deterministic tests for recursive JSON to NetLogo value conversion and key lookup
+// ABOUTME: Covers every JSON type, nesting, and the lookup semantics behind llm:get
+package org.nlogo.extensions.llm.utils
+
+import org.nlogo.core.LogoList
+import org.scalatest.funsuite.AnyFunSuite
+
+class JsonToNetLogoSpec extends AnyFunSuite {
+
+ private def convert(text: String): AnyRef = JsonToNetLogo.convert(ujson.read(text))
+
+ // --- Scalars ---
+
+ test("a string converts to a NetLogo string") {
+ assert(convert(""""hello"""") == "hello")
+ }
+
+ test("a number converts to a boxed Double") {
+ assert(convert("42") == Double.box(42.0))
+ assert(convert("3.14") == Double.box(3.14))
+ }
+
+ test("a boolean converts to a boxed Boolean") {
+ assert(convert("true") == Boolean.box(true))
+ assert(convert("false") == Boolean.box(false))
+ }
+
+ test("null converts to an empty string") {
+ // NetLogo has no null. Empty string is the value modelers can test with
+ // `= \"\"` without a runtime error, which nobody is available for otherwise.
+ assert(convert("null") == "")
+ }
+
+ // --- Containers ---
+
+ test("an array converts to a flat NetLogo list") {
+ val result = convert("""["a","b","c"]""").asInstanceOf[LogoList]
+ assert(result.toVector == Vector("a", "b", "c"))
+ }
+
+ test("an object converts to a list of [key value] pairs preserving order") {
+ val result = convert("""{"name":"Alice","age":30}""").asInstanceOf[LogoList]
+ assert(result.size == 2)
+ val first = result(0).asInstanceOf[LogoList]
+ assert(first.toVector == Vector("name", "Alice"))
+ val second = result(1).asInstanceOf[LogoList]
+ assert(second.toVector == Vector("age", Double.box(30.0)))
+ }
+
+ test("an empty object converts to an empty list") {
+ assert(convert("{}").asInstanceOf[LogoList].size == 0)
+ }
+
+ test("nested objects convert recursively") {
+ val result = convert("""{"addr":{"city":"Chicago"}}""").asInstanceOf[LogoList]
+ val addrPair = result(0).asInstanceOf[LogoList]
+ assert(addrPair(0) == "addr")
+ val inner = addrPair(1).asInstanceOf[LogoList]
+ assert(inner(0).asInstanceOf[LogoList].toVector == Vector("city", "Chicago"))
+ }
+
+ test("arrays of objects convert recursively") {
+ val result = convert("""{"steps":[{"n":1},{"n":2}]}""").asInstanceOf[LogoList]
+ val steps = result(0).asInstanceOf[LogoList](1).asInstanceOf[LogoList]
+ assert(steps.size == 2)
+ val firstStep = steps(0).asInstanceOf[LogoList](0).asInstanceOf[LogoList]
+ assert(firstStep.toVector == Vector("n", Double.box(1.0)))
+ }
+
+ // --- parse entry point ---
+
+ test("parse accepts a JSON string and converts it") {
+ val result = JsonToNetLogo.parse("""{"a":1}""").asInstanceOf[LogoList]
+ assert(result(0).asInstanceOf[LogoList].toVector == Vector("a", Double.box(1.0)))
+ }
+
+ test("parse rejects malformed JSON with a message quoting the text") {
+ val ex = intercept[IllegalArgumentException] { JsonToNetLogo.parse("""{"a":""") }
+ assert(ex.getMessage.toLowerCase.contains("json"))
+ }
+
+ // --- parseObject: the object-only entry point ---
+
+ test("parseObject accepts an object and reports pairs") {
+ val result = JsonToNetLogo.parseObject("""{"a":1}""")
+ assert(result(0).asInstanceOf[LogoList].toVector == Vector("a", Double.box(1.0)))
+ }
+
+ test("parseObject rejects a top-level scalar naming the type it got") {
+ val ex = intercept[IllegalArgumentException] { JsonToNetLogo.parseObject(""""hi"""") }
+ assert(ex.getMessage.contains("a string"))
+ assert(ex.getMessage.contains("expected a JSON object"))
+ }
+
+ test("parseObject rejects a top-level array") {
+ // A flat list would not be readable by llm:get, so this must not slip through.
+ val ex = intercept[IllegalArgumentException] { JsonToNetLogo.parseObject("""["a"]""") }
+ assert(ex.getMessage.contains("an array"))
+ }
+
+ test("parseObject rejects malformed JSON") {
+ val ex = intercept[IllegalArgumentException] { JsonToNetLogo.parseObject("""{"a":""") }
+ assert(ex.getMessage.contains("not valid JSON"))
+ }
+
+ // --- Key lookup (llm:get) ---
+
+ test("lookup finds a top-level key") {
+ val parsed = convert("""{"name":"Alice","age":30}""").asInstanceOf[LogoList]
+ assert(JsonToNetLogo.lookup(parsed, "name").contains("Alice"))
+ assert(JsonToNetLogo.lookup(parsed, "age").contains(Double.box(30.0)))
+ }
+
+ test("lookup returns None for a missing key rather than guessing") {
+ val parsed = convert("""{"name":"Alice"}""").asInstanceOf[LogoList]
+ assert(JsonToNetLogo.lookup(parsed, "nope").isEmpty)
+ }
+
+ test("lookup returns a nested structure so chained gets work") {
+ val parsed = convert("""{"addr":{"city":"Chicago"}}""").asInstanceOf[LogoList]
+ val addr = JsonToNetLogo.lookup(parsed, "addr").get.asInstanceOf[LogoList]
+ assert(JsonToNetLogo.lookup(addr, "city").contains("Chicago"))
+ }
+
+ test("lookup is exact-match and case-sensitive, matching JSON key semantics") {
+ val parsed = convert("""{"Name":"Alice"}""").asInstanceOf[LogoList]
+ assert(JsonToNetLogo.lookup(parsed, "name").isEmpty)
+ assert(JsonToNetLogo.lookup(parsed, "Name").contains("Alice"))
+ }
+
+ test("lookup ignores entries that are not [key value] pairs") {
+ // A plain list such as ["a" "b" "c"] is not a key-value structure; asking
+ // for a key must miss rather than match a bare element.
+ val plain = convert("""["a","b"]""").asInstanceOf[LogoList]
+ assert(JsonToNetLogo.lookup(plain, "a").isEmpty)
+ }
+
+ test("lookup returns the first match when a key repeats") {
+ val list = LogoList(LogoList("k", "first"), LogoList("k", "second"))
+ assert(JsonToNetLogo.lookup(list, "k").contains("first"))
+ }
+}
diff --git a/src/test/ResponseFormatSpec.scala b/src/test/ResponseFormatSpec.scala
new file mode 100644
index 0000000..0939fbc
--- /dev/null
+++ b/src/test/ResponseFormatSpec.scala
@@ -0,0 +1,164 @@
+// ABOUTME: Deterministic tests for ResponseFormat parsing, validation, and schema normalization
+// ABOUTME: Asserts modeler-supplied schema strings are rejected with clear messages or normalized consistently
+package org.nlogo.extensions.llm.models
+
+import org.scalatest.funsuite.AnyFunSuite
+
+class ResponseFormatSpec extends AnyFunSuite {
+
+ private def keys(v: ujson.Value): Set[String] = v.obj.keys.toSet
+
+ // --- Parsing modeler-supplied schema strings ---
+
+ test("a valid object schema parses into JsonSchemaFormat") {
+ val fmt = ResponseFormat.parseSchema(
+ """{"type":"object","properties":{"action":{"type":"string"}},"required":["action"]}"""
+ )
+ assert(fmt.schema("type").str == "object")
+ }
+
+ test("malformed JSON is rejected naming the schema, not a raw parser dump") {
+ val ex = intercept[IllegalArgumentException] {
+ ResponseFormat.parseSchema("""{"type":"object",""")
+ }
+ assert(ex.getMessage.toLowerCase.contains("schema"))
+ assert(ex.getMessage.toLowerCase.contains("valid json"))
+ }
+
+ test("a JSON value that is not an object is rejected") {
+ val ex = intercept[IllegalArgumentException] {
+ ResponseFormat.parseSchema("""["a","b"]""")
+ }
+ assert(ex.getMessage.contains("JSON object"))
+ }
+
+ test("an empty schema string is rejected") {
+ val ex = intercept[IllegalArgumentException] {
+ ResponseFormat.parseSchema(" ")
+ }
+ assert(ex.getMessage.toLowerCase.contains("empty"))
+ }
+
+ test("a schema missing a type is rejected with a message naming 'type'") {
+ val ex = intercept[IllegalArgumentException] {
+ ResponseFormat.parseSchema("""{"properties":{"a":{"type":"string"}}}""")
+ }
+ assert(ex.getMessage.contains("type"))
+ }
+
+ test("an object schema with no properties is rejected") {
+ val ex = intercept[IllegalArgumentException] {
+ ResponseFormat.parseSchema("""{"type":"object"}""")
+ }
+ assert(ex.getMessage.contains("properties"))
+ }
+
+ test("an enum that is not an array is rejected locally rather than at the provider") {
+ // Every provider requires enum to be a list of values. Catching it here
+ // turns a provider-specific 400 into a message naming the offending field.
+ val ex = intercept[IllegalArgumentException] {
+ ResponseFormat.parseSchema("""{"type":"object","properties":{"d":{"type":"string","enum":"north"}}}""")
+ }
+ assert(ex.getMessage.contains("enum"))
+ assert(ex.getMessage.contains("d"))
+ }
+
+ test("an empty enum array is rejected") {
+ val ex = intercept[IllegalArgumentException] {
+ ResponseFormat.parseSchema("""{"type":"object","properties":{"d":{"type":"string","enum":[]}}}""")
+ }
+ assert(ex.getMessage.contains("enum"))
+ }
+
+ test("a valid enum is accepted and preserved through normalization") {
+ val fmt = ResponseFormat.parseSchema(
+ """{"type":"object","properties":{"d":{"type":"string","enum":["north","south"]}}}"""
+ )
+ val strict = ResponseFormat.strictSchema(fmt.schema)
+ assert(strict("properties")("d")("enum").arr.map(_.str) == Seq("north", "south"))
+ }
+
+ // --- Strict normalization (OpenAI/Anthropic requirements) ---
+
+ test("normalization injects additionalProperties false on an object schema") {
+ val schema = ResponseFormat.parseSchema(
+ """{"type":"object","properties":{"a":{"type":"string"}},"required":["a"]}"""
+ ).schema
+ val strict = ResponseFormat.strictSchema(schema)
+ assert(strict("additionalProperties").bool == false)
+ }
+
+ test("normalization lists every property in required, not just the declared ones") {
+ val schema = ResponseFormat.parseSchema(
+ """{"type":"object","properties":{"a":{"type":"string"},"b":{"type":"number"}},"required":["a"]}"""
+ ).schema
+ val strict = ResponseFormat.strictSchema(schema)
+ assert(strict("required").arr.map(_.str).toSet == Set("a", "b"))
+ }
+
+ test("normalization recurses into nested object properties") {
+ val schema = ResponseFormat.parseSchema(
+ """{"type":"object","properties":{"addr":{"type":"object","properties":{"city":{"type":"string"}}}}}"""
+ ).schema
+ val strict = ResponseFormat.strictSchema(schema)
+ val nested = strict("properties")("addr")
+ assert(nested("additionalProperties").bool == false)
+ assert(nested("required").arr.map(_.str) == Seq("city"))
+ }
+
+ test("normalization recurses into array item schemas") {
+ val schema = ResponseFormat.parseSchema(
+ """{"type":"object","properties":{"steps":{"type":"array","items":{"type":"object","properties":{"n":{"type":"number"}}}}}}"""
+ ).schema
+ val strict = ResponseFormat.strictSchema(schema)
+ val items = strict("properties")("steps")("items")
+ assert(items("additionalProperties").bool == false)
+ assert(items("required").arr.map(_.str) == Seq("n"))
+ }
+
+ test("normalization does not mutate the caller's schema") {
+ val fmt = ResponseFormat.parseSchema(
+ """{"type":"object","properties":{"a":{"type":"string"}}}"""
+ )
+ ResponseFormat.strictSchema(fmt.schema)
+ assert(!keys(fmt.schema).contains("additionalProperties"), "strictSchema must return a copy")
+ }
+
+ // --- Enum format ---
+
+ test("EnumFormat renders as a single-property object schema constraining the choice") {
+ val schema = EnumFormat(List("north", "south")).schema
+ assert(schema("type").str == "object")
+ val choiceProp = schema("properties")(EnumFormat.ChoiceKey)
+ assert(choiceProp("enum").arr.map(_.str) == Seq("north", "south"))
+ }
+
+ test("EnumFormat rejects an empty choice list") {
+ intercept[IllegalArgumentException] { EnumFormat(Nil) }
+ }
+
+ // --- Serialization ---
+
+ test("a ChatRequest carrying a response format round-trips through upickle") {
+ // ChatRequest exposes a derived ReadWriter. Adding a sealed-trait field
+ // breaks that derivation unless ResponseFormat provides its own instance.
+ val request = ChatRequest(
+ model = "m",
+ messages = Seq(ChatMessage.user("hi")),
+ responseFormat = Some(EnumFormat(List("a", "b")))
+ )
+ val restored = upickle.default.read[ChatRequest](upickle.default.write(request))
+ assert(restored.responseFormat.contains(EnumFormat(List("a", "b"))))
+ }
+
+ test("a JsonSchemaFormat round-trips with its schema intact") {
+ val fmt = ResponseFormat.parseSchema("""{"type":"object","properties":{"a":{"type":"string"}}}""")
+ val restored = upickle.default.read[ResponseFormat](upickle.default.write[ResponseFormat](fmt))
+ assert(restored.asInstanceOf[JsonSchemaFormat].schema("properties")("a")("type").str == "string")
+ }
+
+ test("JsonObjectFormat round-trips") {
+ val restored = upickle.default.read[ResponseFormat](upickle.default.write[ResponseFormat](JsonObjectFormat))
+ assert(restored == JsonObjectFormat)
+ }
+}
diff --git a/src/test/StructuredChatPlumbingSpec.scala b/src/test/StructuredChatPlumbingSpec.scala
new file mode 100644
index 0000000..d47f1b8
--- /dev/null
+++ b/src/test/StructuredChatPlumbingSpec.scala
@@ -0,0 +1,64 @@
+// ABOUTME: Tests that a requested response format reaches the provider's request body unchanged
+// ABOUTME: Covers the LLMProvider entry point used by llm:chat-with-schema and llm:chat-json
+package org.nlogo.extensions.llm.providers
+
+import org.nlogo.extensions.llm.config.ConfigStore
+import org.nlogo.extensions.llm.models._
+import org.scalatest.funsuite.AnyFunSuite
+import scala.concurrent.ExecutionContext.Implicits.global
+
+/**
+ * Captures the ChatRequest that would have been sent, without any network I/O.
+ */
+private class CapturingProvider extends OllamaProvider {
+ var captured: Option[ChatRequest] = None
+ override def chat(request: ChatRequest): scala.concurrent.Future[ChatResponse] = {
+ captured = Some(request)
+ scala.concurrent.Future.successful(
+ ChatResponse.simple("id", request.model, ChatMessage.assistant("{}"))
+ )
+ }
+}
+
+class StructuredChatPlumbingSpec extends AnyFunSuite {
+
+ ProviderRegistry.reset()
+ ProviderRegistrations.registerAll()
+
+ private val messages = Seq(ChatMessage.user("hi"))
+ private val schema = ResponseFormat.parseSchema(
+ """{"type":"object","properties":{"a":{"type":"string"}}}"""
+ )
+
+ test("chatWithFormat forwards the requested format to the provider request") {
+ val provider = new CapturingProvider
+ provider.chatWithFormat(messages, schema)
+ assert(provider.captured.flatMap(_.responseFormat).contains(schema))
+ }
+
+ test("chatWithFormat still applies configured model and token settings") {
+ val provider = new CapturingProvider
+ provider.setConfig(ConfigStore.MODEL, "llama3.1")
+ provider.setConfig(ConfigStore.MAX_TOKENS, "1234")
+ provider.chatWithFormat(messages, JsonObjectFormat)
+
+ val request = provider.captured.get
+ assert(request.model == "llama3.1")
+ assert(request.maxTokens.contains(1234))
+ }
+
+ test("the ordinary chat path still sends no response format") {
+ val provider = new CapturingProvider
+ provider.chat(messages)
+ assert(provider.captured.get.responseFormat.isEmpty)
+ }
+
+ test("chatWithFormat returns the full response so callers can read content") {
+ val provider = new CapturingProvider
+ val response = scala.concurrent.Await.result(
+ provider.chatWithFormat(messages, JsonObjectFormat),
+ scala.concurrent.duration.Duration(5, "seconds")
+ )
+ assert(response.firstContent.contains("{}"))
+ }
+}
diff --git a/src/test/StructuredOutputRequestSpec.scala b/src/test/StructuredOutputRequestSpec.scala
new file mode 100644
index 0000000..d10ccf0
--- /dev/null
+++ b/src/test/StructuredOutputRequestSpec.scala
@@ -0,0 +1,233 @@
+// ABOUTME: Deterministic tests asserting each provider wraps a response format in its own request shape
+// ABOUTME: No network I/O — only the JSON body each provider would have posted
+package org.nlogo.extensions.llm.providers
+
+import org.nlogo.extensions.llm.models._
+import org.scalatest.funsuite.AnyFunSuite
+import scala.concurrent.ExecutionContext.Implicits.global
+
+/** Exposes each provider's protected request builder for body assertions. */
+private class InspectableOpenAI extends OpenAIProvider {
+ def body(r: ChatRequest): ujson.Value = createProviderRequest(r)
+}
+private class InspectableGroq extends GroqProvider {
+ def body(r: ChatRequest): ujson.Value = createProviderRequest(r)
+}
+private class InspectableTogether extends TogetherProvider {
+ def body(r: ChatRequest): ujson.Value = createProviderRequest(r)
+}
+private class InspectableOpenRouter extends OpenRouterProvider {
+ def body(r: ChatRequest): ujson.Value = createProviderRequest(r)
+}
+private class InspectableClaude extends ClaudeProvider {
+ def body(r: ChatRequest): ujson.Value = createProviderRequest(r)
+}
+private class InspectableGemini extends GeminiProvider {
+ def body(r: ChatRequest): ujson.Value = createProviderRequest(r)
+}
+private class InspectableOllama extends OllamaProvider {
+ def body(r: ChatRequest): ujson.Value = createProviderRequest(r)
+}
+
+class StructuredOutputRequestSpec extends AnyFunSuite {
+
+ // Constructing a provider reads its default model from the registry, which is
+ // normally populated by LLMExtension.load(). Register here so the suite is
+ // self-contained and order-independent.
+ ProviderRegistry.reset()
+ ProviderRegistrations.registerAll()
+
+ private val schemaText =
+ """{"type":"object","properties":{"action":{"type":"string"},"confidence":{"type":"number"}},"required":["action"]}"""
+
+ private val schemaFormat = ResponseFormat.parseSchema(schemaText)
+
+ private def request(
+ model: String,
+ format: Option[ResponseFormat],
+ thinking: Option[ThinkingConfig] = None,
+ temperature: Option[Double] = None,
+ maxTokens: Option[Int] = Some(4000)
+ ): ChatRequest =
+ ChatRequest(
+ model = model,
+ messages = Seq(ChatMessage.user("hi")),
+ maxTokens = maxTokens,
+ temperature = temperature,
+ thinkingConfig = thinking,
+ responseFormat = format
+ )
+
+ private def keys(v: ujson.Value): Set[String] = v.obj.keys.toSet
+
+ // --- OpenAI-compatible family: openai, groq, together, openrouter ---
+
+ private val openAiCompatible: Seq[(String, ChatRequest => ujson.Value)] = Seq(
+ "openai" -> (new InspectableOpenAI).body,
+ "groq" -> (new InspectableGroq).body,
+ "together" -> (new InspectableTogether).body,
+ "openrouter" -> (new InspectableOpenRouter).body
+ )
+
+ test("every OpenAI-compatible provider nests the schema under response_format.json_schema") {
+ openAiCompatible.foreach { case (name, build) =>
+ val body = build(request("some-model", Some(schemaFormat)))
+ val rf = body("response_format")
+ assert(rf("type").str == "json_schema", s"$name response_format.type")
+ assert(rf("json_schema")("strict").bool, s"$name must request strict mode")
+ assert(rf("json_schema")("name").str.nonEmpty, s"$name must send a schema name")
+ assert(
+ rf("json_schema")("schema")("properties")("action")("type").str == "string",
+ s"$name must carry the caller's schema"
+ )
+ }
+ }
+
+ test("OpenAI-compatible providers normalize the schema to strict mode") {
+ openAiCompatible.foreach { case (name, build) =>
+ val body = build(request("some-model", Some(schemaFormat)))
+ val schema = body("response_format")("json_schema")("schema")
+ assert(schema("additionalProperties").bool == false, s"$name additionalProperties")
+ assert(
+ schema("required").arr.map(_.str).toSet == Set("action", "confidence"),
+ s"$name must list every property as required"
+ )
+ }
+ }
+
+ test("OpenAI-compatible providers send response_format json_object for JSON mode") {
+ openAiCompatible.foreach { case (name, build) =>
+ val body = build(request("some-model", Some(JsonObjectFormat)))
+ assert(body("response_format")("type").str == "json_object", s"$name json mode")
+ assert(!keys(body("response_format")).contains("json_schema"), s"$name json mode carries no schema")
+ }
+ }
+
+ test("OpenAI-compatible providers send an enum as a constrained json_schema") {
+ openAiCompatible.foreach { case (name, build) =>
+ val body = build(request("some-model", Some(EnumFormat(List("north", "south")))))
+ val schema = body("response_format")("json_schema")("schema")
+ assert(
+ schema("properties")(EnumFormat.ChoiceKey)("enum").arr.map(_.str) == Seq("north", "south"),
+ s"$name enum values"
+ )
+ }
+ }
+
+ test("no response format leaves the OpenAI-compatible body unchanged") {
+ openAiCompatible.foreach { case (name, build) =>
+ val body = build(request("some-model", None))
+ assert(!keys(body).contains("response_format"), s"$name must omit response_format when unset")
+ }
+ }
+
+ test("structured output survives alongside OpenAI reasoning fields") {
+ val body = (new InspectableOpenAI).body(
+ request("o3-mini", Some(schemaFormat), thinking = Some(ThinkingConfig(enabled = true, reasoningEffort = Some("high"))))
+ )
+ assert(body("reasoning_effort").str == "high")
+ assert(body("response_format")("type").str == "json_schema")
+ }
+
+ // --- Anthropic ---
+
+ test("Claude nests the schema under output_config.format") {
+ val body = (new InspectableClaude).body(request("claude-sonnet-4-5-20250929", Some(schemaFormat)))
+ val format = body("output_config")("format")
+ assert(format("type").str == "json_schema")
+ assert(format("schema")("properties")("action")("type").str == "string")
+ assert(format("schema")("additionalProperties").bool == false)
+ }
+
+ test("Claude keeps output_config.effort when a schema is also present") {
+ // effort and format share one output_config object; writing the schema must
+ // merge into it, not replace the thinking-depth setting.
+ val body = (new InspectableClaude).body(
+ request(
+ "claude-opus-5",
+ Some(schemaFormat),
+ thinking = Some(ThinkingConfig(enabled = true, reasoningEffort = Some("high")))
+ )
+ )
+ assert(body("output_config")("effort").str == "high", "effort must survive a schema")
+ assert(body("output_config")("format")("type").str == "json_schema", "schema must survive effort")
+ assert(body("thinking")("type").str == "adaptive")
+ }
+
+ test("Claude extended-thinking request still carries a schema") {
+ val body = (new InspectableClaude).body(
+ request("claude-haiku-4-5-20251001", Some(schemaFormat), thinking = Some(ThinkingConfig(enabled = true)))
+ )
+ assert(body("thinking")("type").str == "enabled")
+ assert(body("output_config")("format")("type").str == "json_schema")
+ }
+
+ test("Claude JSON mode with no schema sends no output_config.format") {
+ // Anthropic has no schemaless JSON mode, so there is nothing truthful to
+ // send; the prompt-level instruction carries it instead.
+ val body = (new InspectableClaude).body(request("claude-sonnet-4-5-20250929", Some(JsonObjectFormat)))
+ val hasFormat = keys(body).contains("output_config") && keys(body("output_config")).contains("format")
+ assert(!hasFormat, s"Claude must not invent a json_object format: $body")
+ }
+
+ test("no response format leaves the Claude body without output_config.format") {
+ val body = (new InspectableClaude).body(request("claude-sonnet-4-5-20250929", None))
+ assert(!keys(body).contains("output_config"))
+ }
+
+ // --- Gemini ---
+
+ test("Gemini sets responseJsonSchema and the JSON mime type") {
+ val body = (new InspectableGemini).body(request("gemini-2.5-flash", Some(schemaFormat)))
+ val gc = body("generationConfig")
+ assert(gc("responseMimeType").str == "application/json")
+ assert(gc("responseJsonSchema")("properties")("action")("type").str == "string")
+ }
+
+ test("Gemini JSON mode sets the mime type with no schema") {
+ val gc = (new InspectableGemini).body(request("gemini-2.5-flash", Some(JsonObjectFormat)))("generationConfig")
+ assert(gc("responseMimeType").str == "application/json")
+ assert(!keys(gc).contains("responseJsonSchema"))
+ }
+
+ test("Gemini keeps thinkingConfig alongside a response schema") {
+ val body = (new InspectableGemini).body(
+ request("gemini-2.5-flash", Some(schemaFormat), thinking = Some(ThinkingConfig(enabled = true, budgetTokens = Some(2048))))
+ )
+ val gc = body("generationConfig")
+ assert(gc("thinkingConfig")("thinkingBudget").num == 2048)
+ assert(gc("responseMimeType").str == "application/json")
+ }
+
+ test("no response format leaves the Gemini body without responseMimeType") {
+ val body = (new InspectableGemini).body(request("gemini-2.5-flash", None, temperature = Some(0.7)))
+ assert(!keys(body("generationConfig")).contains("responseMimeType"))
+ assert(!keys(body("generationConfig")).contains("responseJsonSchema"))
+ }
+
+ // --- Ollama ---
+
+ test("Ollama puts the raw schema in format") {
+ val body = (new InspectableOllama).body(request("llama3.1", Some(schemaFormat)))
+ assert(body("format")("type").str == "object")
+ assert(body("format")("properties")("action")("type").str == "string")
+ }
+
+ test("Ollama JSON mode sends the string format json") {
+ val body = (new InspectableOllama).body(request("llama3.1", Some(JsonObjectFormat)))
+ assert(body("format").str == "json")
+ }
+
+ test("Ollama keeps think alongside a schema") {
+ val body = (new InspectableOllama).body(
+ request("llama3.1", Some(schemaFormat), thinking = Some(ThinkingConfig(enabled = true)))
+ )
+ assert(body("think").bool)
+ assert(body("format")("type").str == "object")
+ }
+
+ test("no response format leaves the Ollama body without format") {
+ val body = (new InspectableOllama).body(request("llama3.1", None))
+ assert(!keys(body).contains("format"))
+ }
+}
diff --git a/tests.txt b/tests.txt
index b3cedb8..b2ebac2 100644
--- a/tests.txt
+++ b/tests.txt
@@ -502,3 +502,226 @@ LLMCompileErrorBannedListValidation
extensions [llm]
(llm:compile-error "die" [1 2]) => ERROR Extension exception: llm:compile-error expects a list of primitive names as strings, but got: 1, 2
(llm:compile-error "fd 1" ["die" 5]) => ERROR Extension exception: llm:compile-error expects a list of primitive names as strings, but got: 5
+
+LLMGetLookup
+ extensions [llm]
+ llm:get [["name" "Alice"] ["age" 30]] "name" => "Alice"
+ llm:get [["name" "Alice"] ["age" 30]] "age" => 30
+ llm:get [["flag" true]] "flag" => true
+ llm:get [["skills" ["coding" "design"]]] "skills" => ["coding" "design"]
+ llm:get (llm:get [["addr" [["city" "Chicago"]]]] "addr") "city" => "Chicago"
+ llm:get [["name" "Alice"]] "missing" => ERROR Extension exception: llm:get: key "missing" not found. Available keys: name
+ llm:get [] "any" => ERROR Extension exception: llm:get: key "any" not found. The list is empty.
+ llm:get ["a" "b"] "a" => ERROR Extension exception: llm:get: key "a" not found. Available keys: (none - the list is not a list of [key value] pairs)
+
+LLMChatWithSchemaParsesJson
+ extensions [llm]
+ globals [result]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set result llm:chat-with-schema "__TEST_RESPOND:{\"action\":\"explore\",\"confidence\":0.8}" "{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\"},\"confidence\":{\"type\":\"number\"}}}"
+ llm:get result "action" => "explore"
+ llm:get result "confidence" => 0.8
+ is-list? result => true
+ O> llm:clear-history
+
+LLMChatWithSchemaNested
+ extensions [llm]
+ globals [result]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set result llm:chat-with-schema "__TEST_RESPOND:{\"plan\":[\"a\",\"b\"],\"meta\":{\"n\":2}}" "{\"type\":\"object\",\"properties\":{\"plan\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}},\"meta\":{\"type\":\"object\",\"properties\":{\"n\":{\"type\":\"number\"}}}}}"
+ llm:get result "plan" => ["a" "b"]
+ llm:get (llm:get result "meta") "n" => 2
+ O> llm:clear-history
+
+LLMChatWithSchemaRejectsBadSchema
+ extensions [llm]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ llm:chat-with-schema "hello" "not json at all" => ERROR Extension exception: llm:chat-with-schema: Schema is not valid JSON: expected null got "n" at index 0
+ llm:chat-with-schema "hello" "" => ERROR Extension exception: llm:chat-with-schema: Schema cannot be empty. Provide a JSON Schema object, e.g. {"type":"object","properties":{"action":{"type":"string"}}}
+
+LLMChatWithSchemaRejectsNonObjectTopLevel
+ extensions [llm]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ llm:chat-with-schema "hi" "{\"type\":\"string\"}" => ERROR Extension exception: llm:chat-with-schema: schema must have type 'object' at the top level, but got 'string'. llm:chat-with-schema reports a list of [key value] pairs, so the reply has to be a JSON object. Wrap it, e.g. {"type":"object","properties":{"value":{"type":"string"}}}
+ llm:chat-with-schema "hi" "{\"type\":\"array\",\"items\":{\"type\":\"string\"}}" => ERROR Extension exception: llm:chat-with-schema: schema must have type 'object' at the top level, but got 'array'. llm:chat-with-schema reports a list of [key value] pairs, so the reply has to be a JSON object. Wrap it, e.g. {"type":"object","properties":{"value":{"type":"string"}}}
+
+LLMChatWithSchemaRejectsNonObjectReply
+ extensions [llm]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ llm:chat-with-schema "__TEST_RESPOND:\"just a string\"" "{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"string\"}}}" => ERROR Extension exception: llm:chat-with-schema: expected a JSON object in the reply, but got a string. Response text: "just a string"
+ llm:history => []
+ O> llm:clear-history
+
+LLMChatWithSchemaHistory
+ extensions [llm]
+ globals [result]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set result llm:chat-with-schema "__TEST_RESPOND:{\"a\":1}" "{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"number\"}}}"
+ length llm:history => 2
+ item 0 (item 0 llm:history) => "user"
+ item 0 (item 1 llm:history) => "assistant"
+ O> llm:clear-history
+
+LLMChatWithSchemaHistoryUnchangedOnFailure
+ extensions [llm]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> carefully [ let r llm:chat-with-schema "__TEST_FAIL" "{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"number\"}}}" ] [ ]
+ llm:history => []
+ O> llm:clear-history
+
+LLMChatWithSchemaRejectsNonJsonReply
+ extensions [llm]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ llm:chat-with-schema "__TEST_RESPOND:sorry I cannot" "{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"number\"}}}" => ERROR Extension exception: llm:chat-with-schema: Response was not valid JSON: expected json value got "s" at index 0. Response text: sorry I cannot
+ llm:history => []
+ O> llm:clear-history
+
+LLMChatJsonReturnsRawString
+ extensions [llm]
+ globals [result]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set result llm:chat-json "__TEST_RESPOND:{\"a\":1}"
+ is-string? result => true
+ result => "{\"a\":1}"
+ length llm:history => 2
+ O> llm:clear-history
+
+LLMChatJsonRejectsNonJsonReply
+ extensions [llm]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ llm:chat-json "__TEST_RESPOND:Sure! Here is the JSON you asked for." => ERROR Extension exception: llm:chat-json: Response was not valid JSON: expected json value got "S" at index 0. Response text: Sure! Here is the JSON you asked for.
+ llm:history => []
+ O> llm:clear-history
+
+LLMChatJsonAcceptsNonObjectJson
+ extensions [llm]
+ globals [result]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set result llm:chat-json "__TEST_RESPOND:[1,2,3]"
+ result => "[1,2,3]"
+ length llm:history => 2
+ O> llm:clear-history
+
+LLMChatJsonHistoryUnchangedOnFailure
+ extensions [llm]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> carefully [ let r llm:chat-json "__TEST_FAIL" ] [ ]
+ llm:history => []
+ O> llm:clear-history
+
+LLMChooseStructured
+ extensions [llm]
+ globals [choice]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set choice llm:choose "Pick a direction" ["north" "south" "east" "west"]
+ member? choice ["north" "south" "east" "west"] => true
+ length llm:history => 2
+ O> llm:clear-history
+
+LLMChooseStructuredEmptyList
+ extensions [llm]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ llm:choose "Pick" [] => ERROR Extension exception: Choice list cannot be empty
+
+LLMChooseSendsEnumConstraint
+ extensions [llm]
+ globals [choice]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set choice llm:choose "__TEST_ECHO_FORMAT_ENUM" ["north" "south"]
+ choice => "south"
+ length llm:history => 2
+ item 1 (item 1 llm:history) => "south"
+ O> llm:clear-history
+ O> set choice llm:choose "__TEST_ECHO_FORMAT_ENUM" ["alpha" "beta" "gamma"]
+ choice => "gamma"
+ O> llm:clear-history
+
+LLMChatJsonSendsJsonObjectFormat
+ extensions [llm]
+ globals [result]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set result llm:chat-json "__TEST_ECHO_FORMAT"
+ result => "{\"format\":\"json_object\"}"
+ O> llm:clear-history
+
+LLMChatWithSchemaSendsSchemaFormat
+ extensions [llm]
+ globals [result]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set result llm:chat-with-schema "__TEST_ECHO_FORMAT" "{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"string\"}}}"
+ llm:get result "format" => "json_schema"
+ llm:get result "topLevelType" => "object"
+ llm:get result "name" => "netlogo_schema"
+ O> llm:clear-history
+
+LLMChooseDegradesWhenFormatIgnored
+ extensions [llm]
+ globals [choice]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set choice llm:choose "__TEST_IGNORE_FORMAT Pick a direction" ["north" "south" "east" "west"]
+ choice => "north"
+ length llm:history => 2
+ item 1 (item 1 llm:history) => "north"
+ O> llm:clear-history
+
+LLMChooseDegradesCaseInsensitively
+ extensions [llm]
+ globals [choice]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ O> set choice llm:choose "pick __TEST_IGNORE_FORMAT __TEST_RAW_CHOICE:NORTH" ["north" "south"]
+ choice => "north"
+ length llm:history => 2
+ O> llm:clear-history
+
+LLMChooseDegradesOnMalformedConstrainedJson
+ extensions [llm]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ llm:choose "__TEST_IGNORE_FORMAT __TEST_RAW_CHOICE:{\"choice\": pick" ["north" "south"] => ERROR Extension exception: llm:choose: response '{"choice": pick' did not match any choice. Choices: north, south
+ llm:history => []
+ O> llm:clear-history
+
+LLMChooseUnmatchedAfterDegradationLeavesHistoryClean
+ extensions [llm]
+ O> llm:set-api-key "test-key"
+ O> llm:set-provider "openai"
+ O> llm:clear-history
+ llm:choose "__TEST_IGNORE_FORMAT __TEST_RAW_CHOICE:up" ["north" "south"] => ERROR Extension exception: llm:choose: response 'up' did not match any choice. Choices: north, south
+ llm:history => []
+ O> llm:clear-history