diff --git a/docs/dev/adapter_observability.md b/docs/dev/adapter_observability.md deleted file mode 100644 index 973b49de0..000000000 --- a/docs/dev/adapter_observability.md +++ /dev/null @@ -1,135 +0,0 @@ -# Adapter function lifecycle, options, and observability - -Epic #929 Phase 2, issue #1140. Covers three things landed together in the -same PR: the narrowed `AdapterMixin` verb contract, the shared -`resolve_model_options` helper, and the `AdapterFunctionMetricsPlugin` skeleton. - -## AdapterMixin verb contract - -`AdapterMixin` (`mellea/backends/adapters/adapter.py`) exposes **seven** -verbs, not the four stated in #1140's acceptance criteria. That's a direct -conflict with the issue text as written: Phase 1 (PR #1269) already added -`resolve_adapter()`, which depends on `base_model_name` and `add_adapter` -staying on the mixin, so trimming to four verbs isn't possible without -breaking Phase 1. The count below reflects what actually ships. - -### Universal (every backend implements these) - -- `base_model_name` — the underlying model's identifier. Read directly by - `resolve_adapter()` to construct new adapters lazily. -- `add_adapter(adapter)` — registers an adapter with the backend. - `resolve_adapter()` calls this internally the first time an adapter name - is resolved. -- `list_adapters()` — returns every adapter the backend *knows about*, - whether or not it's currently active. Both `LocalHFBackend` and - `OpenAIBackend` now share this "registered/known" contract: - `LocalHFBackend.list_adapters()` reads `self._added_adapters` (previously - it read `self._loaded_adapters`, which only included adapters that had - been explicitly loaded — that mismatch with `OpenAIBackend`'s semantics is - fixed as part of this issue). - -### Reality-specific (each backend overrides only its own) - -Each of the following raises `NotImplementedError` on the mixin by default; -a backend overrides only the verb matching its own adapter reality. - -- `load_peft_adapter(name)` / `unload_peft_adapter(name)` — LocalFile/PEFT - reality (`LocalHFBackend`). Loads or unloads LoRA/aLoRA weights from disk. - Renamed from the previous `load_adapter`/`unload_adapter`. -- `render_controls(name, active: bool)` — Embedded/Granite Switch reality - (`OpenAIBackend`). Weights are already baked into the served model, so - there's nothing to load or unload; this verb exists for future - control-token rendering. `active=True`/`False` map to the intended - `activate()`/`deactivate()` calls once #1142 wires EmbeddedBinding. -- `set_request_adapter(name)` — ServerMediated reality. No backend - implements this yet; the verb name is defined for when that reality is - built. - -`resolve_adapter()` and `adapter_scope()` are unchanged Phase 1 scaffolding -and out of scope for this issue — their real wiring into -`WeightsBinding.activate()`/`deactivate()` belongs to #1141/#1142. - -## resolve_model_options - -`mellea/backends/_options.py` centralizes the model-options merge logic that -`LocalHFBackend._simplify_and_merge` and `OpenAIBackend._simplify_and_merge` -each used to duplicate. Precedence, lowest to highest: - -```text -backend_defaults < helper_defaults < call_options -``` - -`remap` translates backend/caller-specific option names to `ModelOption` -keys before merging; `helper_defaults` is assumed to already be in -`ModelOption` key form. `call_intrinsic` (`mellea/stdlib/components/intrinsic/_util.py`) -also routes through this helper for its `TEMPERATURE: 0.0` default, so -caller-supplied `model_options` can't be silently clobbered by a hardcoded -default — the same class of bug PR #972 fixed elsewhere. - -## AdapterFunctionMetricsPlugin (skeleton) - -`mellea/telemetry/metrics_plugins.py` adds `AdapterFunctionMetricsPlugin`, hooking -`adapter_function_invocation_complete` and `adapter_function_phase_complete` -(`mellea/plugins/hooks/adapter_function.py`). Three metrics: - -- `mellea.adapter_function.invocations` (counter) — labels: `name`, `revision`, - `binding_type`, `adapter_type`, `outcome` (`success` | `schema_error` | - `error`). -- `mellea.adapter_function.phase_duration` (histogram, unit `s`) — - labels: `name`, `phase` (`prepare` | `activate` | `generate` | `parse` | - `deactivate`). -- `mellea.adapter_function.parse_failures` (counter) — labels: `name`, `revision`. - Incremented automatically whenever an invocation's `outcome` is - `schema_error` (i.e. an `AdapterSchemaMismatchError`), acting as a - schema-drift detector. - -No production code fires these hooks yet — this is a skeleton, unit-tested -against synthetic payloads only (`test/telemetry/test_metrics_plugins.py`). -Real wiring from `prepare`/`activate`/`generate`/`parse`/`deactivate` is -expected to go in with #1141 (LocalFileBinding) and #1142 (EmbeddedBinding). - -## Span tree (structure) - -Span *emission* ships with the Bindings (#1141/#1142) — no span code lands in -this PR. What this issue fixes is the *shape*, so the traces align with the -metrics and follow Mellea's existing tracing conventions rather than a bespoke -scheme. Spans are opened through the `start_*_span` helper family in -`mellea/telemetry/tracing.py` (mirroring `start_backend_span` / -`start_action_span`): the span is named by its operation with `gen_ai.*` set -where the semantic conventions apply, and Mellea-specific fields are attached -under the `mellea.*` prefix — the same convention as `mellea.action_type`, -`mellea.num_actions`, etc. - -An invocation opens one parent span with a child span per lifecycle phase: - -- **Parent** (the invocation) — carries `mellea.adapter_function.name`, - `mellea.adapter_function.revision`, `mellea.adapter_function.binding_type`, - `mellea.adapter_function.adapter_type`, and - `mellea.adapter_function.outcome`, mirroring the - `mellea.adapter_function.invocations` counter. -- **Children** (one per phase: `prepare`, `activate`, `generate`, `parse`, - `deactivate`) — each carries `mellea.adapter_function.phase` and - corresponds one-to-one with a `mellea.adapter_function.phase_duration` - histogram sample of the same phase. - -Note the deliberate split, consistent with the rest of Mellea: **metric labels -are bare** (`name`, `phase`, `revision`, …) while **span attributes are -`mellea.*`-prefixed** — same values, different surface, each following its -signal type's existing convention. - -## Content capture (`MELLEA_TRACES_CONTENT`) - -Span *metadata* — names, revisions, phase durations, outcomes — is always safe -to record. Adapter *input and output content* — prompts, retrieved documents, -generated text — is gated behind the **existing** `MELLEA_TRACES_CONTENT` -environment variable: the same content-capture gate Mellea's other spans -already use (it also honours `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`), -**off by default**, so traces never capture PII or proprietary content unless -explicitly opted in. When unset or falsey, the phase spans carry metadata only; -when set truthy, they additionally attach the adapter's input/output content. -The adapter-function spans **reuse this gate rather than introducing a new one**; -content attributes are attached when the Bindings (#1141/#1142) emit spans. - -(#1140's acceptance criteria named this `MELLEA_TRACE_CONTENT`; the real, -already-implemented variable is `MELLEA_TRACES_CONTENT` — see -`mellea/telemetry/tracing.py`.) diff --git a/docs/dev/constrained_decoding.md b/docs/dev/constrained_decoding.md deleted file mode 100644 index 4318443ef..000000000 --- a/docs/dev/constrained_decoding.md +++ /dev/null @@ -1,23 +0,0 @@ -# Constrained Decoding - -## How do constraints get defined? - -Should we be thinking bigger than pydantic? Should it be possible to pass arbitrary grammars? If so, what's the abstract interface for those? Should this be factored out into llm-io? - -## How do constraints get passed around? - -The `m` framework currently uses the `format` argument to pydantic schemas, **outside of model args**. Should we be using `@@@format@@@` within ModelArgs instead? Hendrik describes the behavior of model args like this (paraghased by Nathan): - -> If a keyword had meaning across multiple types of backends, and if it means the same thing in all of those backends but has different names, then we use the `@@@`-style args so that the user can pass these args across all backends in the same way. Otherwise, the arguments in model_args are passed along verbatim. - -This argues for `@@@format@@@` as opposed to a dedicated `format` option in the method signature. Or, in the alternative, for an entire re-think of ModelArgs. - -## Integration with grammar-targeted LLMs - -Some LLMs target generation in a particular grammar. Examples include: - * ALoRAs that target very simple grammars - * code generatorrs that target particular PLs - * models (or model modes) tuned to generate JSON - * models (or model modes) tuned to generate YAML or particular fragments of YAML (such as k8s configs) - -Should we be doing constrained decoding in these cases, or should we treat deviation from the grammar as an exception? Probably the answer is "it depends". Masataro had a nice idea of **taking the sum of logits of grammatically feasible completions** and ensuring that this sum is above some threshold. How would supporting this change the interface described in the "How do constraints get defined?" section? \ No newline at end of file diff --git a/docs/dev/generate_ctx_signature.md b/docs/dev/generate_ctx_signature.md deleted file mode 100644 index a5e58f064..000000000 --- a/docs/dev/generate_ctx_signature.md +++ /dev/null @@ -1,16 +0,0 @@ -# Splitting the `head` and `tail` of the Context on generate calls - -We have decided to split the context into an "action" and "the rest of the context"; i.e., instead of `generate : ctx, ... -> output`, we use `generate: action, ctx, ... -> output`. - - This "car/cdr" separation of the final element from the rest is done because there are many situations where many different requests are made over the same context. Examples include multiple requirement checking, rejection sampling, and so on. - -Advantages of this approach: - * shared context is referentially equal, which makes memory management extremely simple. - * Certain types of code -- especially requirement checking -- are much easier to write. Because the Context does not have to be deep-copied. - -Disadvantages of this approach: - * This solution is extremely specific to a few examples/patterns from stdlib. When we have `span`-based backends, there could be many different points in the span from which generation could continue. The solutino to that problem will sort of rhyme -- separating the generation target from th rest of the context.t However, the current signature is NOT a good solution. So it's possible we will have to change how this works in the fture. - * Not parsimonious with how context is normally used, and perhaps confusing, particularly in the most-common situation whwere the context is "just" a normal chat history. - * It is not yet clear what meaning this will have when contexts cannot be linearized. In particular: what if there's a poset and multiple generation opportunities within that poset? How do we "place the cursor"? Does this design choice make it harder to "place the cursor"? - * Contexts are not in fact immutable, so we have to be extremely careful about when a context gets modified, and may even need to introduce semaphores. - diff --git a/docs/dev/hook_system.md b/docs/dev/hook_system.md deleted file mode 100644 index 35bdae623..000000000 --- a/docs/dev/hook_system.md +++ /dev/null @@ -1,139 +0,0 @@ -# Mellea Plugin Hook System — Internal Design Notes - -> **User-facing documentation:** [Plugins & Hooks](../../docs/docs/concepts/plugins.mdx) covers usage, registration, execution modes, hook types reference, and patterns. This file retains only internal design rationale and decisions for contributors. - ---- - -## Design principles - -1. **Consistent interface**: All hooks follow the same async pattern with payload and context parameters -2. **Composable**: Multiple plugins can register for the same hook, executing in priority order -3. **Fail-safe**: Hook failures can be handled gracefully without breaking core execution -4. **Minimal intrusion**: Plugins are opt-in; default Mellea behavior remains unchanged without plugins. Plugins work identically whether invoked through a session (`m.instruct(...)`) or via the functional API (`instruct(backend, context, ...)`) -5. **Architecturally aligned**: Hook categories reflect Mellea's true abstraction boundaries — Session lifecycle, Component lifecycle, and the (Backend, Context) generation pipeline -6. **Code-first**: Plugins are defined and composed in Python. The `@hook` decorator and `Plugin` base class are the primary registration mechanisms; YAML configuration is a secondary option for deployment-time overrides -7. **Functions-first**: The simplest plugin is a plain async function decorated with `@hook`. Class-based plugins (via the `Plugin` base class) exist for stateful, multi-hook scenarios but are not required - ---- - -## Concurrency model - -Hooks use Python's `async`/`await` cooperative multitasking. Because Python's event loop only switches execution at `await` points, hook code won't be interrupted mid-logic. This means: - -- **Sequential when awaited**: Calling `await hook(...)` keeps control flow deterministic — the hook completes before the caller continues. -- **Race conditions only at `await` points**: Shared state is safe to read and write between `await` calls within a single hook. Races only arise if multiple hooks modify the same shared state and are dispatched concurrently. -- **No preemptive interruption**: Unlike threads, a hook handler runs uninterrupted until it yields control via `await`. - ---- - -## Hook invocation responsibilities - -Hooks are called from Mellea's base classes (`Component.aact()`, `Backend.generate()`, `SamplingStrategy.run()`, etc.). This means hook invocation is a framework-level concern, and authors of new backends, sampling strategies, or components do not need to manually insert hook calls. - -The caller (the base class method) is responsible for both invoking the hook and processing the result. Processing means checking the result for one of three possible outcomes: - -1. **Continue with original payload** — `PluginResult(continue_processing=True)` with no `modified_payload`. The caller proceeds unchanged. -2. **Continue with modified payload** — `PluginResult(continue_processing=True, modified_payload=...)`. The plugin manager applies the hook's payload policy, accepting only changes to writable fields and discarding unauthorized modifications. The caller uses the policy-filtered payload in place of the original. -3. **Block execution** — `PluginResult(continue_processing=False, violation=...)`. The caller raises or returns early with structured error information. - -Hooks cannot redirect control flow, jump to arbitrary code, or alter the calling method's logic beyond these outcomes. This is enforced by the `PluginResult` type. - ---- - -## Payload design principles - -1. **Strongly typed** — Each hook has a dedicated payload dataclass (not a generic dict). This enables IDE autocompletion, static analysis, and clear documentation of what each hook receives. -2. **Sufficient (maximize-at-boundary)** — Each payload includes everything available at that point in time. Post-hooks include the pre-hook fields plus results. This avoids forcing plugins to maintain their own state across pre/post pairs. -3. **Frozen (immutable)** — Payloads are frozen Pydantic models (`model_config = ConfigDict(frozen=True)`). Plugins cannot mutate payload attributes in place. To propose changes, plugins must call `payload.model_copy(update={...})` and return the copy via `PluginResult.modified_payload`. This ensures every modification is explicit and flows through the policy system. -4. **Policy-controlled** — Each hook type declares a `HookPayloadPolicy` specifying which fields are writable. The plugin manager applies the policy after each plugin returns, accepting only changes to writable fields and silently discarding unauthorized modifications. This separates "what the plugin can observe" from "what the plugin can change" — and enforces it at the framework level. -5. **Serializable** — Payloads should be serializable for external (MCP-based) plugins that run out-of-process. All payload fields use types that can round-trip through JSON or similar formats. -6. **Versioned** — Payload schemas carry a `payload_version` so plugins can detect incompatible changes at registration time rather than at runtime. -7. **Isolation** — Each plugin receives a copy-on-write (CoW) snapshot of the payload. Mutable containers (dicts, lists) are wrapped so mutations in one plugin do not affect others. Plugins should not cache payloads beyond the hook invocation — payload fields reference live framework objects (`Context`, `Component`, `MelleaSession`) whose lifecycle is managed by the framework. - ---- - -## GlobalContext design (ambient metadata) - -The `GlobalContext` passed to hooks carries lightweight, cross-cutting ambient metadata that is useful to every hook regardless of type. Hook-specific data (context, session, action, etc.) belongs on the **typed payload**, not on the global context. - -### What goes in GlobalContext - -```python -# GlobalContext.state — same for all hook types -backend_name: str # Derived from backend.model_id (when backend is passed) -``` - -The `backend_name` is a lightweight string extracted from `backend.model_id`. The full `backend` and `session` objects are **not** stored in GlobalContext — this avoids giving plugins unchecked mutable access to core framework objects. - -### Design rationale - -Previously, `context`, `session`, and `backend` were passed both on payloads and in `GlobalContext.state`, creating duplication. The same mutable object accessible via two paths was a footgun — plugins could be confused about which to read/modify. The refactored design: - -1. **Payloads** are the primary API surface — typed, documented, policy-controlled -2. **GlobalContext** holds only truly ambient metadata (`backend_name`) that doesn't belong on any specific payload -3. No mutable framework objects (`Backend`, `MelleaSession`, `Context`) are stored in GlobalContext - ---- - -## Design decision: separate success/error hooks - -`component_post_success` and `component_post_error` are separate hooks rather than a single `component_post` with a sum type over success/failure. The reasons are: - -1. **Registration granularity** — Plugins subscribe to only what they need. An audit logger may only care about errors; a metrics collector may only care about successes. -2. **Distinct payload shapes** — Success payloads carry `result`, `generate_log`, and `sampling_results`; error payloads carry `exception`, `error_type`, and `stack_trace`. A sum type would force nullable fields or tagged unions, adding complexity for every consumer. -3. **Different execution modes** — Error hooks may be fire-and-forget (for alerting); success hooks may be blocking (for output transformation). Separate hooks allow per-hook execution timing configuration. - ---- - -## Design decision: component_pre_create / component_post_create deferral - -`component_pre_create` and `component_post_create` are not implemented. `Component` is currently a `Protocol`, not an abstract base class. This means Mellea has no ownership over component initialization: there are no guarantees about when or how subclass `__init__` methods run, and there is no single interception point that covers all `Component` implementations. - -Placing hook calls inside `Instruction.__init__` and `Message.__init__` works for those specific classes, but it is fragile (any user-defined `Component` subclass is invisible to the hooks) and architecturally wrong (the hook system should not need to be threaded manually into every `__init__`). - -If `Component` were refactored to an abstract base class, Mellea could wrap `__init__` at the ABC level and fire these hooks generically for all subclasses. Until then, use `component_pre_execute` for pre-execution policy enforcement. - ---- - -## Unimplemented hooks - -The following hooks are designed but not yet implemented. They are included in the design for completeness and may be implemented as demand arises. - -| Hook Point | Category | Notes | -| --- | --- | --- | -| `component_pre_create` | Component Lifecycle | Blocked on Component-as-ABC refactoring (see above) | -| `component_post_create` | Component Lifecycle | Blocked on Component-as-ABC refactoring (see above) | -| `generation_stream_chunk` | Generation Pipeline | Per-chunk interception during streaming | -| `adapter_pre_load` | Backend Adapter Ops | Before `backend.load_peft_adapter()` | -| `adapter_post_load` | Backend Adapter Ops | After adapter loaded | -| `adapter_pre_unload` | Backend Adapter Ops | Before `backend.unload_peft_adapter()` | -| `adapter_post_unload` | Backend Adapter Ops | After adapter unloaded | -| `context_update` | Context Operations | When context changes (append/reset) | -| `context_prune` | Context Operations | When context is trimmed for token budget | -| `error_occurred` | Error Handling | Cross-cutting hook for unrecoverable errors | - ---- - -## Scoping implementation - -A single `PluginManager` instance manages all plugins. Plugins are tagged with an optional `session_id`. At dispatch time, the manager filters: global plugins (no session tag) always run; session-tagged plugins run only when the dispatch context matches their session ID. - -With-block scopes use the same `session_id` tagging mechanism. Each `with` block gets a unique UUID scope ID; the plugin manager filters plugins by scope ID at dispatch time and deregisters them by scope ID on exit. - ---- - -## YAML configuration (secondary) - -For deployment-time configuration, plugins can be loaded from YAML. This is useful for enabling/disabling plugins or changing priorities without code changes. The `disabled` mode (`PluginMode.DISABLED`) is available in YAML configuration for deployment-time control but is not exposed in Mellea's public `PluginMode` enum. - ---- - -## Custom hook types - -The plugin framework supports custom hook types for domain-specific extension points beyond the built-in lifecycle hooks. This is particularly relevant for agentic patterns (ReAct, tool-use loops, etc.) where the execution flow is application-defined. Custom hooks use the same `@hook` decorator and follow the same calling convention, payload chaining, and result semantics. As agentic patterns stabilize in Mellea, frequently-used custom hooks may be promoted to built-in hooks. - ---- - -## Functional API support - -The functional API (`instruct(backend, context, ...)`) does not require a session. Hooks still fire at the same execution points. If global plugins are registered, they execute. If no plugins are registered, hooks are no-ops with zero overhead. Session-scoped plugins do not apply because there is no session. diff --git a/docs/dev/intrinsics_and_adapters.md b/docs/dev/intrinsics_and_adapters.md deleted file mode 100644 index 3d1375921..000000000 --- a/docs/dev/intrinsics_and_adapters.md +++ /dev/null @@ -1,38 +0,0 @@ -# Intrinsics and Adapters -Note: Mellea currently only supports IntrinsicAdapters and Intrinsics. - -## Basics -In Mellea, intrinsics are a type of Component that signals one or more of the following to a backend: -- a special adapter must be used for generation -- the input/output for generation must be transformed in a particular way -- the model options must be modified in a particular way - -These changes only happen when the intrinsic is the "action" of the request. Intrinsics should usually not be used as an item in the context of generation (in fact, by default, Intrinsics have no string representation). - -These changes are specified by the Adapter that corresponds to a given Intrinsic. Matching happens based on the adapter name and type. - -## Parts of an Intrinsic -Intrinsics specify: -- an adapter name (ie requirement-check) -- types of adapters suitable to be used (ie alora) -- any kwargs necessary (ie a requirement like "make sure the last user message is...") - -## Parts of an Adapter -Adapters specify: -- compatible backends -- adapter type -- functions for getting a path to load them - -## Using Intrinsics -Mellea Intrinsics currently use the routines under `mellea.formatters.granite` for loading adapters and formatting input/outputs. This means Mellea only allows intrinsics/adapters that follow this pattern. - -## Needed Future Work -### Custom Adapters / Intrinsics -Mellea should support custom intrinsic / adapter implementations. To do this: -- make backend `_generate_from_intrinsic` functions generic and utilize only common adapter functions -- adapters must specify a transformation function that encapsulates the input/output modifications necessary for their generation requests - -### Concurrency Checks -Some backends (currently only LocalHFBackend) that allow adapters to be loaded, cannot independently utilize these adapters without impacting other generation requests. - -These backends should support a generation lock that ensures requests are only performed when the correct set of adapters (or no adapters) are active. diff --git a/docs/dev/mellea_library.md b/docs/dev/mellea_library.md deleted file mode 100644 index 3baf14898..000000000 --- a/docs/dev/mellea_library.md +++ /dev/null @@ -1,15 +0,0 @@ -# Mellea should be as close to a library as possible - -We should make it possible to use mellea as a library (as opposed to a framework). - -In the context of LLM applications, the library vs framework distinction really boils down to how you treat the backend. - -If a piece of software insists on having an exclusive handle on the backend, then that piece of software does not compose with any other piece of software that also insists on an exclusive handle. They both want to be privileged with respect to the backend, so they cannot "play well" together. The `outlines` library is a good example of software that could've been a library but instead acts like a framework. Even `granite-io` takes on a framework-like role when it decides to actually call the backend, as opposed to operating over strings (or perhaps chat histories). - -Writing LLM libraries is kind of difficult. There is a very strong instinct to try to grab control of the backend. Mellea is no exception. In the "intro path", mellea definitely behaves like a framework. We hide the actual backend objects (`PretrainedModel`, `openai.Client`, etc.) from the user. - -But we should try to make it easy for certain parts of mellea to be used as a library. There are many ways in which we could allow mellea to compose with other libraries: - -1. We could have a `m.start_session_with_shared_backend(client:openai.Client)` and similarly for local ollama models and transformers models. Everything would work mostly the same after that, except we would have to make much weaker assumptions about the state of the backend (e.g., cache and LoRAs). -2. We could strive to keep the `Formatter` logic completely separate from Backend-specific code, and the legacy model behavior should treat each Component like a standalone user message. This way people could use `mellea` components without using the `mellea` backend and context management code. -3. We could strive to keep the `Cache` strategies agnostic to the rest of the code base, and figure out what their interface should be with respect to various backend sdks (and transformers in particular) diff --git a/docs/dev/mify.md b/docs/dev/mify.md deleted file mode 100644 index ab3af6e94..000000000 --- a/docs/dev/mify.md +++ /dev/null @@ -1,73 +0,0 @@ -# mify - -In classical programming, object-orientation provides a way to couple data and functionality. -Classes have fields and methods. Fields store data and methods operate over that data. - -The mellea library allows you to interface with objects in the same way, but with the added benefit that an LLM can perform operations for you. - -```python -import mellea - -m = mellea.start_session() - - -class Circle: - """A circle is defined by its center and a radius.""" - center_x: float - center_y: float - radius: float - - -c = Circle(1, 0, 1) - -mify(c) - -# .query is used to compute things. -circumference: float = m.query(c, "compute the circumference of the circle", - format=float) - -# .transform is used to create a new class of the same type but mutated. -flipped_circle = m.transform(c, "Mirror the circle across the y axis.") -``` - -Let's consider a slightly more complicated example. - -```python -class Customer: - customer_id: int - name: str - age: int - email_addr: str - employer: str - meeting_notes: List[str] - - def __init__(customer_id: int): - ... - - def send_email(subject: str, body: str): - ... - - def get_meeting_notes() -> List[str]: - ... -``` - -... - -```python -ctx = mellea.SingleShotContext(backend=WatsonX("ibm/granite4")) - -customer = Customer(customer_id=42) -mify(c) - -meetings_summary = m.query(c, "Summarize the last three interactions with this customer.") - -email_body = ctx.instruct("Based upon the summary of notes from recent meetings, write an email body encouraging the customer to purchase three cases of self-sealing stembolts", grounding_context={"meetings_summary": meetings_summary}) - -email_subject = ctx.instruct("Write a subject for this sales email.", grounding_context={"email_body": email_body}) - -customer.execute("send an email.", email_body, email_subject) -``` - -For more examples and information, see -- [Mify Examples](../examples/mify.py) -- [Mify Implementation](../../mellea/stdlib/mify.py) diff --git a/docs/dev/requirement_aLoRA_rerouting.md b/docs/dev/requirement_aLoRA_rerouting.md deleted file mode 100644 index 163493445..000000000 --- a/docs/dev/requirement_aLoRA_rerouting.md +++ /dev/null @@ -1,90 +0,0 @@ -# Rerouting Requirement Actions in `Backend.generate_*` calls - -Backend will often re-route a `generate` call where `action : Requirement` to an ALora. This document explains how and why that happens. - -## The Requirement Rerouting Rule - -## The Simple Rule - -The simplest version of the Requirement Rerouting Rule is: - -> The most specific constraint checking method will be used when validating generic `Requirement`s. - -The actual rule is slightly more complicated. - -## The Actual Rule - -If a `Requirement` is validated using a backend that could either use a `requirement-check` aLoRA or perform an LLMaJ prompt on the underlying model, then the aLoRA is used for validation, even if the `backend.generate_from_context` method is called instead of the `backend._generate_from_intrinsic` method. - -There are three exceptions to this rule: -1. `Backend.default_to_constraint_checking_alora` is set to `False` (this parameter defaults to `True`). -2. The `Requirement` has a more specific subtype that indicates a more specific intent (`LLMaJRequirement`). -3. The `ALoRA` requirement checker throws an exception. - -There is an exception (or disambiguation) to the first exception: If the user provides an `ALoRARequirement`, then the `backend.generate_from_context` call is rerouted to the constraint checking LoRA, regardless of the value of `default_to_constraint_checking_alora`. - -## Decision Rationale - -### Background and Problem Statement - -The `stdlib` has a `Requirement` class whose `validate` behavior is an LLMaJ call. - -Suppose that the user creates a backend and then adds a generic constraint checking aLoRA: - -```python -from mellea import start_session -from mellea.core import Requirement -from mellea.backends.adapters import IntrinsicAdapter - -m = start_session( - "huggingface.LocalHFBackend:ibm-granite/granite-4.0-micro") - -# By default, the AloraRequirement uses a IntrinsicAdapter with "requirement-check". -m.backend.add_adapter(IntrinsicAdapter("ibm-granite/rag-intrinsics-lib", "requirement-check", base_model_name="granite-4.0-micro")) - -m.instruct( - "Corporate wants you to find the difference between these two strings:\n\naaa\naba") -assert m.validate(Requirement( - description="The answer should mention that one of the strings has the letter b while the other doesn't.")) -``` - -Both the underlying model and the aLoRA adapter know how to validate this requirement, so which should be used? - -## Alternatives to the Proposed Rule - -1. Avoid the problem by forcing the user to be more explicit. -2. Respect control flow in the backends/alora mixins, and have the MelleaSession or the user explicitly implement the appropriate control flow. -3. Have the `Requirement.validate` implementation specify whatever control flow is desired for that particular requirement. - -### Advantages - -1. Reduced cognitive load. To first approximation, there is a simple rule that produces unsurprising results. The exceptions are rare and require explicit intervention from the user. If these exceptions are used, the user almost certainly knows exactly what they are doing. -2. Control is retained. If the user wants to specify the precise semantics of their validate call, then they can use the mpore specific `LLMaJRequirement` and `ALoraRequirement` classes. -3. The backend is the one that needs to make the choice about whether to handle KV cache. - - -### Disadvantages - -All backends that implement the aLoRA mixin need to implement this semantics. - - * This might be a blessing in disguise. It's actually not clear that ALora context construction can be done WLOG outside of the specific backend. - * That code is written rarely in any case. - * Depending on the truth of the first bullet point's conjecture, we can mitigate by implementing this routing in `m.validate` so that even if a backend contributor gets this wrong the proper behavior is still usually observed by most users. - -## Phase 1 change (Epic #929, issue #1136) - -Backends now use a **capability-based lookup** to find the `requirement-check` -adapter, replacing the old `isinstance` + `AdapterType` check: - -```python -# Before (Phase 0) -adapter = get_adapter_for_intrinsic("requirement-check", [AdapterType.ALORA], self._added_adapters) - -# After (Phase 1) -adapter = self._find_adapter("requirement-check", ("alora",)) -``` - -The logical rule (three exceptions above) is unchanged. The change is purely -in how the matching adapter is located: capability name and adapter type are -now read from `adapter.identity` (the new `Identity` dataclass introduced in -Phase 0, issue #1134) rather than derived from the adapter's class hierarchy. diff --git a/docs/dev/spans.md b/docs/dev/spans.md deleted file mode 100644 index 28c19d3b9..000000000 --- a/docs/dev/spans.md +++ /dev/null @@ -1,20 +0,0 @@ -# Design Document for Spans - -## Span Contexts - -We will introduce a SpanContext which will behave kind of like a heap but with transformer-running-on-GPU memory primitives instead of malloc/realloc/free. The public interface to a SpanContext will roughly correspond to the sort of stuff you can do in Span algebras, if you've seen some of that work. - -## Mapping STDLIB to Spans - -There are two broad philosophies to choose from for Spans. - -### The Span Representation Approach - -All Components and CBlocks get a __span_repr__ which maps the all things to a Span representation. The Component owner is responsible for saying how something gets represented as a Span, and is also responsible for defining caching boundaries (via a cache_boundary tag). - -### The Span Formatter Approach - -There is a Formatter which maps Components and CBlocks to Spans, as a pure function. Similar to how the TemplateFormatter works today. - -We need to document which approach we choose and discuss why it was chosen. - diff --git a/docs/dev/tool_calling.md b/docs/dev/tool_calling.md deleted file mode 100644 index fff491d94..000000000 --- a/docs/dev/tool_calling.md +++ /dev/null @@ -1,73 +0,0 @@ -# Tool Calling - -## Problem Statement - -Context management and execution of tool calls are inextricably linked, because most -models expect the output of a tool call to be added to the context at the -moment when the too lcall happens. This means that the `Session` must own the -code that actual performs a tool call. - -This is annoying because *what to do with a tool call* -- or even *how to -implement a tool call* -- is going to vary from application to application. - -We are then faced with two options: - -1. Provide some sort of object protocol for handling tool calls, whereby the - client responsible for tool calling is also responsible for executing a - callback on the session which appropriately modifies the session's context - in light of the tool response; or, -2. Come up with a small number of ways in which a tool may be called, and - expose those in the session. Anyone who wants to do something more complex - must then extend the Session class and implement their own too lcalling - logic. - -## Proposals - - -### Tool Calling Protocol Option - -Basically (2). - -Certain things such as `transform` have a default semantics in the -`MelleaSession` base class. - -For anyone who wants to do free-form tool calling, -there is a `MelleaSessionToolProtocol` mixin which must be inherited from and -implemented. - -### Nothing Fancy Option - -Pass back the `ModelOutputThunk` with tool calls, and do nothing else. - -Note that we already have a `ctx.insert` function, si instead of a mixin with -a protocol, the user is just supposed to know what they are supposed to do and -then use `m.ctx.insert` to implement the relevant logic. - -This is what's done with openai sdk in the status quo anyways. - -### Compromise? - -Can this be implemented such that if you don't specify a tool calling protocol -implementation then the behavior is equivalent to the Nothing Fancy Option? -Probably so. - - -## Final Proposal - -The ModelOutputThunk has a `tools` field where parsed tool calls are surfaced -to the user. This already exists and probably does not need additional -modification. - -1. For certain special tool calling protocols, the Session handles things - automatically for the user. E.g., `m.transform` and `m.query`. We need to - specify the precise semantics for what happens when a user provides tools - in the model_options when using `m.transform` -- probably, you flow through - into the next two cases. -2. If the `Session` has a `SessionToolCallingProtocol` implemented, then the - `def tool_call_result(...)` on that protocol must be called by the user - after a tool is executed. When that method is called, the context is - updated appropriately. We can also provide a `def call_tool(tool)` method - for convenience, which does both the tool call and the context management - for the user. -3. Otherwise, nothing happens. The user is responsible for updating their - context as needed. diff --git a/docs/docs/advanced/lora-and-alora-adapters.md b/docs/docs/advanced/lora-and-alora-adapters.md index da5594530..51ffaa647 100644 --- a/docs/docs/advanced/lora-and-alora-adapters.md +++ b/docs/docs/advanced/lora-and-alora-adapters.md @@ -11,7 +11,7 @@ schemes not well-represented in general training data. Mellea lets you train a [aLoRA](https://github.com/IBM/activated-lora) adapter on your own labeled dataset and use it as a requirement validator in any Mellea program. -**Prerequisites:** `pip install "mellea[cli]"`. Training requires a GPU or +**Prerequisites:** `pip install "mellea[cli,hf]"`. Training requires a GPU or Apple Silicon Mac with sufficient VRAM for the chosen base model. Uploading requires a Hugging Face account. @@ -116,26 +116,35 @@ m alora upload ./checkpoints/my_adapter \ ## Use the adapter in Mellea -Load the trained adapter into a `LocalHFBackend` using `CustomIntrinsicAdapter`: +Load the trained adapter into a `LocalHFBackend` using `CustomIntrinsicAdapter`. + +> **Note:** `CustomIntrinsicAdapter` is deprecated in favor of the `Adapter` / +> `WeightsBinding` model, but is still the only working way to load a +> locally-trained custom adapter — `Adapter`'s weight-loading is not yet +> implemented. This example will move to `Adapter` once that lands; see #1144. ```python from mellea.backends.huggingface import LocalHFBackend from mellea.backends.adapters.adapter import CustomIntrinsicAdapter from mellea.stdlib.context import ChatContext from mellea import MelleaSession -from mellea.stdlib.requirements import req +from mellea.stdlib.requirements import ALoraRequirement backend = LocalHFBackend(model_id="ibm-granite/granite-3.2-8b-instruct") adapter = CustomIntrinsicAdapter( - model_id="your-org/my-adapter", # HF repo ID or local checkpoint path + model_id="your-org/my-adapter", # HF repo ID or local checkpoint path base_model_name="granite-3.2-8b-instruct", + intrinsic_name="custom-failure-check", ) backend.add_adapter(adapter) m = MelleaSession(backend, ctx=ChatContext()) -failure_check = req("The failure mode must not be 'no_failure'.") +failure_check = ALoraRequirement( + "The failure mode must not be 'no_failure'.", + intrinsic_name="custom-failure-check", +) result = m.instruct( "Write a triage summary based on this technician note: {{note}}", user_variables={"note": "High vibration at 3100 RPM, connecting rod suspected."}, @@ -145,9 +154,39 @@ print(str(result)) # Output will vary — LLM responses depend on model and temperature. ``` -When `backend.add_adapter()` is called, Mellea automatically routes requirement -validation through the adapter for any `req()` calls on that session. The adapter -runs at the `check_requirement` prompt position — fast, with minimal context overhead. +> **Note:** `CustomIntrinsicAdapter` emits an advisory `UserWarning` because +> custom capability names are not part of Mellea's built-in capability registry. +> The adapter is still registered for routing. + +`ALoraRequirement` routes validation through the adapter with the matching +`intrinsic_name`. Create `CustomIntrinsicAdapter` before the requirement: its +compatibility shim registers the custom name, which lets `ALoraRequirement` +resolve it. The adapter runs at the `check_requirement` prompt position. Its +`io.yaml` must transform the output into the +`{"requirement_check": {"score": }}` response schema; label-only +adapter output is not compatible with `ALoraRequirement`. + +## How automatic routing works + +When an adapter is loaded via `backend.add_adapter()`, Mellea automatically routes +`req()` validation calls through it rather than falling back to LLM-as-a-judge. The +rule is: use the most specific available method. In practice this means the aLoRA +adapter is preferred whenever one is loaded, with three exceptions: + +1. `backend.default_to_constraint_checking_alora` is set to `False` — the adapter + is loaded but routing is suppressed for the entire backend instance. +2. The requirement uses the `LLMaJRequirement` subtype explicitly — the caller is + asking for LLM-as-a-judge regardless of what adapters are loaded. +3. The adapter is unavailable (e.g. cannot be loaded) — Mellea falls back to + LLM-as-a-judge automatically. This is the *only* fallback case: if the + adapter runs but its output fails schema validation, the error propagates + rather than silently falling back. + +If you want to force the adapter path even when using `generate_from_context` +directly (bypassing the normal `validate()` call), use `ALoraRequirement` from +`mellea.stdlib.requirements` — this bypasses `default_to_constraint_checking_alora`, +but still requires a matching adapter to actually be registered. If none is found, +Mellea logs a warning and falls back to regular generation rather than erroring. ## Disable adapter validation diff --git a/docs/docusaurus.config.ts b/docs/docusaurus.config.ts index 77d7bb7aa..aa0c2c7ca 100644 --- a/docs/docusaurus.config.ts +++ b/docs/docusaurus.config.ts @@ -82,8 +82,10 @@ const config: Config = { {from: '/integrations/bedrock-and-watsonx', to: '/integrations/bedrock'}, {from: '/integrations/huggingface-and-vllm', to: '/integrations/huggingface'}, {from: '/integrations/langchain-and-smolagents', to: '/integrations/langchain'}, + {from: '/dev/adapter-observability', to: '/observability/tracing'}, {from: '/dev/constrained-decoding', to: '/advanced/mellea-core-internals'}, {from: '/dev/generate-ctx-signature', to: '/advanced/mellea-core-internals'}, + {from: '/dev/hook-system', to: '/concepts/plugins'}, {from: '/dev/intrinsics-and-adapters', to: '/advanced/intrinsics'}, {from: '/dev/mellea-library', to: '/concepts/generative-programming'}, {from: '/dev/mify', to: '/concepts/mobjects-and-mify'}, diff --git a/docs/examples/README.md b/docs/examples/README.md index 58a11bd66..2d565e3dc 100644 --- a/docs/examples/README.md +++ b/docs/examples/README.md @@ -261,7 +261,6 @@ Utility functions used across examples. - **Main README**: [../../README.md](../../README.md) - **Agent Guidelines**: [../../AGENTS.md](../../AGENTS.md) -- **Dev Docs**: [../dev/](../dev/) ## 🏃 Running Examples @@ -288,4 +287,4 @@ uv run pytest test/ ## 🤝 Contributing -Found a bug or have an improvement? See [../../AGENTS.md](../../AGENTS.md) for contribution guidelines. \ No newline at end of file +Found a bug or have an improvement? See [../../AGENTS.md](../../AGENTS.md) for contribution guidelines. diff --git a/docs/examples/agents/README.md b/docs/examples/agents/README.md index 869540991..6fb7e43ef 100644 --- a/docs/examples/agents/README.md +++ b/docs/examples/agents/README.md @@ -36,5 +36,4 @@ An alternative implementation of the ReACT pattern using Mellea's instruct-valid ## Related Documentation -- See `docs/dev/tool_calling.md` for more on tool integration - See `mellea/stdlib/requirements/tool_reqs.py` for tool requirements \ No newline at end of file diff --git a/docs/examples/context/README.md b/docs/examples/context/README.md index e7b8b3752..2cb44619f 100644 --- a/docs/examples/context/README.md +++ b/docs/examples/context/README.md @@ -95,4 +95,3 @@ ctx = WindowCompactor(size=0).compact(ctx) # drop body, kee - See `mellea/stdlib/context/` for context and compactor implementations - See `mellea/stdlib/sampling/` for sampling strategies - See `mellea/stdlib/frameworks/react.py` for the ReACT loop -- See `docs/dev/spans.md` for context architecture details diff --git a/docs/examples/generative_stubs/README.md b/docs/examples/generative_stubs/README.md index ae72b8483..bf9d2947b 100644 --- a/docs/examples/generative_stubs/README.md +++ b/docs/examples/generative_stubs/README.md @@ -57,5 +57,4 @@ with start_session() as m: ## Related Documentation -- See `mellea/stdlib/components/genstub.py` for implementation -- See `docs/dev/mellea_library.md` for design philosophy \ No newline at end of file +- See `mellea/stdlib/components/genstub.py` for implementation \ No newline at end of file diff --git a/docs/examples/instruct_validate_repair/README.md b/docs/examples/instruct_validate_repair/README.md index 013654f19..49de71465 100644 --- a/docs/examples/instruct_validate_repair/README.md +++ b/docs/examples/instruct_validate_repair/README.md @@ -169,5 +169,4 @@ result = m.instruct( ## Related Documentation - See `mellea/stdlib/requirements/` for requirement types -- See `mellea/stdlib/sampling/` for sampling strategies -- See `docs/dev/mellea_library.md` for design philosophy \ No newline at end of file +- See `mellea/stdlib/sampling/` for sampling strategies \ No newline at end of file diff --git a/docs/examples/intrinsics/README.md b/docs/examples/intrinsics/README.md index dddf197f7..07c0961e7 100644 --- a/docs/examples/intrinsics/README.md +++ b/docs/examples/intrinsics/README.md @@ -164,5 +164,4 @@ Full example showing multiple adapter functions working together in a RAG pipeli - See `mellea/stdlib/components/intrinsic/` for adapter function implementations - See `mellea/backends/adapters/` for adapter system -- See `docs/dev/intrinsics_and_adapters.md` for architecture details - See `docs/docs/examples/granite-switch/README.md` for more about granite-switch \ No newline at end of file diff --git a/docs/examples/melp/README.md b/docs/examples/melp/README.md index 7610b7994..14fe0b2f8 100644 --- a/docs/examples/melp/README.md +++ b/docs/examples/melp/README.md @@ -54,5 +54,4 @@ actual_result = force(composed) ## Related Documentation -- See `mellea/stdlib/functional.py` for functional programming primitives -- See `docs/dev/mellea_library.md` for design philosophy \ No newline at end of file +- See `mellea/stdlib/functional.py` for functional programming primitives \ No newline at end of file diff --git a/docs/examples/mify/README.md b/docs/examples/mify/README.md index 0c3f6b06b..1e346ae69 100644 --- a/docs/examples/mify/README.md +++ b/docs/examples/mify/README.md @@ -93,5 +93,4 @@ Objects decorated with `@mify` implement the `MifiedProtocol`, which provides: ## Related Documentation - See `mellea/stdlib/components/mify.py` for implementation -- See `docs/dev/mify.md` for design details - See `mellea/templates/` for template system \ No newline at end of file diff --git a/mellea/backends/huggingface.py b/mellea/backends/huggingface.py index b44af1814..f6a31c138 100644 --- a/mellea/backends/huggingface.py +++ b/mellea/backends/huggingface.py @@ -499,7 +499,9 @@ async def _generate_from_context( # Requirements can be automatically rerouted to a requirement adapter. if isinstance(action, Requirement): - # See docs/dev/requirement_aLoRA_rerouting.md + # See "How automatic routing works" in + # docs/docs/advanced/lora-and-alora-adapters.md for the three + # exceptions to this rule. reroute_to_alora = self.default_to_constraint_checking_alora adapter_name = "requirement-check" diff --git a/mellea/core/backend.py b/mellea/core/backend.py index adc911496..352f3b6a4 100644 --- a/mellea/core/backend.py +++ b/mellea/core/backend.py @@ -75,8 +75,8 @@ async def generate_from_context( """Generates a model output from a context. May not mutate the context. This must be called from a running event loop as it creates a task to run the generation request. Args: - action: The last item of the context should be passed in as an `action` instead of as part of the `ctx`. See `docs/dev/generate_signature_decisions.md`. - ctx: The rest of the context. + action: The component to generate from, passed separately from `ctx`. + ctx: The rest of the context, excluding `action`. format: A response format to used for structured outputs / constrained decoding. model_options: Any model options to upsert into the defaults for this call. tool_calls: If `True`, then tool calls are extracts from the `action` `Component`. Assumption: if tool_calls is enabled, then the action `Component` has a TemplateRepresentation @@ -143,8 +143,8 @@ async def _generate_from_context( """Backend implementers should override this method to generate the actual response. Args: - action: The last item of the context should be passed in as an `action` instead of as part of the `ctx`. See `docs/dev/generate_signature_decisions.md`. - ctx: The rest of the context. + action: The component to generate from, passed separately from `ctx`. + ctx: The rest of the context, excluding `action`. format: A response format to used for structured outputs / constrained decoding. model_options: Any model options to upsert into the defaults for this call. tool_calls: If `True`, then tool calls are extracts from the `action` `Component`. Assumption: if tool_calls is enabled, then the action `Component` has a TemplateRepresentation diff --git a/mellea/stdlib/sampling/sampling_algos/budget_forcing_alg.py b/mellea/stdlib/sampling/sampling_algos/budget_forcing_alg.py index e7e34bc29..83eec143d 100644 --- a/mellea/stdlib/sampling/sampling_algos/budget_forcing_alg.py +++ b/mellea/stdlib/sampling/sampling_algos/budget_forcing_alg.py @@ -49,7 +49,8 @@ async def think_budget_forcing( Args: backend: OllamaModelBackend instance to use for generation. action: The last item of the context, passed as an `action` instead of as part - of the `ctx`. See `docs/dev/generate_signature_decisions.md`. + of the `ctx`. See `Backend.generate_from_context` for the rationale + behind the action/context split. ctx: The current conversation context. format: Optional Pydantic model for constrained decoding of the response. tool_calls: If `True`, tool calling is enabled. diff --git a/mellea/telemetry/metrics.py b/mellea/telemetry/metrics.py index f23772b69..f0a3910c8 100644 --- a/mellea/telemetry/metrics.py +++ b/mellea/telemetry/metrics.py @@ -1004,8 +1004,7 @@ def _get_adapter_function_invocations_counter() -> Any: # in the codebase, pre-existing, already-shipped `Intrinsic*` symbols # (the `Intrinsic` component, `call_intrinsic`, etc.) still use the old # name and are renamed in a later, coordinated phase of Epic #929 (#1136) - # rather than here. See docs/dev/adapter_observability.md for the full - # rationale. (Applies to all three metrics below.) + # rather than here. (Applies to all three metrics below.) _adapter_function_invocations_counter = create_counter( "mellea.adapter_function.invocations", description="Total number of adapter function invocations",