refactor(binding-echo): extract semantic-reject into model-vector - #2445
Open
jfallows wants to merge 18 commits into
Open
refactor(binding-echo): extract semantic-reject into model-vector#2445jfallows wants to merge 18 commits into
jfallows wants to merge 18 commits into
Conversation
Introduces Embedding/EmbeddingHandler/EmbeddingContext/EmbeddingFactorySpi in runtime/engine, sibling to catalogs:/vaults:/guards:/stores:, plus the embeddings: config schema, EngineConfig plumbing, and a TestEmbedding engine-test-jar implementation, mirroring the existing catalog wiring end-to-end (EngineBuilder, EngineWorker, EngineRegistry, NamespaceRegistry, EngineContext). EmbeddingHandler.embed() supports both a synchronous local-only fast path and an async CompletionCallback-based variant (modeled on GuardHandler.reauthorize's sync/async split), so an implementation backed by a remote embedding API can resolve a vector without blocking the engine's reactor thread. Scoped to the core SPI + config + schema + test-jar; no embedding implementation module or binding-echo consumer yet.
Drop the synchronous fast-path overload and the NONE placeholder from EmbeddingHandler — a single async embed() is a smaller, less confusing contract, and NONE could not honor the strictly-later completion guarantee without a worker thread to dispatch onto. embed() now always defers to the next event-loop tick via EngineContext.dispatch(), the same mechanism StoreHandler and the async GuardHandler.reauthorize are built on; TestEmbeddingContext already wired its dispatcher that way.
Gives binding-echo an options: embedding:/reject:/threshold: block. When configured, each inbound message is embedded via the referenced embeddings: instance and compared by cosine similarity against a set of configured reject exemplars (also embedded once, lazily, with pending checks queued until ready). A message scoring at or above threshold against any exemplar ends the stream via doEnd() (a policy decision, not a protocol violation) instead of being echoed; everything else echoes exactly as before. Adds the echo.schema.patch.json options schema, EchoOptionsConfig/-Builder/ -Adapter in config/binding-echo.conf, and paired k3po client/server scripts plus spec-level (self-consistency) and runtime-level (live engine, driven by the engine's TestEmbedding double) IT coverage for both the reject and pass-through paths.
Adds embedding-djl.spec/.conf/(runtime) implementing the embeddings: "djl" type via DJL (Deep Java Library)'s Criteria/Predictor/ TextEmbeddingTranslatorFactory against sentence-transformers/ all-MiniLM-L6-v2 on DJL's PyTorch engine. Model resolution is auto-download via DJL's model zoo (djl:// URL), version-pinned, matching the incubator's lower offline/air-gapped bar. DJL and its transitive dependencies (jna, gson, commons-compress + transitives, slf4j-api) ship as automatic-module-only jars with no module-info.class, which conflicts with this repo's fully-modular JPMS discipline. Rather than accept an automatic-module dependency or wrap each third party jar in its own synthesizing shim module, embedding-djl shades and relocates the whole DJL dependency closure into its own module jar under an internal vendor package, so the module-info.java requires nothing beyond runtime engine and its own config module — verified via ModuleFinder that the resulting module is a complete, self-contained 146-package JPMS module. EmbeddingHandler.embed() runs DJL's blocking Predictor.predict() on a dedicated background thread, then hops back onto the calling engine worker via EngineContext.dispatch() to invoke the completion callback, honoring the strictly-later async contract without native inference ever blocking a worker thread. The narrow unit test verifies Criteria construction (engine, types, translator factory) without loading a real model, so it needs no network access or native library.
Adds runtime/common-vector, a small allocation-conscious utility for operations on fixed-size float[] vectors: similarity() (cosine, named generically since it's the only metric today) and normalize() (L2). similarity() never requires pre-normalized inputs -- it divides by both vectors' magnitudes internally -- so normalize() stays a fully independent, optional transform for callers optimizing repeated comparisons via plain dot product, never a hidden requirement of the concept an embedding vector represents. Refactors binding-echo's moderation feature to call Vectors.similarity() instead of its own private cosineSimilarity(), removing the duplicate implementation -- verified via binding-echo's full IT suite (6/6 passing) plus common-vector's own unit tests.
…flag DjlEmbeddingInfo was missing @Incubating, so type: djl was always accepted by the config schema regardless of ZILLA_INCUBATOR_ENABLED, bypassing the incubator safety gate that every other incubator component's *Info class already honors. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TD1DvMrvh3ypdadAYggTGu
Demonstrates the pluggable embeddings: concept end-to-end with a moderated echo server backed by embedding-djl (local MiniLM via DJL, no vendor API). Adds the missing SchemaTest for embedding-djl.conf's djl.schema.patch.json (mirrors catalog-inline.spec's pattern) and wires embedding-djl + common-vector into the Docker image's zpm manifest so the example's incubator dependency is actually packaged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TD1DvMrvh3ypdadAYggTGu
DJL's HuggingFace model-zoo resolution embeds package-shaped groupId strings directly into outbound HTTP requests to mlrepo.djl.ai — relocating those strings (required to keep the shaded module JPMS-clean) silently breaks that live catalog lookup, since the external server only recognizes the original, unrelocated names. That's a hard conflict between shading and DJL's HuggingFace resolution feature, not something fixable from our side. GloVe word-vector averaging needs only java.net.http (JDK-bundled), so it sidesteps shading, native libraries, and the JPMS ServiceLoader gymnastics entirely, at the cost of coarser semantic matching that is an acceptable tradeoff for an OSS example. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TD1DvMrvh3ypdadAYggTGu
…at DJL classes The earlier embedding-djl -> embedding-glove rename only rewrote *.java, *.xml, *.json, and *.yaml file contents; these two extension-less META-INF/services files kept naming the old Djl* classes as the fully qualified provider, which is what the runtime image's service-provider binding table actually reads. That's the real root cause of the "Provider ... DjlEmbeddingFactorySpi not found" failure seen when running the packaged image, reproducing on every zilla.yaml (EmbeddingFactory.instantiate() runs unconditionally at engine startup) regardless of whether embeddings: is configured. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TD1DvMrvh3ypdadAYggTGu
The previous regeneration used a non-recursive install, which lacked enough reactor context to see binding-pgsql-kafka/binding-risingwave's transitive dependencies (binding-kafka.spec, binding-proxy.spec, engine.spec) and their own transitive third-party licenses. A full clean install from repo root produces the complete, correct listing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TD1DvMrvh3ypdadAYggTGu
…ctors nlp.stanford.edu 301-redirects glove.6B.zip requests to downloads.cs.stanford.edu. HttpClient's default redirect policy is NEVER, so the redirect response body (not the zip) was being saved to the cache file, which then failed to parse as a zip.
cloud/docker-image sorts before incubator/embedding-glove in the reactor build order, and Maven only installs a module to the local repo once its own build completes. Without an explicit dependency, docker-image's assembly step ran before embedding-glove was installed, so the artifact was missing from the image's bundled repo; zpmw install then tried (and failed) to resolve it from the remote registry instead. Declaring the dependency here, matching every other incubator binding, forces Maven to build embedding-glove first.
io.aklivity.zilla.config.binding.echo never declared an exports clause, unlike every sibling *.conf module (e.g. binding-tcp.conf). This was latent while EchoOptionsConfig went unread across the module boundary; once EchoServerFactory needed to read its embedding/reject/threshold fields directly, module loading failed with: IllegalAccessError: class EchoServerFactory (in module io.aklivity.zilla.runtime.binding.echo) cannot access class EchoOptionsConfig (in module io.aklivity.zilla.config.binding.echo) because module io.aklivity.zilla.config.binding.echo does not export io.aklivity.zilla.config.binding.echo to module io.aklivity.zilla.runtime.binding.echo
…e directory
Previously cached to java.io.tmpdir, which is ephemeral across container
recreation in a real deployment, forcing a redownload of the 862MB GloVe
zip every restart. Mirrors binding-kafka's local cache convention: adds
GloveEmbeddingConfiguration.cacheDirectory(), resolved from
EngineConfiguration.ENGINE_CACHE_DIRECTORY (a "glove" subdirectory), and
threads it through GloveEmbedding -> GloveEmbeddingContext ->
GloveEmbeddingHandler instead of System.getProperty("java.io.tmpdir").
Also mounts a Docker volume over the cache directory in the
tcp.echo.embedding example so the download survives compose down/up.
0.85 was tuned for MiniLM sentence-transformer embeddings and didn't carry over to GloVe's mean-pooled word vectors, which compress this demo's phrases into a much narrower, higher band: the accepted "stray cat" message scored 0.92-0.93 against the reject phrases, above the old 0.85 threshold, so it was incorrectly rejected. Verified empirically against the real downloaded vectors: the accepted phrase's highest similarity to either reject phrase is 0.927, and each reject phrase (including the same-pattern-different-vocabulary one) scores at least 0.954 against one of the reject phrases. 0.94 sits cleanly between the two with margin on both sides. Confirmed against the full accept/reject/control matrix and the example's own .github/test.sh, all passing.
…vector Introduce a new incubator/model-vector model that rejects a value whose embedding is similar enough to a configured list of reject phrases, and port binding-echo onto it via the generic options.model mechanism instead of embedding-specific echo options. Any binding that already supports an inline model reference gets this behavior for free. The blocking gap was that ModelPipeline/ModelTransform was fully synchronous, but embedding lookups are async-only. Add a minimal, additive ModelStatus.SUSPENDED plus a 3-arg supplyDecoder/supplyEncoder(envelope, transform, resumed) overload to ModelHandler so a model can suspend mid-transform and resume via a callback dispatched back onto the correct engine worker thread -- with zero changes to model-json/model-avro/model-protobuf/model-core. Resolve the embedding's engine-scoped id via the existing generic NamedConfig/refs() resolution mechanism (new EmbeddedConfig type) rather than threading bindingId through new engine SPI overloads. Extend the engine's own type: test model and type: test binding test doubles (additively, all-new fields default off) to support SUSPENDED, so binding-echo's and model-vector's own k3po ITs can genuinely exercise the async suspend/resume path against a live engine without depending on each other's concrete modules. Along the way, fix a real bug this surfaced: TestBindingFactory was tearing down a stream's decode slot on END/ABORT while a model was still awaiting async resolution. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GD1PunXvfUv4Y69nB6YDUY
Avoid the model/model stutter in echo's options (options.model.model:
vector) by renaming the outer field to value, matching the engine's own
type: test binding precedent for "apply a model to the whole payload"
(options.value: { model: <type>, ... }) and MqttTopicConfig.content's
existing convention of never repeating "model" as both the outer field
name and the model's own type discriminator.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GD1PunXvfUv4Y69nB6YDUY
…loVe download CI job "testing (tcp.echo.embedding)" failed: docker compose up -d --wait reported the zilla container unhealthy after ~87s, while the GloVe vector download it waits on took several minutes to complete (observed ~9 minutes under similar conditions). The old healthcheck budget (start_period 15s + retries 15 x interval 5s ~= 90s) was sized for an already-cached download, not a cold one. Widen it to start_period 30s + retries 60 x interval 15s ~= 15.5 minutes so the first run in a fresh environment has room to finish.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
binding-echopreviously carried bespokeembedding:/reject:/threshold:options and its own inline moderation logic. That's a layering violation: echo is a transport-level example binding, and semantic rejection is business logic that belongs behind the existing generic model extension point (model-json,model-avro,model-protobuf,model-core), not bolted onto one binding.This PR extracts that logic into a new incubator model,
model-vector, which rejects a value whose embedding is similar enough to a configured list of reject phrases. Any binding that already supports an inlinemodel:reference gets this behavior for free;binding-echois ported onto it as the first consumer, driving the model generically throughoptions.modelinstead of embedding-specific options.The blocking gap:
ModelPipeline/ModelTransformwas fully synchronous, but an embedding lookup is inherently async. This PR adds a minimal, additiveModelStatus.SUSPENDEDplus a 3-argsupplyDecoder/supplyEncoder(envelope, transform, resumed)overload toModelHandler, so a model can suspend mid-transform and resume via a callback dispatched back onto the correct engine worker thread — with zero changes required tomodel-json/model-avro/model-protobuf/model-core.model-vectorresolves its referenced embedding's engine-scoped id via the engine's existing genericNamedConfig/refs()resolution mechanism (a newEmbeddedConfigtype, mirroringVaultedConfig/GuardedConfig) rather than threading a new lookup path through engine SPI.The engine's own
type: testmodel andtype: testbinding test doubles are extended additively (all new fields/behavior default off, so every existing consumer is unaffected) to supportSUSPENDED, so bothbinding-echo's andmodel-vector's own k3po integration tests can genuinely exercise the async suspend/resume path against a live engine without depending on each other's concrete modules. This surfaced and fixes a real bug in the test binding: it was tearing down a stream's decode slot onEND/ABORTwhile a model was still awaiting async resolution.Test plan
model-vector(accept / reject / suspend-resume) using the engine'sTestEmbeddingHandlermodel-vector's own k3po IT (VectorModelIT) using the engine's generictype: testbinding, exercising the real async suspend/resume path against a live enginebinding-echo's k3po ITs ported to the newoptions.modelshape, using the engine'stype: testmodel (extended withreject/suspend) instead of a concrete model dependencyruntime/enginefullverify(211 ITs) green after extendingTestBindingFactory/TestModelPipeline./mvnw clean installgreen, includingcloud/docker-imagetcp.echo.embeddingexample's Docker image and exercised accept / reject-by-exact-phrase / reject-by-semantic-paraphrase / control scenarios against the realembedding-glovebackend (not a test double)🤖 Generated with Claude Code
Generated by Claude Code