Skip to content

feat(search): support multimodal hybrid queries and AML payloads - #6128

Merged
qin-ctx merged 6 commits into
volcengine:mainfrom
heaoxiang-ai:codex/aml-multimodal-add-20261015
Oct 16, 2026
Merged

qin-ctx merged 6 commits into
volcengine:mainfrom
heaoxiang-ai:codex/aml-multimodal-add-20261015

Conversation

@heaoxiang-ai

@heaoxiang-ai heaoxiang-ai commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Description

AML can now write image-bearing histories and search with image-only, mixed text/image, or multiple-image queries. Native OpenViking hybrid retrieval preserves images in the existing dense channel and sends only text to sparse, instead of rejecting image queries because sparse cannot consume media. Existing string Add/Search requests retain their behavior.

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

N/A.

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Bug fix (non-breaking change that fixes an issue)
  • Documentation update
  • Test update

Changes Made

  • Accept typed text/image content arrays in AML Add and forward them as native Session parts; preserve strings, roles, timestamps, commit/polling, and authentication.
  • Forward AML Search images directly to native Find. Extend native image_url and SDK image normalization to accept multiple images in order. Responses remain text memory evidence.
  • Make composite hybrid capability follow its dense channel. Dense receives complete mixed content; sparse receives text only and is skipped for image-only input.
  • For Volcengine hybrid, retain the existing single combined request for text. Mixed input uses a multimodal dense request plus a separate text-only sparse request, using the same configured model and dense dimension. Both sync and async paths are covered.
  • Distinguish caller-supplied targets from expanded defaults: unscoped image Find/Search retains the resource default; explicit memory targets or context-type filters remain supported.
  • Validate inline PNG/JPEG/WebP Base64 and decoded limits: 10 MiB per image, 30 MiB per Add or Search request.

Testing

  • Added feature and compatibility tests
  • Focused suites: 201 tests passed on the latest-main-based branch, including 72 new Find/Search cases across USER/ADMIN/ROOT, image/mixed/text/code input, default scope, memory URI, and memory type filter. Before the fix, 8 default-scope image/mixed cases failed; all pass after the fix.
  • 90 legacy Textual/Coding cases compared with the previous Add implementation: 78 successful request/response/backend-call matches and 12 matching validation statuses
  • Exact ECS adapter overlay: 49 contract/rerank tests passed
  • Real provider sync/async matrix: text, image-only, mixed, and multiple images; verified 2048-dimensional dense output, request modalities, sparse text isolation, and image-only sparse omission
  • Isolated Linux staging and public ECS checks: image/text/code retrieval, user isolation, authentication, Top20, two 100-document rerank batches for existing text traffic, and existing memory file hashes
  • Ruff lint and git diff --check; unrelated formatting changes removed and existing formatting preserved (AST and comments unchanged by cleanup)
  • macOS
  • Linux

Checklist

  • My code follows the project's coding style
  • I have performed a self-review
  • Added comments where routing behavior is non-obvious
  • Updated the documentation
  • Full repository test suite passes

Screenshots

N/A.

Additional Notes

The dense model and dimension must remain compatible with the stored index. This change does not add query captioning or require a separate VLM configuration. Image-only queries use native vector order where a deployment has a text-only reranker; mixed queries can still rerank with their accompanying text.

Earlier CI failed on unchanged upstream session documentation referencing the removed POST /api/v1/sessions/{}/extract route in both languages. The same API-reference checker fails locally; those session docs and router are unchanged by this PR. Earlier API & CLI integration CI also failed: the filesystem tests receive provider 401 AuthenticationError (API key format is incorrect), surfaced as HTTP 500 on wait=true writes. The log does not establish a regression in the changed hybrid path; a run with valid provider credentials remains needed. An additional optional service-lifecycle test stalled and was interrupted; no full-suite pass is claimed. Existing dependency deprecation warnings remain.

@heaoxiang-ai
heaoxiang-ai force-pushed the codex/aml-multimodal-add-20261015 branch from d2ae040 to 73cf7aa Compare October 15, 2026 13:09
@heaoxiang-ai heaoxiang-ai changed the title feat(benchmark): accept multimodal content in AML Add feat(search): support multimodal hybrid queries and AML payloads Oct 15, 2026
@ZaynJarvis ZaynJarvis added the kernel Core server, runtime, storage, retrieval, SDK, and CLI. label Oct 16, 2026
@heaoxiang-ai
heaoxiang-ai force-pushed the codex/aml-multimodal-add-20261015 branch from 359f539 to d8283a5 Compare October 16, 2026 03:46
@qin-ctx
qin-ctx merged commit 855d092 into volcengine:main Oct 16, 2026
4 of 6 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog to Done in OpenViking project Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel Core server, runtime, storage, retrieval, SDK, and CLI.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants