Add model package support for sdkv2 - #981
Conversation
Add package-aware layout handling, loading, cache behavior, and catalog compatibility filtering.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Copilot resolve the merge conflicts in this pull request |
b75395a to
a6ceeca
Compare
a6ceeca to
88a77c5
Compare
# Conflicts: # sdk_v2/cpp/src/ep_detection/ep_detector.cc Co-authored-by: xiaoyu-work <85524621+xiaoyu-work@users.noreply.github.com>
Resolved in 2922026 — merged |
There was a problem hiding this comment.
Pull request overview
Adds C++ SDK v2 support for downloading, discovering, filtering, and loading ORT model packages.
Changes:
- Adds package layout detection and cache handling.
- Adds package metadata parsing and hardware compatibility filtering.
- Loads selected package variants through ORT GenAI and expands tests.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
sdk_v2/cpp/CMakeLists.txt |
Builds the layout classifier. |
sdk_v2/cpp/src/util/model_layout.h |
Declares model layout classification. |
sdk_v2/cpp/src/util/model_layout.cc |
Implements layout detection. |
sdk_v2/cpp/src/model.cc |
Passes catalog EP and task metadata when loading. |
sdk_v2/cpp/src/model_info.h |
Defines typed package metadata. |
sdk_v2/cpp/src/model_info.cc |
Serializes package metadata. |
sdk_v2/cpp/src/inferencing/model_load_manager.h |
Extends the loading interface. |
sdk_v2/cpp/src/inferencing/model_load_manager.cc |
Handles package runtime metadata. |
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.h |
Tracks package and modality state. |
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.cc |
Loads packages through ORT GenAI. |
sdk_v2/cpp/src/ep_detection/ep_detector.h |
Adds compiled-model compatibility APIs. |
sdk_v2/cpp/src/ep_detection/ep_detector.cc |
Evaluates package compatibility. |
sdk_v2/cpp/src/download/inference_model_writer.h |
Documents package marker behavior. |
sdk_v2/cpp/src/download/inference_model_writer.cc |
Retains package root markers. |
sdk_v2/cpp/src/download/download_manager.h |
Updates download workflow documentation. |
sdk_v2/cpp/src/download/download_manager.cc |
Resolves package cache roots. |
sdk_v2/cpp/src/catalog/local_model_scanner.h |
Documents package discovery. |
sdk_v2/cpp/src/catalog/local_model_scanner.cc |
Discovers package roots without scanning variants. |
sdk_v2/cpp/src/catalog/azure_model_catalog.h |
Adds injectable catalog clients. |
sdk_v2/cpp/src/catalog/azure_model_catalog.cc |
Filters incompatible package entries. |
sdk_v2/cpp/src/catalog/azure_catalog_models.h |
Defines catalog package payloads. |
sdk_v2/cpp/src/catalog/azure_catalog_models.cc |
Parses package catalog metadata. |
sdk_v2/cpp/test/internal_api/model_load_manager_test.cc |
Tests package loading behavior. |
sdk_v2/cpp/test/internal_api/model_info_test.cc |
Tests metadata serialization. |
sdk_v2/cpp/test/internal_api/local_model_scanner_test.cc |
Tests package cache discovery. |
sdk_v2/cpp/test/internal_api/download_test.cc |
Tests package download and caching. |
sdk_v2/cpp/test/internal_api/catalog_cache_test.cc |
Tests cached package metadata. |
sdk_v2/cpp/test/internal_api/azure_catalog_test.cc |
Tests parsing and compatibility filtering. |
model-package-foundry-local-integration.md |
Documents the integration design and scope. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Restrict the catalog-task multimodal fallback to model packages so flat ASR models (e.g. nemotron_speech) are not routed through OgaMultiModalProcessor. Skip the provider override for kCPU, whose GenAI provider name is empty, so default-loaded CPU catalog models are not configured with an empty provider. Move the model package design doc under sdk_v2/cpp/docs and mark Stage 1 as implemented.
|
Copilot resolve the merge conflicts in this pull request |
# Conflicts: # sdk_v2/cpp/src/catalog/azure_model_catalog.cc # sdk_v2/cpp/src/catalog/azure_model_catalog.h Co-authored-by: xiaoyu-work <85524621+xiaoyu-work@users.noreply.github.com>
Resolved in 252ecef. Conflicts were in Verified with |
GetModelCompatibilityForEpDevices reset the device_type optional and then dereferenced it inside the device loop. GCC cannot correlate the has_value() guard with the deref across the loop and fails the Linux build under -Werror. Collapse the optional into plain locals before the loop. Behavior is unchanged: a disengaged optional and kNotSet both mean "no device filter".
cfbda91 to
d39c33c
Compare
The catalog EP is only needed to select a package variant via OgaConfig::CreateFromPackageEp. Applying it to flat models also converted kDefault to a concrete EP, which skipped the load manager's generic-gpu CUDA/WebGPU preference and forced explicit provider configuration on every flat GPU/NPU catalog model. Add ModelInfo::IsModelPackage and gate the override on it.
The scoping document is working material rather than shipped documentation. Kept locally, out of the repo.
Summary
Add support for ORT model packages in the C++ SDK — catalog, download, cache discovery, and loading — alongside the existing flat model directories.
A model package is a directory that bundles several compiled variants of the same model behind a top-level
manifest.json. ORT GenAI loads the directory directly and picks the variant matching the local hardware.Stage 1 scope: one package per execution provider, single component, multiple variants. Design notes:
sdk_v2/cpp/docs/ModelPackageIntegration.md.Changes
ClassifyModelLayout()insrc/util/model_layout.{h,cc}. A rootgenai_config.jsonmeans flat;manifest.jsonwithout it means package.model_load_managerparsesmanifest.jsonand merges the per-variantgenai_config.jsonfiles conservatively.GenAIModelInstanceusesOgaConfig::CreateFromPackageEp()so GenAI does the variant selection.variantMetadata.modelPackage, addModelInfo::IsModelPackage(), and hide entries whose package variants are all unsupported on the current machine (fails open when compatibility is unknown).EpDetector::GetModelCompatibilityForEpDevices()to evaluate a variant's compatibility string against registered EP devices.inference_model.jsonmarker at the package root, and treat a package as a leaf when scanning the cache so variants don't surface as separate models.Model::Load— for packages only, a default load adopts the catalog EP so the right variant is selected.