Skip to content

[CoreAI] Integrate the delegate execute path #23393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
metascroy wants to merge 1 commit into
base: coreai-v2/05a-load-coordinator
Choose a base branch
from
Draft

[CoreAI] Integrate the delegate execute path #23393

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -878,6 +878,7 @@ endif()

if(EXECUTORCH_BUILD_COREAI)
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/backends/apple/coreai)
list(APPEND _executorch_backends coreaidelegate)
endif()

if(EXECUTORCH_BUILD_MLX)
Expand Down
61 changes: 57 additions & 4 deletions backends/apple/coreai/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@ endforeach()
enable_language(OBJCXX)

set(_coreai_runtime_sources
runtime/coreai_assets.mm runtime/coreai_storage.mm
runtime/coreai_bookmarks.mm runtime/coreai_load_coordinator.mm
runtime/coreai_backend.mm runtime/coreai_assets.mm
runtime/coreai_storage.mm runtime/coreai_bookmarks.mm
runtime/coreai_load_coordinator.mm
)

# Keep ARC and C++ exception settings away from the private Swift module.
Expand Down Expand Up @@ -57,14 +58,21 @@ if(EXECUTORCH_BUILD_TESTS)
runtime/test/coreai_fake_loader.mm
runtime/test/coreai_acquisition_test.mm
runtime/test/coreai_acquisition_fixture.mm
runtime/test/coreai_delegate_test.mm
runtime/ETCoreAITensor.mm
${_coreai_runtime_sources}
)
coreai_configure_objc_target(coreai_host_test)
target_compile_definitions(coreai_host_test PRIVATE COREAI_ASSETS_TESTING=1)
find_library(COREAI_FOUNDATION_FRAMEWORK Foundation REQUIRED)
# The consolidated runtime target is created after this directory.
if(EXECUTORCH_BUILD_SHARED)
set(_coreai_host_runtime executorch_shared)
else()
set(_coreai_host_runtime executorch)
endif()
target_link_libraries(
coreai_host_test PRIVATE executorch GTest::gtest
coreai_host_test PRIVATE ${_coreai_host_runtime} GTest::gtest
${COREAI_FOUNDATION_FRAMEWORK}
)
add_test(NAME coreai_host_test COMMAND coreai_host_test)
Expand Down Expand Up @@ -277,8 +285,53 @@ if(EXECUTORCH_BUILD_TESTS
set_tests_properties(coreai_swift_bridge_test PROPERTIES TIMEOUT 120)
endif()

if(EXECUTORCH_BUILD_SHARED)
set(_coreai_library_type SHARED)
set(_coreai_runtime executorch_shared)
set(_coreai_data_loader "")
else()
set(_coreai_library_type STATIC)
set(_coreai_runtime executorch)
set(_coreai_data_loader extension_data_loader)
endif()
add_library(
coreaidelegate ${_coreai_library_type} ${_coreai_runtime_sources}
$<TARGET_OBJECTS:coreai_bridge_obj>
)
coreai_configure_objc_target(coreaidelegate)
# The delegate contains no Swift sources; its archive is assembled by C++.
set_target_properties(coreaidelegate PROPERTIES LINKER_LANGUAGE CXX)
coreai_require_os27(coreaidelegate)
target_include_directories(coreaidelegate PUBLIC ${_common_include_directories})
add_dependencies(coreaidelegate coreai_swift)
target_link_libraries(
coreaidelegate PRIVATE coreai_swift ${_coreai_runtime} ${_coreai_data_loader}
executorch::coreai_dependencies
)
executorch_target_link_options_shared_lib(coreaidelegate)
if(EXECUTORCH_BUILD_SHARED)
set_target_properties(
coreaidelegate PROPERTIES OUTPUT_NAME executorch_backend_coreai
)
executorch_target_soname_policy(coreaidelegate)
executorch_target_shipped_runtime_path(coreaidelegate)
endif()

# A consumer is necessary to check Swift linkage and registration retention.
add_executable(coreai_runtime_smoke runtime/test/runtime_smoke.cpp)
if(EXECUTORCH_BUILD_TESTS)
add_test(NAME coreai_runtime_smoke COMMAND coreai_runtime_smoke)
else()
set_target_properties(coreai_runtime_smoke PROPERTIES EXCLUDE_FROM_ALL TRUE)
endif()
coreai_require_os27(coreai_runtime_smoke)
set_target_properties(coreai_runtime_smoke PROPERTIES LINKER_LANGUAGE CXX)
target_link_libraries(
coreai_runtime_smoke PRIVATE coreaidelegate ${_coreai_runtime}
)

install(
TARGETS coreai_swift
TARGETS coreaidelegate coreai_swift
EXPORT ExecuTorchTargets
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
Expand Down
107 changes: 107 additions & 0 deletions backends/apple/coreai/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,34 @@ iteration order is not a binding contract. The `files` object maps relative
filenames to byte sizes; `bundle_digests` maps each bundle basename to its
export-time SHA-256 digest.

The experimental `CoreAIBackend` runtime supports stateless FP32/FP16 tensor
models in both formats.
ExecuTorch tensors must be contiguous. Output shapes
must fit the executor's declared resize and capacity constraints. Stateful
functions, image values, other dtypes, and interleaved layouts are not supported.
Calls on one session must be serialized by the caller.

Backend `init` and `execute` block the calling thread until Core AI finishes,
and that work runs on Swift's shared concurrency pool. Do not load or execute
from Swift concurrency code (an `async` function or `Task`), because blocking
those threads can starve the pool; call from a dedicated thread or dispatch
queue instead.

Supported inputs are borrowed directly from ExecuTorch storage through Core AI
raw views. The bridge copies shape metadata, not tensor bytes, and does not
allocate intermediate input `NSData`, `Data`, or `NDArray` buffers. Each call
wraps `const_data_ptr()` and the current shape's `nbytes()`, not the allocation's
maximum capacity. Outputs use the SDK's actual returned shape; the backend
resizes them within ExecuTorch's declared bounds before copying only the logical
result bytes. Unused capacity is neither wrapped as input nor written as output.
The backend waits for inference and all input access to finish before returning
from `execute`, including on errors. This uses ExecuTorch's normal input-lifetime
contract; callers must not concurrently mutate, release, or rebind the input
storage. Output data remains owned and is copied into validated ExecuTorch
outputs after inference, so output/input aliases are not written during the
borrow. This does not guarantee that Core AI itself avoids device transfers or
internal copies.

### AOT architecture selection

Configure AOT export with `AOTCompileConfig`, including the target `platform` and
Expand All @@ -42,13 +70,26 @@ architectures=["h17p"])` requests that compiler architecture; omitting the list
lets the compiler emit its supported architectures for the target platform.
These are Core AI architecture names, not CPU names such as `arm64`.

At load time, the runtime queries `AIModel.deviceArchitectureName` and selects
the exact matching entry in the manifest's `archs` map. No runtime architecture
option is required, and there is no fallback to another architecture. Platform,
deployment floor, and architecture checks happen before asset reads or storage
creation. Missing-architecture errors identify the device architecture, list
available architectures, and request a matching export. Missing selected files
are reported separately from an absent architecture entry.

Both formats can require device specialization. Every load first attempts SDK
bookmark restoration. A missing bookmark or SDK-confirmed cache miss falls back
to source materialization and persistent specialization, then binds the requested
function and ordered I/O on that same acquired model.

### Asset storage

Backend initialization reads the string runtime spec `coreai_assets_dir`. If
absent, it sets the directory to `executorch_coreai` beneath user-domain
`NSCachesDirectory`. The coordinator and storage helpers always receive that
explicit directory. It contains raw bookmarks, staged bundles and lock files.

The assets root must be an absolute path. The application chooses it, should
reserve it for this backend, and is trusted not to rename, replace or modify its
contents while the backend uses it; the backend does not defend against other
Expand Down Expand Up @@ -83,6 +124,17 @@ Preparing the assets root sets `NSURLIsExcludedFromBackupKey` on it, which cover
everything beneath it. Ancestors are not modified. This is backup exclusion, not
a control for iCloud Drive synchronization.

Delegate construction and registration do not create storage directories or load
models. Per-model initialization performs SDK acquisition and function loading,
validating source assets only when recovery requires them. There is no synthetic
zero-input inference at initialization. SDK specialization and function resource
loading can still be substantial.

The runtime always uses `AIModelCache.default` with persistent retention and
stores an opaque bookmark for direct restoration. `coreai_assets_dir` does not
choose the SDK artifact directory. The backend does not inspect SDK-private
files, alter their backup flags, or use an App Group cache.

Each partition derives a key from its selected export digest/bundle, platform,
Core AI device architecture, default SDK cache namespace, versioned default
options and persistent policy. Function bindings and delivery location are
Expand All @@ -108,6 +160,36 @@ acquiring a lock does not flush its file or directory. Lock files are not
removed by the backend, including after process exit. Removing the assets root
externally requires all loads, sessions and maintenance to stop.

Automatic source removal is disabled. A durable bookmark and a live model pin
do not prove source independence. The backend does not remove sources on method
unload or delegate destruction; no runtime option enables automatic removal.
Cache storage and SDK persistent entries may be purged, so retain or redeliver
the original PTE and named data for reconstruction after a cache miss.
Source-free inference, fresh-process SDK restoration and persistent-policy
behavior require separate OS 27 hardware qualification.

### Runtime options

Pass options through the public `Module` API before loading. In an error-returning
loader using the `executorch::runtime` namespace:

```cpp
BackendOptions<1> options;
ET_CHECK_OK_OR_RETURN_ERROR(options.set_option("coreai_assets_dir", assets_dir));
LoadBackendOptionsMap map;
ET_CHECK_OK_OR_RETURN_ERROR(map.set_options("CoreAIBackend", options.view()));
ET_CHECK_OK_OR_RETURN_ERROR(module.load(map));
```

No options are required when using the default assets root. `coreai_assets_dir`
selects an application-chosen cache directory reserved for this backend. The
supplied path is validated during preflight, but warm bookmark hits need no
source.

Use separate option maps to choose different storage roots for different models.
The path is a root directory, not the bundle itself. ExecuTorch option strings
have a 255-byte limit.

## SDK Bridge

The private Swift module `CoreAIBridge` (`runtime/ETCoreAIModel.swift`) wraps
Expand Down Expand Up @@ -140,6 +222,25 @@ Ninja builds use one architecture per build directory. Current runtime support
covers arm64 macOS and iOS device builds. x86_64 builds are blocked by Swift
`Float16` availability, and the tested iOS simulator SDKs do not contain Core AI.

Enable `EXECUTORCH_BUILD_EXTENSION_DATA_LOADER` for inference consumers.
Link the CMake target `coreaidelegate`, not just its archive filename. Its
transitive dependencies supply Swift and Foundation/CoreAI linkage, and its
link interface retains static backend registration. With
`EXECUTORCH_BUILD_SHARED=ON`, the delegate is a shared library linked to the
single consolidated `executorch_shared` runtime. Otherwise it is static.
Consumers of the shared delegate must use that same shared runtime.
Installed CMake targets resolve framework and Swift runtime dependencies using
the consumer's selected SDK and toolchain, not the producer's Xcode paths.
Select the Apple SDK/toolchain when configuring the consumer project.
SwiftPM and XCFramework distribution are not integrated yet.

The `coreai_runtime_smoke` target checks final linkage of a C++ consumer,
including Swift dependencies and backend registration retention. With
`EXECUTORCH_BUILD_TESTS=ON` it is built and
registered with CTest; otherwise it is excluded from the default build. On OS 27,
running it checks registration and availability; it does not perform model
inference. Do not execute SDK27 binaries on an older host.

## Host Tests

`EXECUTORCH_BUILD_COREAI=ON` with `EXECUTORCH_BUILD_TESTS=ON` registers the
Expand All @@ -159,3 +260,9 @@ cmake -S . -B <build-dir> -G Ninja \
cmake --build <build-dir> --target coreai_host_test
ctest --test-dir <build-dir> -R '^coreai_host_test$' --output-on-failure
```

For a compile-only production smoke build, use a fresh build directory without
`EXECUTORCH_BUILD_TESTS`, set `CMAKE_OSX_ARCHITECTURES=arm64`, and build
`coreai_runtime_smoke`. For iOS also set `CMAKE_SYSTEM_NAME=iOS` and
`CMAKE_OSX_SYSROOT=iphoneos`. Do not run the iOS binary on the host or install
a simulator as part of this build.
Loading
Loading