Skip to content

Commit 558473d

Browse files
authored
Ship a C++ SDK in the wheel (pytorch#21639)
The wheel ships the runtime, kernels, delegate, thread pool and profiler as separate shared libraries, but nothing outside Python can use them, because the installed CMake package names none of them. A C++ application would have to hard-code paths into the wheel's private layout. The headers have the same gap. The wheel installs only the subset a custom-operator build needs, which leaves out `extension/module`, the entry point the documentation tells C++ callers to use. So the wheel ships the libraries to run a model and no way to call them. Name each shipped library as a CMake component, so `find_package` locates them, and ship the headers a caller needs. A component is just a name a consumer can ask for, and CMake reports a missing one while configuring rather than at link time. ```cmake find_package(executorch 1.5 REQUIRED COMPONENTS kernels_optimized) target_link_libraries(my_app PRIVATE executorch::runtime executorch::kernels_optimized) ``` | component | library it resolves to | | --- | --- | | `executorch::runtime` | `libexecutorch.so` | | `executorch::kernels_optimized` | `libexecutorch_kernels_optimized.so` | | `executorch::backend_xnnpack` | `libexecutorch_backend_xnnpack.so` | | `executorch::threadpool` | `libexecutorch_threadpool.so` | | `executorch::etdump` | `libexecutorch_etdump.so` | Each component records where the wheel keeps its libraries, so an application built against it finds them without the caller setting a library search path. The version file answers on CMake's SameMajorVersion rule: a request is accepted when it is no newer than the installed version and names the same major, and rejected otherwise, including a request one major below. ExecuTorch promises nothing across a major boundary, so answering a request written and tested against an older major would hand the consumer a package it has never seen and move the failure to compile or run time instead of to `find_package`. The other package generators in this tree already use that rule. Headers include the module and tensor entry points, the CPU kernel helpers, the allocator and data loader concrete classes Module's constructors take, the profiler entry points, and the FlatTensorDataMap and MergedDataMap types plus the .ptd file header a caller writing a .ptd needs. These headers are part of the published artifact, so the wheel workflows build on a change to one. Without that a header change reaches a nightly wheel with no wheel build having run on it. They are listed as whole trees rather than as the directories the copy list names, so growing that list cannot quietly leave a header outside the filter. Measured over the last 147 commits on main: the whole trees would newly build a wheel on four of them, the exact directories on one. CMake 3.28 or newer gets these targets. Older versions do not, because they write the `$ORIGIN` marker (the "look next to me" token in a library search path) incorrectly: ``` 3.24.3, 3.27.9 Makefiles double the dollar sign, Ninja drops the name 3.28.4, 3.31.8 both write the token correctly ``` That would produce a target that runs where it was built and fails once the application is copied elsewhere, so no target is defined below 3.28. Those versions get plain variables instead: `EXECUTORCH_LIBRARIES` with the runtime and every shipped library by path, plus `EXECUTORCH_INCLUDE_DIRS`, `EXECUTORCH_COMPILE_DEFINITIONS` and `EXECUTORCH_CXX_STANDARD`. All four are needed, because an imported target carries the definitions and the C++ standard along with the library and a plain path carries neither. Linking the libraries alone stops at `#error "You need C++17 to compile ExecuTorch"`. `ET_USE_THREADPOOL` is added to `EXECUTORCH_COMPILE_DEFINITIONS` on the pre-3.28 route when the thread pool library ships. Without it the runtime header supplies a local inline serial fallback for `parallel_for`, so a consumer following the documented recipe linked the thread pool library and still ran serial code with no diagnostic. Built the wheel, installed it into a clean environment, and built a C++ application against the installed wheel alone: - the application links the runtime, runs a model, and matches eager PyTorch, and still runs after being copied away from the wheel. - asking for a component the wheel does not ship fails while configuring, naming the component. - a version request is honoured, including ranges, and a request naming a different major is rejected in both directions. Measured with cmake 3.31.8 against fabricated 1.5.0 and 2.0.0 packages: 1.5.0 accepts no request at all, `1.0`, `1.5`, `1.5.0`, `1.0...<2.0` and `1.0...2.0`, and rejects `0.1` and `2.0`; 2.0.0 accepts `2.0` and rejects `1.5` and `2.1`. - shipped headers can be included on their own, and one entry point per shipped component also links against the shipped libraries. A small number are exempt because they need something outside the package: a Windows shim, a test framework, or a header that says in its own text not to include it directly. The exempt list is compiled too, so an entry that starts working is reported rather than left in place. - the thread pool probe compiles with `ET_USE_THREADPOOL`, on both the modern-CMake route (from the runtime target) and the pre-3.28 route (from `EXECUTORCH_COMPILE_DEFINITIONS`). Without it the header supplies a local inline definition and the probe linked identically whether or not the library was on the link line, so it could not detect the component being dropped. Measured both ways. - an application's runtime search path is recorded as `DT_RUNPATH`, not the older `DT_RPATH`. That matters because `DT_RPATH` is searched ahead of `LD_LIBRARY_PATH` and is inherited by dependencies, so a consumer could not point a locally built or instrumented runtime at their application. Verified by shadowing the runtime through `LD_LIBRARY_PATH` and watching the loader pick it up, which `DT_RPATH` ignores. - on real CMake 3.24 and 3.27, an application configures, builds and runs through the variables. Measured what each one contributes, with the consumer pinned to C++14 so its own standard does not hide the package's requirement: linking `EXECUTORCH_LIBRARIES` alone fails on a missing header, adding the include directories and definitions then fails on the C++ standard, and applying `EXECUTORCH_CXX_STANDARD` builds and loads a model. The kernels also need scoped retention there, because a registration-only library exports nothing the application references and the linker drops it, which showed up as "Missing operator" at run time rather than as a link error. The smoke test now runs the same shape automatically when `EXECUTORCH_PRE_328_CMAKE` points at an older cmake binary, so a future change on the fallback path fails a check rather than only showing up on the first user with older cmake. - `find_package` succeeds when the interpreter on PATH is not the one the wheel was built for. The extension's own file name carries its suffix, so asking a different interpreter for it reported a complete install as not found. Ran on Linux x86_64 and aarch64. The macOS wheel keeps the fused extension and ships no separate libraries, so these checks do not apply there and its smoke test does not run them.
1 parent 7d84f08 commit 558473d

17 files changed

Lines changed: 2452 additions & 65 deletions

‎.ci/scripts/wheel/test_cpp_sdk.py‎

Lines changed: 1255 additions & 0 deletions
Large diffs are not rendered by default.

‎.ci/scripts/wheel/test_linux.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
from pathlib import Path
1313

1414
import test_base
15+
import test_cpp_sdk
1516
import test_shared_libraries
1617
from examples.models import Backend, Model
1718

@@ -51,6 +52,13 @@
5152
with tempfile.TemporaryDirectory() as work_dir:
5253
test_shared_libraries.run_tests(Path(work_dir))
5354

55+
# And that a C++ application outside the wheel can actually use them.
56+
# Nothing above covers this: the Python extension links those libraries
57+
# itself, so it passes whether or not the package config names them or the
58+
# shipped headers are complete.
59+
with tempfile.TemporaryDirectory() as work_dir:
60+
test_cpp_sdk.run_tests(Path(work_dir))
61+
5462
model_tests = [
5563
test_base.ModelTest(
5664
model=Model.Mv3,

‎.ci/scripts/wheel/test_linux_aarch64.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
from pathlib import Path
1010

1111
import test_base
12+
import test_cpp_sdk
1213
import test_shared_libraries
1314
from examples.models import Backend, Model
1415

@@ -36,6 +37,11 @@
3637
with tempfile.TemporaryDirectory() as work_dir:
3738
test_shared_libraries.run_tests(Path(work_dir))
3839

40+
# And that a C++ application outside the wheel can actually use those
41+
# libraries, which nothing above covers.
42+
with tempfile.TemporaryDirectory() as work_dir:
43+
test_cpp_sdk.run_tests(Path(work_dir))
44+
3945
test_base.run_tests(
4046
model_tests=[
4147
test_base.ModelTest(

‎.github/workflows/build-wheels-aarch64-linux.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,11 @@ on:
1313
- docs/source/using-executorch-cpp.md
1414
- install_utils.py
1515
- tools/cmake/**/*
16+
# The wheel ships these as its C++ SDK. Whole trees rather than the exact directories
17+
# setup.py copies from, so adding one there cannot silently drop it from this list.
18+
- devtools/etdump/**.h
19+
- extension/**.h
20+
- runtime/**.h
1621
push:
1722
branches:
1823
- nightly

‎.github/workflows/build-wheels-linux.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,11 @@ on:
1313
- docs/source/using-executorch-cpp.md
1414
- install_utils.py
1515
- tools/cmake/**/*
16+
# The wheel ships these as its C++ SDK. Whole trees rather than the exact directories
17+
# setup.py copies from, so adding one there cannot silently drop it from this list.
18+
- devtools/etdump/**.h
19+
- extension/**.h
20+
- runtime/**.h
1621
push:
1722
branches:
1823
- nightly

‎.github/workflows/build-wheels-macos.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,11 @@ on:
1313
- docs/source/using-executorch-cpp.md
1414
- install_utils.py
1515
- tools/cmake/**/*
16+
# The wheel ships these as its C++ SDK. Whole trees rather than the exact directories
17+
# setup.py copies from, so adding one there cannot silently drop it from this list.
18+
- devtools/etdump/**.h
19+
- extension/**.h
20+
- runtime/**.h
1621
push:
1722
branches:
1823
- nightly

‎.github/workflows/build-wheels-windows.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,11 @@ on:
1212
- docs/source/using-executorch-cpp.md
1313
- install_utils.py
1414
- tools/cmake/**/*
15+
# The wheel ships these as its C++ SDK. Whole trees rather than the exact directories
16+
# setup.py copies from, so adding one there cannot silently drop it from this list.
17+
- devtools/etdump/**.h
18+
- extension/**.h
19+
- runtime/**.h
1520
push:
1621
branches:
1722
- nightly

‎docs/source/getting-started.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,23 @@ For more information on iOS integration, including an API reference, logging set
171171
ExecuTorch provides C++ APIs, which can be used to target embedded or mobile devices. The C++ APIs provide a greater level of control compared to other language bindings, allowing for advanced memory management, data loading, and platform integration.
172172

173173
#### Installation
174+
On Linux and macOS the quickest route is the pip package, which ships the runtime as prebuilt
175+
libraries with headers and a CMake package, so there is nothing to build:
176+
177+
```
178+
pip install executorch
179+
```
180+
181+
```cmake
182+
find_package(executorch REQUIRED COMPONENTS kernels_optimized)
183+
target_link_libraries(my_target PRIVATE executorch::runtime executorch::kernels_optimized)
184+
```
185+
186+
See [Using the prebuilt libraries from the pip package](using-executorch-cpp.md#using-the-prebuilt-libraries-from-the-pip-package)
187+
for a complete walkthrough, and [Running on a GPU with the CUDA package](using-executorch-cpp.md#running-on-a-gpu-with-the-cuda-package)
188+
for GPU support.
189+
190+
For a platform the package does not cover, or to change build options, build from source instead.
174191
CMake is the preferred build system for the ExecuTorch C++ runtime. To use with CMake, clone the ExecuTorch repository as a subdirectory of your project, and use CMake's `add_subdirectory("executorch")` to include the dependency. The `executorch` target, as well as kernel and backend targets will be made available to link against. The runtime can also be built standalone to support diverse toolchains. See [Using ExecuTorch with C++](using-executorch-cpp.md) and [Building from Source](using-executorch-building-from-source.md) for a detailed description of build integration, targets, and cross compilation.
175192

176193
```

‎docs/source/using-executorch-building-from-source.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,12 @@
11
# Building from Source
22

3+
On Linux and macOS you may not need to build at all. `pip install executorch` ships the
4+
runtime as prebuilt libraries with headers and a CMake package, so a C++ program can link
5+
it directly. See [Using the prebuilt libraries from the pip package](using-executorch-cpp.md#using-the-prebuilt-libraries-from-the-pip-package),
6+
including the CUDA packages for running on a GPU. Build from source when you need a
7+
platform the package does not cover, a build option it does not enable, or your own
8+
changes to the runtime.
9+
310
ExecuTorch uses [CMake](https://cmake.org/) as the primary build system.
411
Even if you don't use CMake directly, CMake can emit scripts for other format
512
like Make, Ninja or Xcode. For information, see [cmake-generators(7)](https://cmake.org/cmake/help/latest/manual/cmake-generators.7.html).

‎docs/source/using-executorch-cpp.md‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,213 @@ Running a model using the low-level runtime APIs allows for a high-degree of con
4040
4141
## Building with CMake
4242
43+
There are two ways to get the C++ runtime. Linking the prebuilt libraries from the pip
44+
package needs no source checkout and is the quicker option. Building from source gives
45+
you every option the project has, and is what you need for a platform the wheel does not
46+
cover.
47+
48+
### Using the prebuilt libraries from the pip package
49+
50+
On Linux, `pip install executorch` ships the runtime as prebuilt shared libraries together with the
51+
headers and a CMake package. So a C++ program can use ExecuTorch without building it from source,
52+
and without knowing much CMake.
53+
54+
#### Run your first model in four steps
55+
56+
Copy these three files into an empty folder and follow along. No prior CMake knowledge needed.
57+
58+
**1. Install, and make a model file.**
59+
60+
```
61+
pip install executorch torch --extra-index-url https://download.pytorch.org/whl/cpu
62+
```
63+
64+
`torch` is named explicitly because the wheel does not depend on it, so you bring your own torch and
65+
keep the version under your control. You need it only to create a model file in step 1, not to run
66+
the C++ program.
67+
68+
A C++ program loads a `.pte` file, which is a model that has already been exported. C++ cannot
69+
create one, so make it in Python first:
70+
71+
```python
72+
# export.py
73+
import torch
74+
from executorch.exir import to_edge_transform_and_lower
75+
76+
class Add(torch.nn.Module):
77+
def forward(self, x, y):
78+
return x + y
79+
80+
example = (torch.ones(2, 2), torch.ones(2, 2))
81+
program = to_edge_transform_and_lower(
82+
torch.export.export(Add(), example)
83+
).to_executorch()
84+
open("model.pte", "wb").write(program.buffer)
85+
```
86+
87+
```
88+
python export.py
89+
```
90+
91+
**2. Write the program.**
92+
93+
```cpp
94+
// main.cpp
95+
#include <executorch/extension/module/module.h>
96+
#include <executorch/extension/tensor/tensor.h>
97+
#include <cstdio>
98+
99+
using namespace executorch::extension;
100+
101+
int main() {
102+
Module module("model.pte");
103+
104+
std::array<float, 4> a{1, 2, 3, 4};
105+
std::array<float, 4> b{10, 20, 30, 40};
106+
107+
const auto result = module.forward({make_tensor_ptr({2, 2}, a.data()),
108+
make_tensor_ptr({2, 2}, b.data())});
109+
if (!result.ok()) {
110+
std::printf("forward failed: 0x%x\n", (unsigned)result.error());
111+
return 1;
112+
}
113+
114+
const auto out = result->at(0).toTensor();
115+
for (int i = 0; i < out.numel(); ++i) {
116+
std::printf("%g ", out.const_data_ptr<float>()[i]);
117+
}
118+
std::printf("\n");
119+
return 0;
120+
}
121+
```
122+
123+
**3. Write six lines of CMake.**
124+
125+
```cmake
126+
# CMakeLists.txt
127+
cmake_minimum_required(VERSION 3.28)
128+
project(app CXX)
129+
130+
find_package(executorch REQUIRED COMPONENTS kernels_optimized)
131+
132+
add_executable(app main.cpp)
133+
target_link_libraries(app PRIVATE executorch::runtime
134+
executorch::kernels_optimized)
135+
```
136+
137+
Two lines matter. `find_package` finds the installed ExecuTorch, and `target_link_libraries` says
138+
which parts you want. Every model needs at least these two: `runtime` is the engine that executes a
139+
program, and a kernel component such as `kernels_optimized` provides the maths the model computes
140+
with. With only the engine, a model loads and then fails with a missing operator.
141+
142+
**4. Build and run.**
143+
144+
```
145+
cmake -S . -B build \
146+
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch, pathlib; print(pathlib.Path(executorch.__path__[0]) / "share" / "cmake")')"
147+
cmake --build build
148+
./build/app
149+
```
150+
151+
```
152+
11 22 33 44
153+
```
154+
155+
That is the two input arrays added together. The long `python -c` part just prints where pip put the
156+
CMake package, so CMake can find it. Run `./build/app` from the folder holding `model.pte`, because
157+
the path in `main.cpp` is relative.
158+
159+
#### Adding kernels and backends
160+
161+
Add a component to both lines to get more. Nothing else in the program changes.
162+
163+
```cmake
164+
find_package(executorch REQUIRED COMPONENTS kernels_optimized backend_xnnpack)
165+
166+
target_link_libraries(app PRIVATE executorch::runtime
167+
executorch::kernels_optimized
168+
executorch::backend_xnnpack)
169+
```
170+
171+
These are the components the Linux package provides:
172+
173+
| Component | What it gives you | Where |
174+
| --- | --- | --- |
175+
| `runtime` | The engine. Always needed. | Linux |
176+
| `kernels_optimized` | Fast CPU operators. The usual choice. | Linux |
177+
| `backend_xnnpack` | The XNNPACK backend, for models exported with it. | Linux |
178+
| `threadpool` | Multi-threaded execution. | Linux |
179+
| `etdump` | Profiling, to record what ran and how long it took. | Linux |
180+
181+
To see what your own install offers, ask CMake:
182+
183+
```cmake
184+
find_package(executorch REQUIRED)
185+
foreach(_component runtime kernels_optimized backend_xnnpack threadpool etdump)
186+
if(TARGET executorch::${_component})
187+
message(STATUS "have ${_component}")
188+
endif()
189+
endforeach()
190+
```
191+
192+
The macOS wheel keeps everything inside the Python extension rather than shipping separate C++
193+
libraries, so there is nothing for a C++ application to link there; use it from Python, or build
194+
from source if you need C++ on macOS.
195+
196+
A backend is only needed if the model was exported for it. Linking XNNPACK does not make a plain
197+
model faster, and a model exported for XNNPACK will fail to load without it. If you are not sure
198+
what a model needs, start with `runtime` and `kernels_optimized` and add what the error asks for.
199+
200+
If you would rather not choose, one variable links the common set:
201+
202+
```cmake
203+
find_package(executorch REQUIRED)
204+
target_link_libraries(app PRIVATE ${EXECUTORCH_LIBRARIES})
205+
```
206+
207+
#### When something does not work
208+
209+
- `find_package` could not find executorch: the `-DCMAKE_PREFIX_PATH=...` argument is missing or
210+
points somewhere else. Run the `python -c` line on its own and check the folder exists.
211+
- The program builds but fails to load the model: the path is relative, so run it from the folder
212+
containing the `.pte` file.
213+
- A missing operator at run time: add a kernel component, usually
214+
`executorch::kernels_optimized`.
215+
- The model fails to load complaining about a backend: link the backend it was exported for.
216+
- `executorch::runtime` is not a target: imported targets need CMake 3.28 or newer. On an older
217+
CMake the package still works, but you name variables instead of targets, and you have to pass on
218+
the definitions and the C++ standard requirement yourself:
219+
220+
```cmake
221+
cmake_minimum_required(VERSION 3.19)
222+
project(app CXX)
223+
224+
find_package(executorch REQUIRED)
225+
226+
add_executable(app main.cpp)
227+
target_include_directories(app PRIVATE ${EXECUTORCH_INCLUDE_DIRS})
228+
target_compile_definitions(app PRIVATE ${EXECUTORCH_COMPILE_DEFINITIONS})
229+
target_compile_features(app PRIVATE cxx_std_${EXECUTORCH_CXX_STANDARD})
230+
target_link_libraries(app PRIVATE ${EXECUTORCH_LIBRARIES})
231+
set_target_properties(
232+
app PROPERTIES INSTALL_RPATH "${EXECUTORCH_RUNTIME_LIBRARY_DIR}"
233+
)
234+
```
235+
236+
Leaving out `EXECUTORCH_COMPILE_DEFINITIONS` fails with a missing
237+
`torch/headeronly/macros/cmake_macros.h`, because the vendored headers look for a file that only
238+
exists inside a PyTorch build.
239+
240+
`INSTALL_RPATH` matters once you run `cmake --install`. CMake gives your program a search path while
241+
it sits in the build directory and removes that path when installing, so an installed program cannot
242+
find the libraries unless you record where they live.
243+
244+
You should not need `LD_LIBRARY_PATH`. The shipped libraries record where their neighbours live, so
245+
they find each other once the program links against the installed package.
246+
247+
### Building from source
248+
249+
43250
ExecuTorch uses CMake as the primary build system. Inclusion of the module and tensor APIs are controlled by the `EXECUTORCH_BUILD_EXTENSION_MODULE` and `EXECUTORCH_BUILD_EXTENSION_TENSOR` CMake options. As these APIs may not be supported on embedded systems, they are disabled by default when building from source. The low-level API surface is always included. To link, add the `executorch` target as a CMake dependency, along with `executorch_backends`, `executorch_extensions`, and `extension_kernels`, to link all configured backends, extensions, and kernels.
44251

45252
```

0 commit comments

Comments
 (0)