Qualcomm AI Engine Direct - QNN ExecuTorch Intermediate Output Debugger (pytorch#15735)

### Summary

- Enabled ExecuTorch QNN Intermediate Tensor Debugger.
- Provide an API for users to define their own metrics
- Offers a variety of output format to visualize the debug results: svg,
csv, raw files.
- A README file and tutorial script to guide users on how to debug a
model. Example script: `python
examples/qualcomm/util_scripts/qnn_intermediate_debugger_demo.py -b
build-android -m SM8550 --device $DEVICE --dataset ../imagenet-mini/val/
--dump_intermediate_outputs`

#### An example use case
MobileVit V2 has significant drop in accuracy in certain QNN versions,
while QNN 2.29 has good accuracy. With the help of accuracy debugger, we
have targeted the node native_group_norm_default_1 in the model. As
shown below, in QNN 2.29, this node has a cos_similarity (QNN V.S. CPU)
of 0.997, while all other QNN versions has cos_similarity of 0, which
provides us some hint it is possibly this group_norm node that is
causing accuracy drop.
<img width="1385" height="596" alt="image"
src="https://github.com/user-attachments/assets/ba81bb5f-1cae-4d3b-a945-a00ca92efeef"
/>


#### What's Coming Next?
- Currently, we dump CPU outputs by manually inserting observer nodes.
However, ExecuTorch actually has built in methods
(intermediate_output_capturer) that could dump intermediate output for
us, in format of a dict{debug_handle : tensor_output}. We will enable
`debug_handle` and reuse
https://github.com/pytorch/executorch/blob/main/devtools/inspector/_intermediate_output_capturer.py
in future instead.
- Support graph with partitions
- Support LLM models

### Test plan
- E2E example script test
- `python backends/qualcomm/tests/test_qnn_delegate.py -k
TestExampleUtilsScript.test_intermediate_debugger -s $DEVICE --model
SM8650 --build_folder build-android/ --executorch_root . --image_dataset
../imagenet-mini/val/ --artifact ./e2e_test_debug`
- Simple model test
- `python backends/qualcomm/tests/test_qnn_delegate.py -k
TestQNNQuantizedUtils.test_qnn_backend_dump_intermediate_outputs_simple_model
--model SM8550 --device $DEVICE --build_folder build-android`
- `python backends/qualcomm/tests/test_qnn_delegate.py -k
TestQNNQuantizedUtils.test_qnn_backend_dump_intermediate_outputs_topk
--model SM8550 --device $DEVICE --build_folder build-android`

Author: winskuo-quic

backends/qualcomm/README.md

@@ -133,7 +133,7 @@ PRs are always welcome to help improve the codebase in a comprehensive manner. B
 - **Code Reviews**:<br/>
     Please ping authors in Qualcomm AI Engine Direct related PRs for reviewing, possible candidates are listed below:
     - [shewu-quic](https://github.com/shewu-quic)
-    - [chunit-quic](https://github.com/chunit-quic)
+    - [chenweng-quic](https://github.com/chenweng-quic)
     - [winskuo-quic](https://github.com/winskuo-quic)
     - [DannyYuyang-quic](https://github.com/DannyYuyang-quic)
     - [haowhsu-quic](https://github.com/haowhsu-quic)

backends/qualcomm/builders/README.md

@@ -225,7 +225,7 @@ Now, we can start to fill in function body step by step:
     - **tensor_source_node**: current graph source node of the tensor
     - **target_build_node**: current node to build, which is important for fixed point mixed-precision to work properly
     - **tensor**: torch tensor emitted by node
-    - **tensor_type**: type compatible with QNN SDK, oftenly use `QNN_TENSOR_TYPE_NATIVE` for intermediate outputs and `QNN_TENSOR_TYPE_STATIC` for constant parameters
+    - **tensor_type**: type compatible with QNN SDK, often use `QNN_TENSOR_TYPE_NATIVE` for intermediate outputs and `QNN_TENSOR_TYPE_STATIC` for constant parameters
     - **nodes_to_wrappers**: dictionary of graph node and its output tensor (note: the tensor here is not a torch tensor but a wrapped object for QNN)
     - **node_name**: (optional) tensor name for user to specify
     - **wrapper_idx**: (optional) defaults to zero if node is not a tuple, otherwise it acts as an indexer to output tensors. e.g. when slicing input tensor into multiple outputs, `wrapper_idx` is necessary for getting correct wrapped tensor object
@@ -280,7 +280,7 @@ Now, we can start to fill in function body step by step:
             nodes_to_wrappers,
         )
     ```
-    Althought the input / output activations might map to the graph IOs (a.k.a. user inputs / outputs) with corresponding type   `QNN_TENSOR_TYPE_APP_READ` / `QNN_TENSOR_TYPE_APP_WRITE`. Users are still expected to have `QNN_TENSOR_TYPE_NATIVE` for all nodes' IOs and leave the  detection logic handled inside `define_tensor` method.
+    Although the input / output activations might map to the graph IOs (a.k.a. user inputs / outputs) with corresponding type   `QNN_TENSOR_TYPE_APP_READ` / `QNN_TENSOR_TYPE_APP_WRITE`. Users are still expected to have `QNN_TENSOR_TYPE_NATIVE` for all nodes' IOs and leave the  detection logic handled inside `define_tensor` method.
 
 5. Generate operator object in QNN graph:
     ```python
@@ -330,7 +330,7 @@ Now, we can start to fill in function body step by step:
     - **data_type**: type compatible with QNN SDK, e.g. `QNN_DATATYPE_FLOAT_32`, `QNN_DATATYPE_UINT_32`, etc.
     - **rank**: dimensions of tensor
     - **dims**: shape of tensor
-    - **data**: tesnor data
+    - **data**: tensor data
     - **copy_data**: user should specify to True for constant parameters
 
 8. Last, return operator object for partitioner to conduct validation:

backends/qualcomm/builders/node_visitor.py

@@ -30,6 +30,7 @@
     QCOM_SCALE,
     QCOM_SCALE_OFFSET,
     QCOM_SCALES,
+    QCOM_TENSOR_NAME,
     QCOM_ZERO_POINT,
     QCOM_ZERO_POINTS,
 )
@@ -395,6 +396,13 @@ def get_tensor_name(
             tensor_name = f"output_mutbuf_{position_index}_{tensor_name}"
         elif is_graph_output(node):
             tensor_name = f"output_{tensor_name}"
+
+        # Save this for intermediate debugger
+        # Needs idx since node like topk has 2 outputs
+        if QCOM_TENSOR_NAME in node.meta:
+            node.meta[QCOM_TENSOR_NAME][wrapper_idx] = tensor_name
+        else:
+            node.meta[QCOM_TENSOR_NAME] = {wrapper_idx: tensor_name}
         return tensor_name
 
     def define_custom_tensor_wrapper(

backends/qualcomm/debugger/README.md

@@ -18,7 +18,7 @@ python -m examples.qualcomm.util_scripts.qairt_visualizer_demo -H ${host} -s {de
 - If online prepare mode is `enabled`, the following artifacts will be generated:
     - `model`.dlc
     - `optrace`.json
-    - `QHAS`
+    - `QHAS`.json
 - If online prepare mode is `disabled`, the following artifacts will be generated:
     - `model`.bin
     - `optrace`.json
@@ -92,3 +92,160 @@ Note: Files ending with `.bin ` do not support graph visualization in qairt_visu
 </figure>
 
 For more details, visit the [QAIRT Visualizer](https://pypi.org/project/qairt-visualizer/).
+
+
+# ExecuTorch QNN Intermediate Output Debugger
+
+ExecuTorch QNN Intermediate Output Debugger is a tool that helps users debug intermediate output accuracy by comparing CPU outputs with QNN outputs. This tool offers a variety of output formats and flexibility for users to define their own metrics when debugging.
+
+Below, we will go through the details step by step on how to customize your own debugger. By the end of this tutorial, users should understand the mechanism behind the ExecuTorch QNN Debugger and how to apply the debugger to the desired model. In the rest of the tutorial, we will use the term `intermediate output` and `per-layer dump` interchangeably. 
+
+To make the implementation process smooth, we have also provided an example script, [qnn_intermediate_debugger_demo.py](../../../examples/qualcomm/util_scripts/qnn_intermediate_debugger_demo.py), which is an end-to-end example that goes through the steps for implementation. Refer to [Example Script](#example-script) section for more information.
+
+## Introduction
+
+1. Why do we need ExecuTorch QNN Intermediate Output Debugger?
+    During inference, there might be gaps between QNN and CPU final outputs. This leaves developers unsure about the root cause of accuracy drop. By using this debugger, users can gain better insight into which operation is causing the accuracy drop. Please note that the accuracy drop here refers to comparing QNN with CPU outputs, not the ground truth.
+    
+2. Who is this tool for?
+   This tool is mainly for developers aiming to align QNN with CPU accuracy. Users will be able to identify which layer in the model is causing the accuracy drop, helping them either circumvent the issue by replacing the layer with other operations or contact authors in Qualcomm AI Engine Direct to resolve the accuracy issue. Please refer to the last section under [README.md](../README.md) for authors to contact when encountering any issues.
+
+
+## Design Flow
+```mermaid
+flowchart TB;
+    nn.Module;
+    nn.Module --> edge_program["Edge Program"];
+    edge_program --> qnn_lower["QNN with Per-Layer Dump"];
+    qnn_lower --> qnn_inference[QNN Inference];
+    qnn_inference --> debug
+    edge_program --> cpu_lower["Edge CPU with Per-Layer Dump"];
+    cpu_lower --> cpu_inference["CPU Inference"];
+    cpu_inference --> debug["Debug"];
+    debug --> output["Output Results"]
+```
+
+## Instructions
+
+### 1. Setup
+1. Follow the [tutorial](https://pytorch.org/executorch/main/getting-started-setup) to set up ExecuTorch.
+2. Follow the [tutorial](https://pytorch.org/executorch/stable/build-run-qualcomm-ai-engine-direct-backend.html) to build Qualcomm AI Engine Direct Backend.
+
+### 2. Enable Flag
+
+When executing the script, please add the flag `--dump_intermediate_outputs`. This tells QNN to dump all intermediate tensors during execution.
+
+### 3. Add debugger to the example script
+Initialize a `QNNIntermediateDebugger`. Please pass initialized `QNNIntermediateDebugger` and the `args.dump_intermediate_outputs` to `build_executorch_binary` method as well.
+#### Example:
+```python
+from executorch.examples.qualcomm.utils import build_executorch_binary
+from executorch.backends.qualcomm.debugger.qnn_intermediate_debugger import QNNIntermediateDebugger
+
+qnn_intermediate_debugger = QNNIntermediateDebugger()
+build_executorch_binary(
+    model=MyModel(),
+    inputs=(torch.randn(200, 768),),
+    soc_model="SM8650",
+    file_name="my_model",
+    dataset=my_dataset,
+    dump_intermediate_outputs=args.dump_intermediate_outputs, # Add this flag
+    qnn_intermediate_debugger=qnn_intermediate_debugger, # Add this flag
+)
+```
+
+### 4. Set data num to 1
+It is perfectly fine for users to pass the desired amount of datasets to `build_executorch_binary`, which helps achieve better quantization results. However, after `build_executorch_binary` is called, we need to ensure that we only perform one inference during execution. Please ensure that CPU and QNN is using the same input during execution; otherwise, the debugging results might not be accurate.
+
+### 5. Pass flag to SimpleADB
+When creating `SimpleADB`, please also pass the flag `args.dump_intermediate_outputs`. This tells the runner to create files that store the intermediate output schema and binary data.
+#### Example:
+```python
+adb = SimpleADB(
+    qnn_sdk=os.getenv("QNN_SDK_ROOT"),
+    build_path=f"{args.build_folder}",
+    pte_path=f"{args.artifact}/{pte_filename}.pte",
+    workspace=f"/data/local/tmp/executorch/{pte_filename}",
+    device_id=args.device,
+    host_id=args.host,
+    soc_model=args.model,
+    shared_buffer=args.shared_buffer,
+    dump_intermediate_outputs=args.dump_intermediate_outputs, # Add this flag
+)
+```
+
+### 6: Pull and process the results.
+After QNN execution with the runner, if the previous steps are done correctly, we should be able to get two files: `etdump.etdp` and `debug_output.bin`.
+The following example pulls the files back and calls a callback function to process the results. In this callback function, we create the `Inspector`. Then we perform CPU inference to get CPU intermediate results. Now, we have both QNN and CPU intermediate results, we can start generating results to compare the accuracy. Taking the following example, we should be able to get `debug_graph.svg` as an output in the current directory.
+#### Example:
+```python
+from executorch.backends.qualcomm.debugger.qnn_intermediate_debugger import  OutputFormat
+def validate_intermediate_tensor():
+    inspector = Inspector(
+        etdump_path=f"{args.artifact}/etdump.etdp",
+        debug_buffer_path=f"{args.artifact}/debug_output.bin",
+    )
+    qnn_intermediate_debugger.intermediate_output_module(*(inputs[0]))
+    qnn_intermediate_debugger.generate_results(
+        title="debug_graph",
+        path=".",
+        output_format=OutputFormat.SVG_GRAPHS,
+        inspector=inspector,
+        evaluator=CosineSimilarityEvaluator(0.9),
+    )
+
+adb.pull_debug_output(
+    args.artifact, args.artifact, callback=validate_intermediate_tensor
+)
+```
+
+#### Additional Options
+The above example sets output formats as SVG and evaluation metrics using Cosine Similarity. Based on different needs, users can choose other output formats as shown in the `OutputFormat` class under [qnn_intermediate_debugger](./qnn_intermediate_debugger.py)
+```python
+class OutputFormat(IntEnum):
+    SVG_GRAPHS = 0
+    CSV_FILES = 1
+    DUMP_RAW = 2
+```
+
+For evaluation metrics, if users would like to implement their own metrics, we have provided the option to implement [MetricEvaluatorBase](./metrics_evaluator.py). The following shows how to define custom metrics.
+```python
+class RootMeanSquaredErrorEvaluator(MetricEvaluatorBase):
+    def __init__(self, threshold=0.02):
+        self.threshold = threshold
+
+    def metric_name(self) -> str:
+        return "Root Mean Squared Error"
+
+    def evaluate(
+        self, qnn_output: torch.Tensor, cpu_output: torch.Tensor
+    ) -> Tuple[Any, bool]:
+        mse = F.mse_loss(qnn_output, cpu_output)
+        rmse = torch.sqrt(mse)
+        valid = rmse < self.threshold
+        return rmse, valid
+
+qnn_intermediate_debugger.generate_results(
+    title="my_metric",
+    path=".",
+    output_format=OutputFormat.SVG_GRAPHS,
+    inspector=inspector,
+    evaluator=RootMeanSquaredErrorEvaluator(),
+)
+```
+
+### Example Script
+We have provided an inception_v3 demo script to help users better understand how to apply the debugger to their scripts. Please refer to [qnn_intermediate_debugger_demo.py](../../../examples/qualcomm/util_scripts/qnn_intermediate_debugger_demo.py) for the example script.
+
+Before running the example script, please ensure that dataset is downloaded. Example dataset can be retrieved [here](https://www.kaggle.com/datasets/ifigotin/imagenetmini-1000).
+
+To execute the model:
+```bash
+python examples/qualcomm/util_scripts/qnn_intermediate_debugger_demo.py -b build-android -m ${SOC_MODEL} --device ${SERIAL_NUM} --dataset ${PATH_TO_DATASET} --dump_intermediate_outputs
+```
+
+### Limitation
+1. The current debugger only supports performing one execution. Multiple executions may cause unknown behavior and are not recommended.
+2. Please ignore this if you are using `qnn_executor_runner`. If you have decided to write your own runner, please follow the [tutorial](https://pytorch.org/executorch/stable/etdump.html) on how to implement etdump into your own runner.
+3. The current debugger does not support graph with partitions. (WIP)
+4. The current debugger does not support LLM models. (WIP)

backends/qualcomm/debugger/TARGETS

@@ -10,3 +10,21 @@ runtime.python_library(
         "fbsource//third-party/pypi/pandas:pandas",
     ]
 )
+
+runtime.python_library(
+    name = "qnn_intermediate_debugger",
+    srcs = [
+        "format_outputs.py",
+        "metrics_evaluator.py",
+        "qnn_intermediate_debugger.py",
+    ],
+    deps = [
+        "//caffe2:torch",
+        "//executorch/backends/qualcomm/_passes:passes",
+        "//executorch/backends/qualcomm/utils:utils",
+        "//executorch/devtools:lib",
+        "//executorch/exir:sym_util",
+        "fbsource//third-party/pypi/graphviz:graphviz",
+        "fbsource//third-party/pypi/pandas:pandas",
+    ],
+)