diff --git a/docs/contribute/development/cpp/code_analysis.rst b/docs/contribute/development/cpp/code_analysis.rst index 0a9e879a137..164ff096b46 100644 --- a/docs/contribute/development/cpp/code_analysis.rst +++ b/docs/contribute/development/cpp/code_analysis.rst @@ -102,13 +102,22 @@ Adress/ Leak Sanitizer (ASAN/LSAN) If both tools are combined at runtime memory leaks and the corresponding address can be investigated. -Coverage -======== +Code Coverage +============= -As required by the verification guideline coverage needs to be calculated for the code which is used in the project. Therefore two approaches should be available: +As required by the verification guideline code coverage needs to be calculated for the code which is used in the project. Coverage is calculated on the host using LLVM's source-based coverage: -* As a quick solution it is possible to calculate the coverage on the host via gcc. -* But for a more accurate statement coverage can also be calculated with the qcc compiler with the appropriate libraries and POSIX interfaces. This method will also be used for the reporting. +* Coverage is calculated on the host via clang/llvm. This method is also used for the reporting. + +In Bazel-based development, this does not imply that every build uses the same compiler configuration. Normal host builds may use the default host compiler/toolchain, while coverage builds select a dedicated host configuration with LLVM source-based instrumentation enabled. The resulting raw profiles are merged with ``llvm-profdata`` and evaluated with ``llvm-cov``. + +S-CORE determines structural coverage on the host using LLVM's source-based coverage. Structural coverage on the target is not determined by S-CORE. It shall be determined by the user or distributor for the target on which the S-CORE software is integrated. Target structural coverage is needed to identify uncovered target-specific code and to provide evidence for the absence of undefined behaviour on the target. + +LLVM's source-based coverage is preferred over ``gcov``-based coverage for the following reasons: + + +* **Precision with templates, generics and modern C++:** legacy GCC line-based coverage ``gcov`` is line-oriented, so with heavy templating, inlining and macros the mapping is coarse and multiple template instantiations collapse onto the same lines, producing imprecise or misleading results. LLVM's source-based coverage is region- and instantiation-based and therefore significantly more accurate for modern C++. +* **MC/DC support:** LLVM/clang supports MC/DC natively (``-fcoverage-mcdc``), which is required for the higher ASIL levels. To enable this, following tools are used: @@ -116,11 +125,11 @@ To enable this, following tools are used: object "Coverage" as coverage object "gtest" as gtest - object "gcov + gcovr" as gcov + object "llvm-cov + llvm-profdata" as llvm object "host" as host - object "QNX" as qnx coverage --> gtest - gtest --> gcov - gcov --> host - gcov --> qnx + gtest --> llvm + llvm --> host + +Host and target coverage have different responsibilities. LLVM coverage on the host provides S-CORE's structural-coverage result. The user or distributor shall determine structural coverage on the target used for integration, including target-specific code paths, as required by the platform AoU :need:`aou_req__platform__target_structural_coverage`. Target execution tests and target structural-coverage results are separate verification evidence and shall not be treated as interchangeable. diff --git a/docs/design_decisions/DR-001-infra.md b/docs/design_decisions/DR-001-infra.md index 60b4f827d02..64784a617c5 100644 --- a/docs/design_decisions/DR-001-infra.md +++ b/docs/design_decisions/DR-001-infra.md @@ -167,7 +167,7 @@ _Provided via devcontainer_ - `buildifier`, `bazel-compile-commands` - `curl`, `qemu-system-aarch64`, `sshpass` (??) - `protoc` -- `gcovr` +- `llvm-cov`, `llvm-profdata` **Not Natively Cacheable in Bazel** _Provided via devcontainer_ diff --git a/docs/platform_management_plan/software_verification.rst b/docs/platform_management_plan/software_verification.rst index a887ac8d669..656cee7af48 100644 --- a/docs/platform_management_plan/software_verification.rst +++ b/docs/platform_management_plan/software_verification.rst @@ -547,10 +547,10 @@ components, documentation, and automated tests. The software components of the project written in C++ are unit tested with the help of `GoogleTest `__. -.. rubric:: gcov/gcovr +.. rubric:: Code Coverage (llvm-cov/llvm-profdata) -For C++ code the structural coverage reached by unit testing in the project is evaluated by the gcov/gcovr tool chain -`gcovr `__ - gcov is part of the GNU compiler collection (gcc). +For C++ code the structural coverage reached by unit testing in the project is evaluated by the llvm-cov/llvm-profdata tool chain +`llvm-cov `__ and `llvm-profdata `__ are part of the LLVM toolchain. The C++ unit test tooling supports several coverage metrics: @@ -558,7 +558,7 @@ The C++ unit test tooling supports several coverage metrics: - "branch" - used in S-CORE for the ``structural-branch-coverage`` method - "function" - used in S-CORE for the ``structural-function-coverage`` as additional supporting coverage value for further analysis. -Note gcov/gcovr are not applicable for Rust code. Here coverage values are created using the tooling provided by the Ferrocene. +Note: Rust coverage evidence in S-CORE is primarily generated via Rust/Ferrocene tooling. llvm-cov/llvm-profdata may be used in dedicated host-based LLVM workflows, but this is not the default Rust coverage reporting path. .. rubric:: Integration Testing Framework (ITF) diff --git a/docs/requirements/platform_assumptions/index.rst b/docs/requirements/platform_assumptions/index.rst index 8805571e99c..b178dee49ad 100644 --- a/docs/requirements/platform_assumptions/index.rst +++ b/docs/requirements/platform_assumptions/index.rst @@ -256,6 +256,19 @@ This is the highest level of integration. This is the level where the S-CORE SW- Note: The SW-platform integtion tests provided by S-CORE for :need:`aou_req__platform__testing` are for demonstration purpose only, as described in the tailoring section of :need:`doc__score_platform_safety_plan` and thus do not claim correctness and completeness. +.. aou_req:: Target structural coverage + :id: aou_req__platform__target_structural_coverage + :reqtype: Non-Functional + :security: YES + :safety: ASIL_B + :status: valid + :version: 1 + :tags: user + + If the system using the SW-platform has safety goals, the system integrator or distributor shall determine structural coverage for the target on which the SW-platform is integrated. + + Note: S-CORE determines structural coverage on the host using LLVM's source-based coverage. Target structural coverage is required to cover target-specific code and undefined behaviour and is not determined by S-CORE. + .. aou_req:: Integrator safety anomaly reporting :id: aou_req__platform__integration_safety_anomaly :reqtype: Non-Functional diff --git a/docs/score_tools/score_tools_evaluation_list.rst b/docs/score_tools/score_tools_evaluation_list.rst index 59ceebb966e..dc36a65615b 100644 --- a/docs/score_tools/score_tools_evaluation_list.rst +++ b/docs/score_tools/score_tools_evaluation_list.rst @@ -186,6 +186,14 @@ release, as part of the Tool Verification Report. - :need:`rl__infrastructure_tooling_community` - N/A - YES + * - 2a-6 + - Clang + llvm-cov + llvm-profdata + - Code coverage tool from LLVM, used with Clang compiler + - 19.x (see [1]_) + - YES + - :need:`rl__infrastructure_tooling_community` + - :need:`doc_tool__clang` + - YES 2b Rust @@ -227,6 +235,14 @@ release, as part of the Tool Verification Report. - :need:`rl__infrastructure_tooling_community` - N/A - YES + * - 2b-4 + - llvm-cov + llvm-profdata + - LLVM source-based code coverage toolchain, usable in dedicated Rust/LLVM host workflows + - 19.x (see [1]_) + - YES + - :need:`rl__infrastructure_tooling_community` + - :need:`doc_tool__clang` + - YES 2c Python @@ -547,6 +563,14 @@ release, as part of the Tool Verification Report. - :need:`rl__infrastructure_tooling_community` - :need:`doc_tool__qcc` - N/A + * - 5a-3 + - Clang + - Host C++ compiler used for LLVM source-based coverage instrumentation and reporting workflows + - 19.x (see [1]_) + - YES + - :need:`rl__infrastructure_tooling_community` + - :need:`doc_tool__clang` + - YES 5b Rust Compiler diff --git a/docs/score_tools/tools_compiler/_assets/clang-instrumentation.drawio.svg b/docs/score_tools/tools_compiler/_assets/clang-instrumentation.drawio.svg new file mode 100644 index 00000000000..8910bd09d61 --- /dev/null +++ b/docs/score_tools/tools_compiler/_assets/clang-instrumentation.drawio.svg @@ -0,0 +1,349 @@ + + + + + + + + + + + +
+
+
+ Instrumented +
+ binary +
+
+
+
+ + + Instrumented... + + + + + + + + + + + + + +
+
+
+
+ + Software + +
+
+ + sources (C++) + +
+
+
+
+ + + Software... + + + + + + + + + + + + + + + +
+
+
+ + clang / llvm_cov + +
+
+
+ + + clang / llvm_cov + + + + + + + + + + + + + + + + + +
+
+
+
+ Test + + sources (C++) + +
+
+
+
+ + + Test sources (C++) + + + + + + + + + + + + + +
+
+
+
+ Configuration +
+
+
+
+ + + Configuration + + + + + + + + + + + + + + + +
+
+
+ Build log +
+
+
+ + + Build log + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + Dependencies + +
+
+
+ + + Dependencies + + + + + + + + + + + + + + + +
+
+
+ Coverage data +
+ (*.profraw) +
+
+
+
+ + + Coverage data... + + + + + + + + + + + + + + + +
+
+
+ llvm-profdata +
+
+
+ + + llvm-profdata + + + + + + + + + + + + + + + +
+
+
+ llvm-cov +
+
+
+ + + llvm-cov + + + + + + + + + + + + + + + +
+
+
+ Coverage data +
+ (*.profdata) +
+
+
+
+ + + Coverage data... + + + + + + + + + + + + + + + +
+
+
+ Coverage data +
+ (Text/JSON/HTML/LCOV-export) +
+
+
+ + + Coverage data... + + + + + + + + + + Text is not SVG - cannot display + + + + diff --git a/docs/score_tools/tools_compiler/_assets/clang.drawio.svg b/docs/score_tools/tools_compiler/_assets/clang.drawio.svg new file mode 100644 index 00000000000..b06b756282b --- /dev/null +++ b/docs/score_tools/tools_compiler/_assets/clang.drawio.svg @@ -0,0 +1,216 @@ + + + + + + + + + + + +
+
+
+ Binary +
+
+
+ + + Binary + + + + + + + + + + + + + +
+
+
+
+ + Software + +
+
+ + sources (C++) + +
+
+
+
+ + + Software... + + + + + + + + + + + + + + + +
+
+
+ + clang + +
+
+
+ + + clang + + + + + + + + + + + + + +
+
+
+ + + Object files + + +
+
+
+ + + Object files + + + + + + + + + + + + + + + + + +
+
+
+
+ Dependencies +
+
+
+
+ + + Dependencies + + + + + + + + + + + + + +
+
+
+
+ Configuration +
+
+
+
+ + + Configuration + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ Build log +
+
+
+ + + Build log + + + + + + + + + + + + + + Text is not SVG - cannot display + + + + diff --git a/docs/score_tools/tools_compiler/clang.rst b/docs/score_tools/tools_compiler/clang.rst new file mode 100644 index 00000000000..66031869415 --- /dev/null +++ b/docs/score_tools/tools_compiler/clang.rst @@ -0,0 +1,248 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +.. doc_tool:: clang + :id: doc_tool__clang + :status: draft + :version: 1 + :tool_version: 19.x + :tcl: HIGH + :safety_affected: YES + :security_affected: NO + :realizes: wp__tool_verification_report[version==1] + :tags: tool_management, tools_compiler + +Clang Compiler Verification Report +================================== + +Introduction +------------ +Scope and purpose +~~~~~~~~~~~~~~~~~ +Clang is the C/C++ compiler front end of the LLVM toolchain. + +In the context of the S-CORE project, clang is used on the Linux host for +verification activities that require LLVM source-based instrumentation. +The associated LLVM coverage tooling (``llvm-profdata`` and ``llvm-cov``) is +used together with clang-generated instrumentation for structural coverage +evidence generation and reporting. +For traceability in S-CORE naming, this report explicitly refers to +``llvm_cov`` (llvm-cov) and ``llvm_cov_profdata`` (combined llvm-profdata/ +llvm-cov workflow). + +This report covers the Linux host verification workflow only. It does not +cover production-target compilation with clang, Rust compiler qualification, +or QNX-specific coverage flows. + +Inputs and outputs +~~~~~~~~~~~~~~~~~~ +| Inputs: Software sources (C++), compiler options, build configuration, profile runtime output (``*.profraw``) +| Outputs: Object files, host test binaries, instrumentation metadata, merged profile data (``*.profdata``), coverage reports + +.. figure:: _assets/clang.drawio.svg + :width: 80% + :align: center + :alt: clang build + +.. figure:: _assets/clang-instrumentation.drawio.svg + :width: 80% + :align: center + :alt: Code coverage with clang + + + clang overview + +Available information +~~~~~~~~~~~~~~~~~~~~~ +- Version: 19.x [1]_ +- Official documentation: https://clang.llvm.org/docs/ +- Official documentation llvm-cov: https://llvm.org/docs/CommandGuide/llvm-cov.html +- Official documentation llvm-profdata: https://llvm.org/docs/CommandGuide/llvm-profdata.html + +Installation and integration +---------------------------- +Installation +~~~~~~~~~~~~ +clang is provided via the LLVM host toolchain used by the project. + +Integration +~~~~~~~~~~~ +clang is selected in dedicated host build configurations used for coverage +instrumentation and related verification workflows. + +Coverage is generated as part of the host verification workflow: + +#. Build and execute instrumented unit tests. +#. Merge raw profiles with llvm-profdata. +#. Generate coverage reports with llvm-cov. + +The qualification boundary is explicitly limited to this workflow: + +* clang adds instrumentation during host builds through LLVM source-based + coverage options. +* ``llvm-profdata`` merges ``*.profraw`` execution data into ``*.profdata``. +* ``llvm-cov`` converts the instrumented binaries and merged profile data into + developer-facing coverage reports. + +The resulting coverage report is a development artifact used by human +reviewers in verification activities. Instrumented binaries are executed only +during test runs and are not deployed as production software artifacts. + +Environment +~~~~~~~~~~~ +Requires Linux host environment and Bazel toolchain integration. + +Safety evaluation +----------------- +This section outlines the safety evaluation of clang for its use within the +S-CORE project. + +.. list-table:: clang safety evaluation + :header-rows: 1 + :widths: 1 2 8 2 6 4 2 2 + + * - Malfunction identification + - Use case description + - Malfunctions + - Impact on safety? + - Impact safety measures available? + - Impact safety detection sufficient? + - Further additional safety measure required? + - Confidence (automatic calculation) + * - 1 + - Host compilation for coverage-enabled tests + - | Semantically wrong host test binary + | Could distort verification conclusions. + - yes + - yes + - yes + - no + - high + * - 2 + - Coverage reporting + - | Coverage data too high + | Reported coverage is higher than actual, masking untested code. + - yes + - yes + - yes + - no + - high + * - 3 + - Coverage reporting + - | Coverage data too low + | Reported coverage is lower than actual, causing unnecessary rework. + - no + - yes + - yes + - no + - high + * - 4 + - Profile merge/report generation + - | Incomplete profile merge or incorrect report filtering + | Can distort coverage results. + - yes + - yes + - yes + - no + - high + +Security evaluation +------------------- +This section outlines the security evaluation of clang for its use within the +S-CORE project. + +.. list-table:: clang security evaluation + :header-rows: 1 + + * - Threat identification + - Use case description + - Threats + - Impact on security? + - Impact security measures available? + - Impact security detection sufficient? + - Further additional security measure required? + * - 1 + - TBD + - TBD + - TBD + - TBD + - TBD + - TBD + +Confidence measures +------------------- +To increase confidence in the clang/``llvm_cov``/``llvm_cov_profdata`` coverage +workflow, the following measures are applied or recommended for the S-CORE +verification environment: + +* Treat the coverage workflow as a separate verification path from production + builds so that instrumentation-related effects stay confined to host test + execution. +* Provide a small validation suite with dedicated C++ source files whose + expected structural coverage is known in advance. +* Cover representative language constructs such as straight-line code, + conditional branches, loops, switch statements, short-circuit conditions, + templates and excluded regions. +* Provide unit tests for these validation sources so that specific coverage + outcomes are exercised intentionally, for example full coverage, partial + branch coverage and deliberately uncovered code. +* Execute the validation suite in CI with LLVM instrumentation enabled and + compare the produced ``*.profraw``/``*.profdata`` data and final + ``llvm-cov`` report against expected results. +* Include at least one intentionally uncovered file or line in the validation + suite so that false-positive reporting is easier to detect during review and + CI execution. +* Re-run the coverage workflow from a clean build in CI so that profile merge + failures, missing reports and threshold regressions are visible as build + failures rather than silent degradations. +* Use a secondary plausibility cross-check that compares the expected coverage + baseline (for example number of source files and rough line-count totals in the project) + against the files and aggregates reported by ``llvm-cov``. +* Compare known-covered and known-uncovered lines in the generated report as a + targeted spot check when the LLVM version or reporting flow changes. +* Re-run the validation suite whenever the LLVM/clang toolchain version changes + or the coverage workflow is modified. +* Keep the validation sources and expected results version-controlled so that + tool behavior is reproducible and regressions become reviewable. + +The dedicated validation suite with intentionally designed reference cases is +the primary confidence measure. The file/line-count comparison is a secondary +sanity check that can reveal missing files, empty profiles or implausible report +totals, but it does not by itself prove correct branch/region attribution. + +For the LLVM coverage workflow, the relevant error-detection mechanisms are +independent review of selected report lines, CI re-execution from scratch, +threshold-based gating and validation cases with known expected outcomes. +These measures are intended to detect both optimistic and pessimistic coverage +misreporting before the results are used in a safety argument. + +Result +------ +clang is used in host verification workflows and is currently evaluated with +confidence level HIGH for this use case. +The associated llvm-profdata/llvm-cov coverage tooling is covered by this +verification context. +This includes ``llvm_cov`` and ``llvm_cov_profdata``. + +Within this scope, ``llvm-profdata`` and ``llvm-cov`` are treated as +development tools whose output is a verification report for human assessment. +Incorrectly low coverage remains a conservative failure mode. Incorrectly high +coverage is the relevant safety concern, and the confidence measures above are +the basis for keeping this workflow in the project-internal high-confidence +category for Linux host verification use. On that basis, this report does not +identify an additional formal qualification action for the current Linux host +coverage-reporting scope. + +.. [1] The tool version mentioned in this document is preliminary. It is subject to + change and will be updated in future. diff --git a/docs/score_tools/tools_compiler/gcc.rst b/docs/score_tools/tools_compiler/gcc.rst index 35d3f3a931e..d9ac25ee378 100644 --- a/docs/score_tools/tools_compiler/gcc.rst +++ b/docs/score_tools/tools_compiler/gcc.rst @@ -34,6 +34,8 @@ GCC is open-source C/C++ compiler. It is widely used in the software development In the context of the S-CORE project, GCC is used as a development tool to compile software components during the development phase. However, it is not used for production builds of safety-related software components. +The GCC toolchain also provides ``gcov`` for coverage instrumentation/reporting. This capability is available in the project environment if needed, but it is not the primary structural-coverage reporting path used in S-CORE. + Therefore, the safety and security impact of GCC is "NO". @@ -198,6 +200,7 @@ Result ------ GCC is not used for production builds nor during software verification phases. The tool is used for development purposes only to support early-stage issue identification and resolution. +Coverage with ``gcov`` remains technically available through the GCC toolchain for dedicated use cases. .. [1] The tool version mentioned in this document is preliminary. It is subject to change and will be updated in future. diff --git a/docs/score_tools/tools_compiler/index.rst b/docs/score_tools/tools_compiler/index.rst index 61463f5935e..3e04fc278c5 100644 --- a/docs/score_tools/tools_compiler/index.rst +++ b/docs/score_tools/tools_compiler/index.rst @@ -29,5 +29,6 @@ Compiler Tools Overview :hidden: :maxdepth: 2 + clang gcc qcc