Skip to content

code_coverage.md

Latest commit

 

History

History
246 lines (173 loc) · 8.29 KB

code_coverage.md

File metadata and controls

246 lines (173 loc) · 8.29 KB
Raw

Remote, Daemon, sudo test coverage

official docs

https://github.com/taiki-e/cargo-llvm-cov

background

In order to gather code coverage reports for fpgad, we need to run the daemon separately to the test executable and to gather the results after the fact. llvm-cov can be used to assemble a coverage report by combining raw profile data (.profraw extensions are typical) and with source code and executable binaries. It does this using internal path discovery which will be explained within this document.

We cannot use tarpaulin because

  1. it does not have support for subprocess/external profiling
  2. it does not run on arm

We can't just run the direct llvm-cov because that

  1. doesn't start fpgad
  2. does not build the daemon with the right flags
  3. requires running cargo with sudo (bad)

Instead, we need to use llvm-cov to assemble a coverage report from separate profiles generated by each test stream, and cross-reference that with the binaries we ran. These binaries need to expose llvm coverage information so we need to build them with special flags.

dependencies

The llvm-cov tool can be installed using cargo:

cargo install cargo-llvm-cov

then on first run it prompts you to install the llvm-tools-preview component:

$ cargo llvm-cov show-env
info: cargo-llvm-cov currently setting cfg(coverage); you can opt-out it by passing --no-cfg-coverage
I will run `rustup component add llvm-tools-preview --toolchain <aarch triple>` to install the `llvm-tools-preview` component for the selected toolchain.
Proceed? [Y/n] y
info: downloading component 'llvm-tools'
info: installing component 'llvm-tools'

This can be avoided by running

rustup component add llvm-tools-preview

manually.

build and run

llvm-cov provides a way to conveniently collect results from external binaries as described very briefly in the docs.

So following that, in order to build the tests and also gather results from unit tests, certain environment variables must be set. Handily, this can be done using their method:

source <(cargo llvm-cov show-env --export-prefix)

or alternatively

eval "$(cargo llvm-cov show-env --export-prefix)"

the body of which (without eval $(...)) will print something like

RUSTFLAGS='-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target'
LLVM_PROFILE_FILE='<working dir>/target/<working dir name>-%p-%4m.profraw'
CARGO_LLVM_COV=1
CARGO_LLVM_COV_SHOW_ENV=1
CARGO_LLVM_COV_TARGET_DIR=<working dir>/target

and for most approaches, this is enough. In our case, we need to append one additional flag
(-C llvm-args=-runtime-counter-relocation) to allow for continuous printing of code coverage profile data while executing our daemon. Without this flag, the .profraw file can only be written to after a successful exit code (exit(0)) and our daemon does not catch signals and exit safely, it instead must be terminated. Therefore, we can do

export RUSTFLAGS="$RUSTFLAGS -C llvm-args=-runtime-counter-relocation"

to append this flag. This means that all of our binaries (including test binaries) will be capable of continuous output, but it must be enabled with a llvm flag within LLVM_PROFILE_FILE (explained further below).

Now we must build our executable binaries using

cargo build

and our test binaries using

 cargo test --no-run

Then we can run our unit tests by running, for example,

cargo test --bins

Build/test targets can be specified here as normal. In our case,

cargo test --bin fpgad

will run only fpgad's unit tests, and we don't want to gather the results of the target which we do not currently test so instead of cargo build we can select only our daemon like cargo build --bin fpgad so that cli target is never built.

There are --exclude-* flags available for llvm-cov but these cannot be used here, nor in the cargo llvm-cov report steps later, so the only way I found to avoid having unwanted binaries in the coverage report misrepresenting results, is to never build them (at least not with llvm related compiler flags) in the first place.

With the flags being set by show-env as with above, the .profraw files are dumped into ./target unless the CARGO_TARGET_DIR and/or CARGO_LLVM_COV_TARGET_DIR variables are overridden. If you change these directories, the executable and test binaries must be present in the proper places inside these target dirs. It is easiest to use the defaults.

The test executables for integration tests are produced with a hash string appended to the binary name. Some bash magic can be used to extract the path to these binaries from the cargo output by doing something like

<target>_test=$(
  cargo test --no-run --test <target> | \
  grep 'Executable tests/<target>.rs' | \
  awk '{ gsub(/[()]/, ""); print $3 }'
)

in our case we need to run a daemon with continuous collection (enabled by having the aforementioned %c flag present in the LLVM_PROFILE_FILE variable) before running out integration tests so we can do that like

sudo env LLVM_PROFILE_FILE="${CARGO_LLVM_COV_TARGET_DIR}/<daemon>-%p%c.profraw" <daemon> [--options] &> <daemon>.log &

DAEMON_PID=$! # to use with kill when done

and in our case we can check our dbus is up and running by polling <daemon>.log for a known string like

timeout 5 bash -c '
  while ! grep -q "<known string>" <daemon>.log; do
    sleep 0.1
  done
'

Once that is up and running, we can run our integration test binaries:

sudo env LLVM_PROFILE_FILE="${CARGO_LLVM_COV_TARGET_DIR}/test_<target>>-%p.profraw" "$<target>_test" [--test-options]

where LLVM_PROFILE_FILE is set to output to the same directory as the already generated .profraw files (from running unit tests). If you want to edit the filename but otherwise not change the LLVM_PROFILE_FILE, you can use bash magic to do that.

Once all our integration tests are done, we can kill the daemon

sudo kill $DAEMON_PID
wait $DAEMON_PID

# print leftover `Terminated....` stdout buffer
echo $()
unset DAEMON_PID

collect and process

Now that we have the collection of .profraw files (typically floating about in ./target), we can use llvm-cov to gather them up, cross-reference them with the original binaries, and work out what lines of code were never run. To do this, from whatever directory contains the top level Cargo.toml (where we have been for this entire doc), run

cargo llvm-cov report

to print the coverage report summary in the terminal (or pipe to file or whatever you want).

If you're not using ssh (using a window manager) you can run

cargo llvm-cov report --open

to generate and open the html report in a browser.

For other formats, specify the appropriate flag (and probably --output-path too ) i.e.

cargo llvm-cov report --json --output-path=coverage.json
cargo llvm-cov report --lcov --output-path=coverage.lcov
cargo llvm-cov report --text > coverage.txt
cargo llvm-cov report --html # doesn't support --output-path :(

If you do generate html output and want to copy it to a host machine just do, for example

scp -r <user>@<device>:<contents of CARGO_LLVM_COV_TARGET_DIR>/llvm-cov/html/* ./html

from the host system. Note that $CARGO_LLVM_COV_TARGET_DIR is an environment variable set on the machine so cannot be used directly.

Setting coverage limits

The docs provide these options for the llvm-cov report to enable a script to fail (useful for GitHub actions, for example) under certain conditions

--fail-under-functions <MIN>
    Exit with a status of 1 if the total function coverage is less than MIN percent

--fail-under-lines <MIN>
    Exit with a status of 1 if the total line coverage is less than MIN percent

--fail-under-regions <MIN>
    Exit with a status of 1 if the total region coverage is less than MIN percent

--fail-uncovered-lines <MAX>
    Exit with a status of 1 if the uncovered lines are greater than MAX

--fail-uncovered-regions <MAX>
    Exit with a status of 1 if the uncovered regions are greater than MAX

--fail-uncovered-functions <MAX>
    Exit with a status of 1 if the uncovered functions are greater than MAX