Please test and defend use of insert-gcov-profiling for coverage measurement

[ctz]

This is a feature request (tangentially related to #4865) for cargo to include an integration test that ensures that this method of building code with coverage measurement enabled continues to work in future versions.

Here's documentation of how I currently do this for the sct and rustls projects:

Prerequisites: a compatible llvm installation, lcov, nightly rustc

1. Clean up: cargo clean and delete all .gcda and .gcno intermediate files
2. Build: run cargo rustc --all-features --profile test --lib, with RUSTC_WRAPPER set to a shell script that adds COVERAGE_OPTIONS to the rustc invocation for a specific crate build (see below).
3. Execute the library unit tests program that now exists, eg ./target/debug/sct-43d2a05ec780a9bc
4. Use lcov to extract coverage data from resulting gcno files: run lcov --gcov-tool ./admin/llvm-gcov --rc lcov_branch_coverage=1 --rc lcov_excl_line=assert --capture --directory . --base-directory . -o sct.info
5. Merge all .info files into one coverage.info: run lcov --gcov-tool ./admin/llvm-gcov --rc lcov_branch_coverage=1 --rc lcov_excl_line=assert --add sct.info -o coverage.info
6. Trim coverage.info to remove uninteresting sources: run lcov --gcov-tool ./admin/llvm-gcov --rc lcov_branch_coverage=1 --rc lcov_excl_line=assert --extract coverage.info /home/full/path/to/sct/src/\* -o final.info

My RUSTC_WRAPPER program is:

#!/bin/bash -e

get_crate_name()
{
  while [[ $# -gt 1 ]] ; do
    v=$1
    case $v in
      --crate-name)
        echo $2
        return
        ;;
    esac
    shift
  done
}

case $(get_crate_name "$@") in
  sct)
    EXTRA=$COVERAGE_OPTIONS
    ;;
  *)
    ;;
esac

exec "$@" $EXTRA

admin/llvm-gcov is:

#!/bin/sh -e
llvm-cov gcov $*

This method works in cargo 0.23.0 (61fa02415 2017-11-22) but is broken in cargo 0.25.0-nightly (e08f31018 2017-12-24) (and not just due to #4865).

[ctz]

Hello, is there anybody I can lobby for this to be prioritized? Afaict this regression was now released in stable(!!!) and now all my builds are broken.

[alexcrichton]

@ctz I believe code coverage and such are unstable features in rustc so they don't have upstream support or stability guarantees, this is a feature which likely needs its own champion upstream in rustc before stability can be achieved.

[ctz]

How does one go about finding such a champion?

I kinda feel like this bug was me shouting "Hey! I'm walking here!" as soon as I saw breakage in nightly, but I got run over anyway. I presumed that this was the point of regularly testing with nightlies -- to prevent regressions making it to release :(

[alexcrichton]

Ah that'd either probably be championing it yourself or posting on internals to see if others are interested. You never know who might bite nowadays!

[epage]

FYU there is rust-lang/rfcs#3287 for finding a way to get built-in support for cargo.

The coverage landscape has changed a lot since this was opened. I'm going to close as this approach doesn't sound as relevant from what I've seen of coverage. If there is something I've missed, let us know and we can re-evaluate!