kcov and bash_unit doesn't play well out of the box

[jbergstroem]

I'm trying to add coverage to hadolint-gh-action and the most output friendly tool seems to be kcov. It does not work very well out of the box though. It only shows the coverage paths through bash_unit and not the test files themselves:

kcov coverage bash_unit test/e2e.sh test/unit.sh
Running tests in test/e2e.sh
test_bash_glob_expansion
test_ci_vs_non_ci_annotation_behavior
test_ci_vs_non_ci_hadolint_version_output
test_custom_config
test_custom_dockerfile_path
test_custom_hadolint_path
test_custom_output_format
test_default_hadolint_config
test_default_path
test_default_path_with_dockerfile
test_disable_annotate
test_multiple_dockerfiles
test_output_with_nonzero_exit
test_override_errorlevel
test_version_output ]]
        Running test_annotate ... SUCCESS
        Running test_bash_glob_expansion ... SUCCESS
        Running test_ci_vs_non_ci_annotation_behavior ... SUCCESS
        Running test_ci_vs_non_ci_hadolint_version_output ... SUCCESS
        Running test_custom_config ... SUCCESS
        Running test_custom_dockerfile_path ... SUCCESS
        Running test_custom_hadolint_path ... SUCCESS
        Running test_custom_output_format ... SUCCESS
        Running test_default_hadolint_config ... SUCCESS
        Running test_default_path ... SUCCESS
        Running test_default_path_with_dockerfile ... SUCCESS
        Running test_disable_annotate ... SUCCESS
        Running test_multiple_dockerfiles ... SUCCESS
        Running test_output_with_nonzero_exit ... SUCCESS
        Running test_override_errorlevel ... SUCCESS
        Running test_version_output ... SUCCESS
Running tests in test/unit.sh
test_missing_hadolint_binary
test_missing_jq_binary
test_output_hadolint_version
test_validate_annotate
test_validate_error_level
test_validate_invalid_annotate
test_validate_invalid_error_level
test_validate_invalid_output_format
test_validate_output_format ]]
        Running test_exit_with_error ... SUCCESS
        Running test_missing_hadolint_binary ... SUCCESS
        Running test_missing_jq_binary ... SUCCESS
        Running test_output_hadolint_version ... SUCCESS
hadolint_version=2.10.0 =~ .*hadolint_version=2.10.0.* ]]
        Running test_validate_annotate ... SUCCESS
        Running test_validate_error_level ... SUCCESS
        Running test_validate_invalid_annotate ... SUCCESS
        Running test_validate_invalid_error_level ... SUCCESS
        Running test_validate_invalid_output_format ... SUCCESS
        Running test_validate_output_format ... SUCCESS
Overall result: SUCCESS

jq . coverage/bash_unit.1cf006142dc477ff/coverage.json:

{
  "files": [
    {
      "file": "/opt/homebrew/Cellar/bash_unit/2.3.3/bin/bash_unit",
      "percent_covered": "45.74",
      "covered_lines": "129",
      "total_lines": "282"
    }
  ],
  "percent_covered": "45.74",
  "covered_lines": 129,
  "total_lines": 282,
  "percent_low": 25,                                                                                                                                            "percent_high": 75,
  "command": "bash_unit",
  "date": "2025-09-06 12:51:01"
}

I wrote a few small tests with bats and it seems to work out of the box.

Mostly sharing as I go at this point to see the differences and possibly lead to a PR or just documentation.

[pgrange]

Hi @jbergstroem and thanks for using bash_unit.

It seems kcov has an interesting way of encapsulating a shell script before analyzing coverage. For instance, I noted that it just ignores #! headers and launches bash directly so if you add that as the first line of your test file:

#!/usr/bin/env bash_unit

Then you can launch your test directly like that and the tests will run:

./test/e2e.sh

But with kcov, if you try to compute converage that way it will fail because the file is executed as a bash script and not as a bash_unit test suite:

kcov coverage ./test/e2e.sh

I'm not sure how kcov handles this bash coverage thingy 🤔

[pgrange]

FYI, I have way more chance with bashcov. For instance with the getting_started project:

git clone https://github.com/bash-unit/getting_started.git
cd getting_started
bashcov bash_unit tests/test_*
open coverage/index.html

It's interesting to observe, though, that when the code under test is sourced, as is the case in tests/test_hello_i18n then bashcov is lost, hence the 0% coverage for hello_i18n in the screenshot above.

But when the code under test is just called as a regular script as is the case with test_hello_timed, for instance, then all is fine and we see a very good coverage for hello_timed.

[jbergstroem]

FYI, I have way more chance with bashcov. For instance with the getting_started project:

git clone https://github.com/bash-unit/getting_started.git
cd getting_started
bashcov bash_unit tests/test_*
open coverage/index.html

It's interesting to observe, though, that when the code under test is sourced, as is the case in tests/test_hello_i18n then bashcov is lost, hence the 0% coverage for hello_i18n in the screenshot above.

But when the code under test is just called as a regular script as is the case with test_hello_timed, for instance, then all is fine and we see a very good coverage for hello_timed.

Thanks for taking a look and exploring options as well. I started with bashcov too, but the output is mostly html where i am looking for the actual lcov files (and optimally pairing it with junit output from bash_unit), kcov seemed like the better option. Long game: hook up with a CI service like codecov.

I will probably tinker a bit more with kcov the coming weekend to see if there is a solution.

[pgrange]

Thanks for taking a look and exploring options as well. I started with bashcov too, but the output is mostly html where i am looking for the actual lcov files (and optimally pairing it with junit output from bash_unit)

You're mentioning junit. FYI, bash_unit also supports tap format with the option -f tap. Just in case it helps you with your exploration.

Let me know how it goes.

Cheers,

[jbergstroem]

Thanks for taking a look and exploring options as well. I started with bashcov too, but the output is mostly html where i am looking for the actual lcov files (and optimally pairing it with junit output from bash_unit)

You're mentioning junit. FYI, bash_unit also supports tap format with the option -f tap. Just in case it helps you with your exploration.

Yeah; I'm planning on converting tap to junit so codecov can pick up and monitor the test suite.

Let me know how it goes.

Will do!