Deprecation of test framework support

[Warwolt]

As of version 0.9, support for test frameworks such as GoogleTest have been deprecated as a consequence of phasing out just in time compilation support.

Going forward, will the general approach for using Mull to be to just run an executable that returns a non-zero return code if the code-under-test does not conform to the tests? I've only used Mull together with gtest, so it would be nice to hear you maintainers thoughts around the best practice of Mull v0.9 together with frameworks moving forward.

[AlexDenisov]

Hi @Warwolt,

Very short answer: yes, any test program that returns a non-zero exit code is considered as a failing test, and therefore killing the mutant.

Somewhat more historical details follow.

1. Call trees.
   The explicit test framework was just an optimization. Initially, when we just started working on Mull, it was extremely slow, so we were looking for a way to decrease the execution time.
   One of the first ideas was to gather code coverage and to only mutate functions that were executed during the original test run.
   Let's say we have the following program:

#include "gtest/gtest.h"

int sum(int a, int b) {
  return a + b;
}
TEST(GoogleTest, example) {
  ASSERT_EQ(sum(4, 5), 9);
}

int main() {
  testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS();
}

For this program, Mull would gather code coverage info in the form of a call tree, e.g.:

<root>
  main
    testing::InitGoogleTest
    RUN_ALL_TESTS
      /// more gtest internal calls that eventually call the test function
        GoogleTest_example
          sum

In the case of the explicit test framework support, we know that the GoogleTest_example function is the test. Therefore, we should only mutate functions that are below the GoogleTest_example in the call tree, i.e., the sum function.
The advantage of this approach is that we just skip a bunch of work: we do not mutate the test, we do not mutate main, we do not mutate gtest internals.

The drawback of this approach is that we cannot know for sure (at least I'm not aware of a way to do it) how to build the call-tree for a multithreaded program: we'd need to build a call-tree for each thread and then somehow merge those into one.

2. Mutation distance.
   Another optimization that we used a long time ago is the so-called mutation distance. Given the call tree, we can calculate how far a function from the test. In the example above, the distance from GoogleTest_example to sum would be 1. The optimization was to cut-off functions that are farther than a certain threshold, e.g., do not mutate functions with a distance more than 10 or 20.

3. Exit code and survived mutants.
   As of 0.9.0, Mull considers a program with a non-zero exit code to be a failing test and therefore killing the mutant. The fun fact is that it was like that even before. Mull was doing the same thing, but through the JIT interface: the NativeRunner was just calling the main function and then checked the return code. So nothing changed in this regard.

4. Per test code coverage
   With the explicit test framework support, we could run each test separately and gather code coverage information per test. That allowed us to save some time. Let's say a test A tests function a and a test B tests function b. Previously, we would identify that mutations at a only reachable from the test A (and the same for b and B). Then, Mull would only run test A against mutations in a because the test B cannot kill unreachable mutations (and vice versa for b and B). So in the case both a and b have one mutant each we would run tests A and B only once (two test runs in total).
   Currently, without explicit test framework support we cannot gather this information the same way so we would run both tests against both mutants (four test runs in total). So there is an overhead in this case.

---

I apologize for the chaotic response, I didn't have time to write a short and conscise answer :)
If you are interested, you can find some more implementation details in the paper and in my old talk.