How to get LauncherDiscoveryRequest to see pre-compiled tests?

[jramsley]

Hello,

Full disclosure, I am not much of a java developer so a lot of the concepts around JARs and classpaths have a cursory knowledge of.

We've been trying to update our test infrastructure to be more programmatic as well as handle a level of test sharding for integration tests that isn't default out-of-the-box with gradle Junit 5. Our goal is to be able to write a custom LauncherDiscoveryRequest to generate a TestPlan by pointing to some pre-compiled tests/JARs and then do some filtering + ordering based on that. However, we are never able to get any tests found.

1. I can see that I have a JAR of tests on my local file system, /Users/jramsley/stage/test/libs/tests.jar
2. I can inspect the JAR to see that there are compiled classes in them which look like our test classes

jramsley:stage jramsley$ jar tvf ./test/lib/tests.jar | grep Test
...
   870 Mon May 20 12:55:08 EDT 2024 com/company/restapi/web/controller/RestReportTest$17.class
   943 Mon May 20 12:55:08 EDT 2024 com/company/restapi/web/controller/RestPermissionsTest$1.class
  2724 Mon May 20 12:55:08 EDT 2024 com/company/restapi/web/controller/TranslatorTest.class
...

3. If I run a javap -v -classpath tests.jar <class> I can see that the @Test annotation is on the class

jramsley:stage jramsley$ javap -v -classpath ./test/lib/tests.jar com.company.restapi.web.controller.RestPermissionsTest
...
RuntimeVisibleAnnotations:
      0: #922()
        org.junit.jupiter.api.Test

  public void <test name>() throws java.lang.Exception;
    descriptor: ()V
...

4. If I point to this JAR in our custom test runner, the test plan discovers no tests

@TaskAction
void runTestDiscovery() {
    Path classPath = Path.of("/Users/jramsley/stage/test/libs/tests.jar");
    println(classPath);

    LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
        .selectors(
            selectClasspathRoots(Collections.singleton(classPath))
        )
        .build();

    try (LauncherSession session = LauncherFactory.openSession()) {
        TestPlan testPlan_ = session.getLauncher().discover(request);
        println('Print testPlan');
        println(testPlan_);
        println(testPlan_.containsTests());
        println(testPlan_.getRoots().size());
        println(testPlan_.getRoots());
        testPlan_.getRoots().forEach(root -> {
            println('print root');
            println(root);
            try {
                Integer printDescendantsSize = testPlan_.getDescendants(root).size();
                println('print descendants size');
                println(printDescendantsSize);

                println('print descendants');
                println(testPlan_.getDescendants(root));
                println("print children");
                println(testPlan_.getChildren(root));
            } catch (Exception e) {
                println('error getting descendants');
                println(e);
            }
        })
    }
    catch (Exception error) {
        println('There is an error');
        println(error);
    }
}

with output

> Task :java:test:discoveredTests
/Users/jramsley/stage/test/libs/tests.jar
Print testPlan
org.junit.platform.launcher.core.InternalTestPlan@3d8c6278
false
1
[TestIdentifier [uniqueId = [engine:junit-jupiter], parentId = null, displayName = 'JUnit Jupiter', legacyReportingName = 'JUnit Jupiter', source = null, tags = [], type = CONTAINER]]
print root
TestIdentifier [uniqueId = [engine:junit-jupiter], parentId = null, displayName = 'JUnit Jupiter', legacyReportingName = 'JUnit Jupiter', source = null, tags = [], type = CONTAINER]
print descendants size
0
print descendants
[]
print children
[]

I would expect this to scan the compiled test classes, find the annotations and add them as children to the root/containers. At minimum I would expect testPlan_.containsTests() to be returning true.

So at this point I am sort of at a loss how to get the selector for LauncherDiscoveryRequest to be able to scan and find these annotations. I've tried following Discovering Tests but unable to get the launcher to find any tests. Can anyone help point me in the right direction to debug this?

[mpkorstanje]

Have you verified that the test jar is on your class path with the expected class path root?

[jramsley]

That seems to be the issue. Looking at https://junit.org/junit5/docs/5.0.3/api/org/junit/platform/engine/discovery/ClasspathResourceSelector.html there is a note that the classpath has to be on the engine. I'm able to get this working outside of running through gradle, but in the context of gradle it looks like the jar is on the discovery JVM and not the engine JVM.

[marcphilipp]

@jramsley If you provide a reproducer, I'd be happy to take a look.

[nadiramra]

I am not sure my problem is related to this or not, but I am trying to run junit tests from command line. The tests run fine within IDE, I am using eclipse.

So I have a shell script file:

JAVA_CLASSPATH="\
${JAVA_HOME}/lib/tools.jar:\
/build/junit-jupiter-api-5.12.1.jar:\
/build/junit-jupiter-engine-5.12.1.jar:\
/build/junit-platform-engine-1.12.1.jar:\
/build/junit-platform-commons-1.12.1.jar:\
/build/junit-platform-launcher-1.12.1.jar:\
/build/iwsut.jar:\
"

JAVA_FLAGS="\
 -Dfile.encoding=UTF-8\
 "

${JAVA_HOME}/bin/java ${JAVA_FLAGS} -classpath ${JAVA_CLASSPATH} com.tests.Tester "$@"

iwsut.jar contains the com.tests.Tester and the junit tests. The Tester has the following:

LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
			                .selectors(DiscoverySelectors.selectPackage("com.tests.spi"))
			                .filters(ClassNameFilter.includeClassNamePatterns(".*Test"))
			                .build();
 
Launcher launcher = LauncherFactory.create();

  TestPlan testPlan = launcher.discover(request);
   launcher.registerTestExecutionListeners(listener);
  launcher.execute(request);

  TestExecutionSummary summary = listener.getSummary();
  summary.printTo(new PrintWriter(System.out));  

Only thing that is output is:

Test run finished after 130 ms        

[         0 containers aborted    ]  
[         0 containers successful ]  
[         1 containers failed     ]  
[         0 tests found           ]  
[         0 tests skipped         ]  
[         0 tests started         ]  
[         0 tests aborted         ]  
[         0 tests successful      ]  
[         0 tests failed          ]  

What am I doing wrong?

[marcphilipp]

Hmm, there's not much to go on. Could you please try the following?

LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
			                .selectors(DiscoverySelectors.selectPackage("com.tests.spi"))
			                .filters(ClassNameFilter.includeClassNamePatterns(".*Test"))
							.configurationParameter("junit.platform.stacktrace.pruning.enabled", "false")
			                .build();
 
Launcher launcher = LauncherFactory.create();

TestPlan testPlan = launcher.discover(request);
launcher.registerTestExecutionListeners(listener);
launcher.execute(request);

TestExecutionSummary summary = listener.getSummary();
summary.printFailuresTo(new PrintWriter(System.out), 1000);

[nadiramra]

Failures (1):
JUnit Jupiter
=> org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-jupiter' failed to execute tests
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:213)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:170)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:94)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:59)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:142)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:58)
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:103)
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:85)
org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:47)
org.junit.platform.launcher.core.InterceptingLauncher.lambda$execute$1(InterceptingLauncher.java:39)
org.junit.platform.launcher.core.ClasspathAlignmentCheckingLauncherInterceptor.intercept(ClasspathAlignmentCheckingLauncherInterceptor.java:25)
org.junit.platform.launcher.core.InterceptingLauncher.execute(InterceptingLauncher.java:38)
org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:47)
org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:63)
com.ibm.systemi.iws.tests.JUnitTestRunner.main(JUnitTestRunner.java:154)
Caused by: java.lang.NoClassDefFoundError: org.opentest4j.TestAbortedException
org.junit.jupiter.engine.support.OpenTest4JAndJUnit4AwareThrowableCollector.(OpenTest4JAndJUnit4AwareThrowableCollector.java:38)

    org.junit.jupiter.engine.support.JupiterThrowableCollectorFactory.createThrowableCollector(JupiterThrowableCollectorFactor

y.java:34)
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:94)
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTest
ExecutorService.java:35)
org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:201)
[...]
Caused by: java.lang.ClassNotFoundException: org.opentest4j.TestAbortedException
java.net.URLClassLoader.findClass(URLClassLoader.java:610)
java.lang.ClassLoader.loadClassHelper(ClassLoader.java:950)
java.lang.ClassLoader.loadClass(ClassLoader.java:895)
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:353)
java.lang.ClassLoader.loadClass(ClassLoader.java:878)
[...]

[nadiramra]

Seemed like I needed opentest4j jar file? after downloading I now get no container errors, but no tests identified either.

Test run finished after 122 ms
[ 1 containers found ]
[ 0 containers skipped ]
[ 1 containers started ]
[ 0 containers aborted ]
[ 1 containers successful ]
[ 0 containers failed ]
[ 0 tests found ]
[ 0 tests skipped ]
[ 0 tests started ]
[ 0 tests aborted ]
[ 0 tests successful ]
[ 0 tests failed ]

[nadiramra]

Note that if I specify:
.selectors(DiscoverySelectors.selectClass(CreateWebServicesServerTest.class))
Things work. But would really, really like to use selectPackage. Anyway to debug this?

[marcphilipp]

Sure, you can attach a debugger and add a breakpoint in ClassContainerSelectorResolver#resolve(PackageSelector, Context). I suspect findAllClassesInPackage comes back empty for some reason. If that's the case, I'd go deeper and see what happens in DefaultClasspathScanner.