Potential issue with tests derived from IndexBindingResolutionTestBase and JUnit5

[i-garrison]

I am not aware of any tests having a problem with running with JUnit5 runner. If there is a specific test(s) that you see an issue with, please let me know and I can look closer.

For clarity I mean running with JUnit5 means the JUnit5 dropdown in the launch configuration. This picture is from TESTING.md

Yes, that's the window I'm using to change between JUnit5 and JUnit3. Now if I run parser tests under JUnit5 I only see 4699 unit tests are run. If I change to JUnit3 then there are 5768 tests run. Not sure what's the actual issue (could be my openjdk-21).

I vaguely recall that e.g. those inheriting from IndexBindingResolutionTestBase and adding test cases during setup phase are not actually found by JUnit5 because they do not match JUnit5 unit test class discovery expectations. I remember looking into this while working on IndexConcepTest which is most recent of this kind. I decided to switch to JUnit3 for parser tests locally for now and just ask about current state of JUnit3 vs JUnit5 so we probably can pick the right version and fix any tests which are not found or fail with that.

Originally posted by @i-garrison in #1366 (comment)

[jonahgraham]

Yes, I can confirm this now 😢 Thank you for bringing this to my attention!

I will do some refactoring or instruction updating asap because I don't think these tests are running on CI either

[jonahgraham]

Have a look at eclipse-jdt/eclipse.jdt.ui#2591 - it turns out one of the issues is a bug in JDT (afaict) that JUnit5 doesn't discover tests in inner classes properly.

[i-garrison]

Just wanted to confirm finding about junit5 not finding tests eclipse-jdt/eclipse.jdt.ui#2591, looked at my exact setup and using current main branch locally:

• running org.eclipse.cdt.tests/parser with JUnit3 there are 5765 tests (that's what I normally used before this ticket)
• running org.eclipse.cdt.tests/parser with JUnit5 and there are 4696 tests (that's what I tried before discovering issue)

Now as recommended in eclipse-jdt/eclipse.jdt.ui#2591 (comment)

• running org.eclipse.cdt.tests/parser/org with JUnit5 there are 6039 tests

[jonahgraham]

The 6k number looks correct as running running org.eclipse.cdt.tests/parser with JUnit3 was ommitting the JUnit4/5 style tests (that uses @Test instead of TestCase - confusingly, some tests use both, but extending TestCase and using annotations too). And running org.eclipse.cdt.tests/parser with JUnit5 was ommitting the inner classes.

Best as I can tell the org.eclipse.cdt.tests/parser/org with JUnit5 runs all the tests. Which ironically means the refactoring I have been doing has been less valuable because the underlying issue is the JDT bug.

However, because I have started I will aim to complete JUnit5 migration because it makes the code cleaner and the test reporting more details. I'll see if I can fix the bug in JDT too.

[jonahgraham]

As the original issue is with JDT eclipse-jdt/eclipse.jdt.ui#2591 (comment). I have unassociated the JUnit conversion from this issue. I will add a bit of documentation to TESTING.md and then close this.