Convert org.eclipse.cdt.core.parser.tests.ast2.SemanticTestBase to JUnit5

[jonahgraham]

SemanticTestBase is the base class for a huge hierarchy of tests and all those test updates are included in this commit.

Part of #1380

[github-actions]

Test Results

  585 files  ± 0    585 suites  ±0   13m 40s ⏱️ -6s
9 939 tests +22  9 910 ✅ +17  29 💤 +5  0 ❌ ±0 
9 943 runs   - 12  9 914 ✅  - 17  29 💤 +5  0 ❌ ±0 

Results for commit 573e498. ± Comparison against base commit 6574af3.

This pull request removes 1312 and adds 1334 tests. Note that renamed tests count towards both.

NonIndexingTests ‑ test16bitUnicodeEscapeSequence
NonIndexingTests ‑ testAccessGlobalVariableFromConstexprFunction
NonIndexingTests ‑ testAccessGlobalVariableFromGlobalConstexpr
NonIndexingTests ‑ testAccessMemberOfCompositeParameter
NonIndexingTests ‑ testAccessOfStaticField
NonIndexingTests ‑ testAliasDeclaration
NonIndexingTests ‑ testAliasTemplate1
NonIndexingTests ‑ testArgumentEvaluation
NonIndexingTests ‑ testAssignmentOfArrays
NonIndexingTests ‑ testAssignmentOfCompositeValues
…

org.eclipse.cdt.core.parser.tests.ast2.cxx14.constexpr.ArrayTests$NonIndexingTests ‑ testAssignmentOfArrays
org.eclipse.cdt.core.parser.tests.ast2.cxx14.constexpr.ArrayTests$NonIndexingTests ‑ testAssignmentOfMultiDimensionalArrays
org.eclipse.cdt.core.parser.tests.ast2.cxx14.constexpr.ArrayTests$NonIndexingTests ‑ testGlobalArrayAccessValue
org.eclipse.cdt.core.parser.tests.ast2.cxx14.constexpr.ArrayTests$NonIndexingTests ‑ testInitializationOfArrays
org.eclipse.cdt.core.parser.tests.ast2.cxx14.constexpr.ArrayTests$NonIndexingTests ‑ testInitializationOfMultiDimensionalArrays
org.eclipse.cdt.core.parser.tests.ast2.cxx14.constexpr.ArrayTests$NonIndexingTests ‑ testPassArrayToFunctionAsPointerAndModifyCell
org.eclipse.cdt.core.parser.tests.ast2.cxx14.constexpr.ArrayTests$NonIndexingTests ‑ testPassMultiDimensionalArrayToFunctionAsPointerAndModifyCell
org.eclipse.cdt.core.parser.tests.ast2.cxx14.constexpr.ArrayTests$NonIndexingTests ‑ testPointerArithmeticsOnMultidimensionalArray
org.eclipse.cdt.core.parser.tests.ast2.cxx14.constexpr.ArrayTests$NonIndexingTests ‑ testPointerDereferencingOnArrayName
org.eclipse.cdt.core.parser.tests.ast2.cxx14.constexpr.ArrayTests$NonIndexingTests ‑ testPointerToArrayReturnedFromMemberFunction1
…

This pull request skips 5 tests.

org.eclipse.cdt.core.parser.tests.ast2.AST2CPPSpecTest ‑ _test14_8_2s8b
org.eclipse.cdt.core.parser.tests.ast2.AST2CPPTests ‑ _testConversionOperator
org.eclipse.cdt.core.parser.tests.ast2.AST2CPPTests ‑ _testInstanceMemberInStaticMethod_263154
org.eclipse.cdt.core.parser.tests.ast2.AST2TemplateTests ‑ _testAmbiguityResolution_497931
org.eclipse.cdt.core.parser.tests.ast2.AST2Tests ‑ _testBug210019_nestedDesignatedInitializers

♻️ This comment has been updated with latest results.

[jonahgraham]

This PR makes the number of tests go up, but the number of runs go down. I still would like to understand why.

[jonahgraham]

This PR makes the number of tests go up, but the number of runs go down. I still would like to understand why.

Its because I missed @Test on all the tests in one class. Update on the way. Still checking that there are no other anomalies.

[jonahgraham]

The failing tests in the JUnit3 format have now been updated to be marked as @Disabled which accounts for the last part of the difference between running main and this commit in Eclipse with Junit5 runner. But you need to use the workaround described here or else inner classes are skipped eclipse-jdt/eclipse.jdt.ui#2591 (comment)

[jonahgraham]

This one is ready to go, the total number of tests run within Eclipse is the same before and after this change and Jenkins reports the same (before after)

I don't know why GitHub reports different numbers.

[i-garrison]

This one is ready to go, the total number of tests run within Eclipse is the same before and after this change and Jenkins reports the same (before after)

I don't know why GitHub reports different numbers.

Assuming you are looking at two test summary lines in Test Results of this PR, I cannot find any reason for GitHub to report different numbers. I looked into logs and found a few differences between job output on your branch vs job output on merge branch. Could be flaky test or infra issue, one example of failed test on your branch junit5-SemanticTestBase https://github.com/eclipse-cdt/cdt/actions/runs/18987447595/job/54234004968?pr=1378

org.eclipse.cdt.internal.index.tests.IndexIncludeTest.testUnresolvedInclude -- Time elapsed: 0.029 s <<< FAILURE!
junit.framework.AssertionFailedError: Can't find /home/runner/work/cdt/cdt/core/org.eclipse.cdt.core.tests/target/work/data/IndexTest_testUnresolvedInclude_1761953146259/source20070214.cpp
        at junit.framework.Assert.fail(Assert.java:57)
        at junit.framework.Assert.assertTrue(Assert.java:22)
        at junit.framework.TestCase.assertTrue(TestCase.java:192)
        at org.eclipse.cdt.internal.index.tests.IndexIncludeTest.getIndexFile(IndexIncludeTest.java:150)
        at org.eclipse.cdt.internal.index.tests.IndexIncludeTest.testUnresolvedInclude(IndexIncludeTest.java:230)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at junit.framework.TestCase.runTest(TestCase.java:177)
        at junit.framework.TestCase.runBare(TestCase.java:142)
        at org.eclipse.cdt.core.testplugin.util.BaseTestCase.runBare(BaseTestCase.java:183)
        at junit.framework.TestResult$1.protect(TestResult.java:122)
        at junit.framework.TestResult.runProtected(TestResult.java:142)
        at junit.framework.TestResult.run(TestResult.java:125)
        at junit.framework.TestCase.run(TestCase.java:130)
        at org.eclipse.cdt.core.testplugin.util.BaseTestCase.run(BaseTestCase.java:192)
        at junit.framework.TestSuite.runTest(TestSuite.java:241)
        at junit.framework.TestSuite.run(TestSuite.java:236)

Running org.eclipse.cdt.internal.index.tests.IndexIncludeTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.028 s -- in org.eclipse.cdt.internal.index.tests.IndexIncludeTest
org.eclipse.cdt.internal.index.tests.IndexIncludeTest.testUnresolvedInclude -- Time elapsed: 0.028 s

Results:

Flakes:
org.eclipse.cdt.internal.index.tests.IndexIncludeTest.testUnresolvedInclude
  Run 1: IndexIncludeTest>BaseTestCase.run:192->BaseTestCase.runBare:183->testUnresolvedInclude:230->getIndexFile:150 Can't find /home/runner/work/cdt/cdt/core/org.eclipse.cdt.core.tests/target/work/data/IndexTest_testUnresolvedInclude_1761953146259/source20070214.cpp
  Run 2: PASS


Tests run: 6613, Failures: 0, Errors: 0, Skipped: 6, Flakes: 1

This test passes on merge branch https://github.com/eclipse-cdt/cdt/actions/runs/18987447396/job/54234004263?pr=1378 confirming it is likely just a flaky test.

If I just compare Tests run: lines from both logs (and ignore test times) I find the only difference is this flaky test, and total numbers of tests exactly match otherwise.

[jonahgraham]

If I just compare Tests run: lines from both logs (and ignore test times) I find the only difference is this flaky test, and total numbers of tests exactly match otherwise.

Thanks for doing this extra investigation. This PR is now ready.

I don't know why #1378 (comment) reports as it does, but I am going to have to leave that for another time. Perhaps by the time I finish all the JUnit5 migrations I will better understand.