Asynchronous tests are broken

[acarioni]

Environment
macos monterey 12.1
haxe 4.2.4
utest 1.13.2

Asynchronous tests don’t work when they happen to run on threads without event loop. The problem lies in the object haxe.Timer, used internally by utest, which throws the exception NoEventLoopException when it is called from a thread without an event loop. The error affects all the threaded targets (I’ve tried with java, cs and python). You may find useful this discussion on the same problem.

Is it possible to use another implementation of timer, for example haxe-concurrent, instead of that of the standard library?

The issue can be reproduced by the following code

package foo;

class TestCase extends utest.Test {

  function testFoo(async: utest.Async) {
    veryComplexFunctionToBeTested(() -> {
      utest.Assert.pass();
      async.done();
    });
  }

  function testBar(async: utest.Async) {
    veryComplexFunctionToBeTested(() -> {
      utest.Assert.pass();
      async.done();
    });
  }

  function veryComplexFunctionToBeTested(callback: ()->Void) {
    sys.thread.Thread.create(callback);
  }
}

The reported error is

results: SOME TESTS FAILURES (success: false)
foo.TestCase
  testBar: ERROR E
    Event loop is not available. Refer to sys.thread.Thread.runWithEventLoop.
Called from sys.thread.Thread$Thread_Impl_.get_events (/Users/foobar/haxe/versions/4.2.4/std/java/_std/sys/thread/Thread.hx line 74)
Called from haxe.Timer.new (/Users/foobar/haxe/versions/4.2.4/std/haxe/Timer.hx line 76)
Called from haxe.Timer.delay (/Users/foobar/haxe/versions/4.2.4/std/haxe/Timer.hx line 141)
Called from foo.TestCase$Closure___initializeUtest___1.invoke (foo/TestCase.hx line 12)
Called from foo.TestCase$Closure___initializeUtest___1.invoke (foo/TestCase.hx line -1)

[RealyUniqueName]

I agree this should work. The issue is not only the Timer implementation but also utest does not take thread safety into account at all.
As a temporary solution you could shadow haxe.Timer module with your own implementation, and maybe it would be enough for your case. But it's also possible you would face random test failures due to races.

[RealyUniqueName]

Another solution is to modify tests like this:

  function testBar(async: utest.Async) {
    var thread = Thread.current();
    veryComplexFunctionToBeTested(() -> {
      thread.events.run(() -> {
        utest.Assert.pass();
        async.done();
      });
    });
  }

[acarioni]

Thank you for your comments.
Currently I have resorted to a solution similar to that outlined in your last comment.
Regarding your first comment, how can I shadow a standard library class with my implementation?

[RealyUniqueName]

Just create a class with the same package and name in your project.

[ravendyne]

Maybe not related to this, but also probably is, so I didn't want to open a new issue: async in setupClass blocks even after async.done() has been called, but only if it's been called from a Timer or Thread callback.

For example, running this test:

class TestSmoke extends utest.Test {

    function setupClass( async : utest.Async ) {
        trace('setup');

        sys.thread.Thread.current().events.run(()->{
            async.done();
            trace('delay done');
        });
        // haxe.Timer.delay(()->{
        //     async.done();
        //     trace('delay done');
        // },50);
    }

    function teardownClass() {
        trace('teardown');
    }

    function testSmoke() {
        utest.Assert.fail();
    }
}

will not fail as expected, instead it will just hang forever:

src/test/TestSmoke.hx:12: setup
src/test/TestSmoke.hx:25: teardown
src/test/TestSmoke.hx:16: delay done

Thread.current.events and Timer.delay give the same result (test execution hangs) on macro, neko and hl targets (haven't tried others).