Tox4 sometimes hangs in CI after --notest run

[ziima]

I encountered a strange behavior of tox sometimes hanging in CI. It only affects projects which has separate tox --parallel all --notest prior to the actual tox run to speed things up.

I tried to debug it, but I found following output in affected runs:

$ tox -vvv
ROOT: 98 D setup logging to NOTSET on pid 167 [tox/report.py:221]
ERROR: Job failed: execution took longer than 1h0m0s seconds

I'm unable to trigger the problem on purpose, it seems to be in portion random. It also seems to affect only a subset of our projects.

It seems that running the tox right away without the tox --parallel all --notest phase doesn't trigger the issue.

[gaborbernat]

The _TOX_SHOW_THREAD environment variable can be used to print alive threads when tox exists (useful to debug when tox hangs because of some non-finished thread) and also now prints the pid of the local subprocess when reporting the outcome of an execution

Can you use this feature to get a stack trace, by putting the invocation into a timeout?

[ziima]

Tried that, it still didn't produce any useful output:

$ _TOX_SHOW_THREAD=1 timeout 600 tox -vvv
ROOT: 100 D setup logging to NOTSET on pid 142 [tox/report.py:221]
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1

It seems the tox got stuck just after it started.

[ziima]

Since the report we have installed a new Gitlab CI machine with a new set of runners configured differently. The problem still occurs on the new CI runners. It was never encountered on the desktop.

[ziima]

See #3254 for further details