TCP causes performance penalty on single-node runs

[jsquyres]

At the OMPI dev meeting in June 2015, @hjelmn mentioned that on OS X in particular, he sees that when he runs on his Mac, the TCP BTL is being disabled via vader exclusivity, but it still somehow doesn't decrement the opal progress count, so libevent still gets polled frequently.

This seems like a bug -- it causes a large performance penalty on OS X (it seems like the performance hit is less on Linux).

We all agree that this sounds like a bug -- because vader has a higher exclusivity, it should be disabling the TCP BTL.

Additionally, when we move to an async modex, it may be a little slow on the first connection for the TCP BTL (because libevent will be polled less frequently) -- but that's ok.

[bosilca]

This issue is more subtle than this ticket make it sound. The exclusivity works locally, so a process might decide that TCP will not be used to communicate with its peers. However, as the peers decision is not known locally, TCP BTL cannot simply be discarded. We need at least the listen socket setup and checked on a regular basis.

We can solve this either by making assumptions about decisions taken by the peers, or by adding a thread in the TCP BTL that will handle the listen socket. The first approach is sketchy and unreliable, the second requires some coding, and some extra synchronizations in the non multi-threaded case.

[jsquyres]

@bosilca @hjelmn I think you guys talked about this a bit more at the June 2015 dev meeting today -- did you conclude something different than what George said above?

[jsquyres]

@hjelmn @bosilca Did this issue get resolved?