Cuda-aware OpenMPI fails on Jetson K1

[dribbroc]

As stated in
https://devtalk.nvidia.com/default/topic/790095/embedded-systems/cuda-aware-openmpi-fails-on-jetson-k1/
the cuda-aware mpi fails after finishing the programm correctly with cuMemHostRegister errors.

I dunno wether its a jetson, cuda or ompi related bug.
The bug does not occur on a usual x86 desktop ubuntu pc.

[rolfv]

I have not tried CUDA-aware on Jetson K1. However, I think I know what may be happening. CUDA-aware registers some host allocated buffers with CUDA using cuMemHostRegister driver API (which is similar to cudaHostRegister call from CUDA runtime library). From the CUDA 6.5 Release Notes document, I find this note:

http://docs.nvidia.com/cuda/cuda-toolkit-release-notes/#armv7-linux-general-cuda-known-issues

7.1. Linux on ARMv7 Specific Issues
•Mapping host memory allocated outside of CUDA to device memory is not allowed on ARM; because of this, cudaHostRegister() is not supported by the CUDA driver on ARM platforms. If required, cudaHostAlloc() with the flag cudaHostAllocMapped can be used to allocate device-mapped host-accessible memory.

So the library is getting error 801 which is CUDA_ERROR_NOT SUPPORTED.

The registering of the host memory with CUDA library is an optimization that can potentially speed up copying memory form/to host and GPU memory. Need to figure out how this can be fixed.

[jsquyres]

@rolfv Is this a factor for the v1.8 series, or just master?

[rolfv]

This is a factor for both v1.8 and master.

[dribbroc]

Just tried out the 1.8.5 release and the bug is gone on my jetson board.
Thank you for that!

Unfortunately a seg fault now occurs every time i execute a mpi programm (that uses no cuda at all) on x86 ubuntu 12.04 with cuda-mpi support enabled.

Precisely the call to cuMemHostUnregister segfaults, because the variable checkmem is zero.
https://github.com/open-mpi/ompi-release/blob/v1.8/ompi/mca/common/cuda/common_cuda.c#L771
If is skip the complete section until https://github.com/open-mpi/ompi-release/blob/v1.8/ompi/mca/common/cuda/common_cuda.c#L842 (when checkmem is zero) everything runs smooth.

[rolfv]

Glad things work although I cannot take credit for that. Are you using CUDA 7.0 now instead of CUDA 6.5? In terms of the other issue, yes, I just ran into that also. I have already put a fix back and it will be in Open MPI 1.8.6. If you have time, could you try 1.8.6rc1 and verify that fixes things for you.

http://www.open-mpi.org/software/ompi/v1.8/downloads/openmpi-1.8.6rc1.tar.gz

[dribbroc]

Ok i found your commit
b3e4ae7
which might fix the above issue :)

[dribbroc]

Duh, should refresh the page before posting new messages.

No, I'm still using cuda 6.5 on the jetson board.
As far as i know, there does not exist any cuda 7 release for jetson tegra k1.
At least https://developer.nvidia.com/linux-tegra does not contain any cuda 7 release.

On the x86 side, i testet cuda 6.5 and 7.0, just to be sure it was not related to a 7.0 only error.
Both failed with 1.8.5

I'll give 1.8.6rc1 a try and report back the result.

Cheers
Dirk

[dribbroc]

Your patch seems to fix the x86 issues with cuda not initialized at all.

Thanks in advance for your quick help.

[dribbroc]

Just to sum things up:
The 801 error arises on the jetson with 2.0 but not with 1.8.5.
I will check the output of openmpi 1.8.8, once the jetson cluster is free from users.