openmpi osc_ucx_component error

[SLgramps]

Thank you for taking the time to submit an issue!

Background information

I have been building/running some simple parallel code (specifically, from https://github.com/modern-fortran, any of the example projects such as tsunami or weather-buoys). When using openmpi v4.1.5 (Fedora 39), everything built/ran as expected. Upon upgrading to Fedora 40, openmpi was also upgraded to v5.0.2 and the code, though running correctly and finishing, issues copious amounts of osc_ucx_component.c:369 errors as shown in the attached screenshot. Downgrading openmpi back to v4.1.5 eliminates the errors.

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v4.1.5 (works correctly)
v5.0.2 (issues osc_ucx_component errors)

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

packages installed from fedora repositories, OpenCoarrays downloaded from opencoarrays.org and built locally

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

• Operating system/version: linux kernel 6.8.7 and several predecessors
• Computer hardware: Intel i7 12700KF, Asus Z-690 motherboard
• Network type:

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

Using v5.0.2, code runs but issues copious osc_ucx_component.c:369 errors as shown in the attached screenshot
Using v4.1.5, code runs without issuing errors
Does not seem to affected by version of linux, gfortran, or OpenCoarrays used

0. install gfortran from fedora repositories, build OpenCoarrays from opencoarrays.org
1. git clone, for instance, https://github.com/modern-fortran/weather-bouys (or tsunami, ch08) and build using included makefiles with either v4.1.5 or v5.02 openmpi
2. run parallel versions of code using cafrun -n 2 (or other number of cores)

(Attached screenshot is from running tsunami/ch08. Code from weather-buoys also exhibits this behavior.)

Note: If you include verbatim output (or a code block), please use a GitHub Markdown code block like below:

shell$ mpirun -n 2 ./hello_world

[devreal]

@janjust Looks like this might be triggered by failed initialization of ucp. The message is definitely misleading.

[janjust]

yeah look that way, @SLgramps , I'm guessing this is a local machine, no network?

[janjust]

as a simple work around just disabling the component will get your around this error -mca osc ^ucx

[SLgramps]

Yes, this was a local machine. Your suggested workaround eliminates the error. Thanks for looking into this.

[yizeyi18]

I'm also facing this issue, and have more information could provide:
This output comes from ompi/mca/osc/ucx/osc_ucx_component.c:369, which inferrs component_set_priority() implemented in the same file returns a non-success value:

ompi/ompi/mca/osc/ucx/osc_ucx_component.c

Lines 366 to 371 in 45258b6

	if (mca_osc_ucx_component.priority_is_set == false) {
	ret = component_set_priority();
	if (OMPI_SUCCESS != ret) {
	OSC_UCX_ERROR("OSC UCX component priority set inside component query failed \n ");
	return ret;
	}

And in the component_set_priority(), there are 2 if that returns fail:

ompi/ompi/mca/osc/ucx/osc_ucx_component.c

Lines 301 to 317 in 45258b6

	if (mca_osc_ucx_component.wpool == NULL) {
	mca_osc_ucx_component.wpool = opal_common_ucx_wpool_allocate();
	}
	if (mca_osc_ucx_component.wpool->ucp_ctx == NULL) {
	ret = ucp_context_init(mca_osc_ucx_component.enable_mpi_threads, ompi_proc_world_size());
	if (OMPI_ERROR == ret) {
	return OMPI_ERR_NOT_AVAILABLE;
	}
	}
	support_level = opal_common_ucx_support_level(mca_osc_ucx_component.wpool->ucp_ctx);
	if (OPAL_COMMON_UCX_SUPPORT_NONE == support_level) {
	ucp_cleanup(mca_osc_ucx_component.wpool->ucp_ctx);
	mca_osc_ucx_component.wpool->ucp_ctx = NULL;
	return OMPI_ERR_NOT_AVAILABLE;
	}

Before the second one, ucp_context_init would be called, which would do OSC_UCX_VERBOSE once failed; adjusting environ OSC_UCX_VERBOSE and UCX_LOG_LEVEL both made no excess output other than line 369, inferring the routine fails on the first if.

Before the first if, opal_common_ucx_wpool_allocate() was called to init mca_osc_ucx_component.wpool, which subroutine is implemented in opal/mca/common/ucx/common_ucx_wpool.c:

ompi/opal/mca/common/ucx/common_ucx_wpool.c

Lines 129 to 135 in 45258b6

	OPAL_DECLSPEC opal_common_ucx_wpool_t *opal_common_ucx_wpool_allocate(void)
	{
	opal_common_ucx_wpool_t *ptr = calloc(1, sizeof(opal_common_ucx_wpool_t));
	ptr->refcnt = 0;
	return ptr;
	}

in which only calloc but no more work was done; this would result as mca_osc_ucx_component.wpool->ucp_ctx == NULL, as the first if judges. I think there lacks an opal_common_ucx_wpool_init() in ompi/mca/osc/ucx/osc_ucx_component.c:303. EDIT: nope, no opal_common_ucx_wpool_init() needed in component_set_priority(). THe problem lies in ucp_context_init().