Skip to content

developers: Doc how to build against external PMIx/PRTE #12946

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

developers: Doc how to build against external PMIx/PRTE #12946

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions docs/developers/building-open-mpi.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,87 @@
Building Open MPI
=================

General
-------

Once you have run ``autogen.pl`` successfully, you can configure and
build Open MPI just like end users do with official distribution Open
MPI tarballs.

See the :doc:`general "Install Open MPI" documentation for more
details. </installing-open-mpi/index>`

Building Against External OpenPMIx / PRRTE
------------------------------------------

One thing that developers and/or packagers may need to do is to build
Open MPI against an external OpenPMIx and/or PRRTE source tree (i.e.,
an OpenPMIx and/or PRRTE installation that was not built from the
embedded copies inside the Open MPI source tree / Git submodules).

With regards to :doc:`Open MPI's required dependent libraries
</installing-open-mpi/required-support-libraries>` (Hwloc, Libevent,
OpenPMIx, and PRRTE), it generally is simplest to build Open MPI in
one of two ways:

#. Build and use all the **internal** copies of Open MPI's required
dependent libraries.

* Specifically: use the Hwloc, Libevent, OpenPMIx, and PRRTE source
trees that are bundled in with Open MPI's source code.

#. Build and use all **external** copies of Open MPI's required
dependent libraries.

* Specifically: ignore the Hwloc, Libevent, OpenPMIx, and PRRTE source
trees that are bundled in with Open MPI's source code, and,
instead, compile and link Open MPI against already-installed
versions of these libraries.

Other variations are possible, but can get tricky and complicated with
subtle linker consequences, and are therefore not recommended.

Some facts that are relevant to know when building against an external
OpenPMIx / PRRTE:

1. Open MPI, OpenPMIx, and PRRTE must all be built against the
**same** installation of Hwloc and Libevent. Meaning:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not fully correct - PRRTE doesn't need to have the same.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can (will) run into super weird errors if PMIx, PRTE, and/or Open MPI are using different libevents and/or hwlocs at run-time. They should be completely orthogonal, but we have definitely seen cases where they are not. I'm ok documenting it as a must even if there are some cases where it actually does work to -- for example -- have PMIx run-time link against hwloc version X and PRTE run-time link against hwloc version Y.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is where you are getting into trouble. Whatever PMIx PRRTE is using, that PMIx and PRRTE must use the same hwloc and libevent.

However, as I said elsewhere, there is no requirement that PRRTE and OMPI use the same PMIx. So there is no transitive property involved here - there is a complete airbreak between PRRTE and OMPI.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, so something like:

  1. Open MPI and the OpenPMIx library Open MPI links against must be built against the same installation of hwloc and Libevent. It is not required that the runtime and Open MPI be built with the same version of PMIx, but the same hwloc/libevent linking rules also apply to PRRTE and its OPenPMIx library.

* Assumedly the external OpenPMIx and PRRTE were built against
external Hwloc and Libevent. Open MPI **must** compile and link
against the **same** Hwloc and Libevent that the external
OpenPMIx and PRRTE were built against.

.. admonition:: Critical
:class: Danger

Open MPI, OpenPMIx, and PRRTE must all use the same Hwloc and
Libevent libraries at run time (e.g., they must all resolve to
the same run-time loadable libraries at run time).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not totally correct - PRRTE doesn't have to use the same as OMPI never loads "libprrte", so there is no potential for confusion.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By transitive property, though, isn't this true? OMPI must use the same hwloc + libevent as PMIX, and PRTE must use the same hwloc + libevent as PMIx, so therefore don't they all have to be the same?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can have two versions of PMIx installed. I don't think this is super common for a developer, but given how Slurm is packaged, may be more common there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PBS does it too now - in fact, they include PRRTE and PMIx, so not uncommon to see multiple installs there.


.. important:: This statement applies regardless of whether
Open MPI -- and/or the other libraries -- are
built as static or dynamically-loadable
libraries.

* Unless you really know what you are doing, this usually means
building and installing Open MPI against the same installation
tree(s) of Hwloc and Libevent that OpenPMIx and PRRTE used to
build themselves.

For example, consider an environment where you install Hwloc,
Libevent, OpenPMIx, and PRRTE via the operating system's package
manager. Assuming that the package-manager installs of OpenPMIx
and PRRTE were built against the package-manager-provider Hwloc
and Libevent, then Open MPI will *also* need to be built against
the package-manager-provided Hwloc and Libevent. To build Open
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, true for PMIx but not for PRRTE

MPI this way, you may need to install the package manager's
"developer" Hwloc, Libevent, OpenPMIx, and/or PRRTE packages.

1. Open MPI and PRRTE must be built against the **same** installation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, not true - PMIx is designed to handle cross-version messaging. Only issue would be ensuring that the PMIx being used by PRRTE is new enough to support any OMPI-used features. Given the minimums we set, that shouldn't be a problem.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, fair point here. I was really thinking about Hwloc and Libevent when I was writing this bullet. I'll update.

of OpenPMIx.

.. important:: Similar to how OpenPMIx, PRRTE, and Open MPI, must
be built against the same Hwloc and Libevent, PRRTE
and Open MPI must be built against the same
OpenPMIx.