Managing shared C++ dependencies between two pybind11 projects

[K20shores]

I have a dependency issue between two libraries that my team maintains. Both libraries are wrapped with pybind11 and exposed to Python. I don't know if this is the correct forum for this, but given my question, it may be the most appropriate.

• Mechanism Configuration - This library provides data structures to create chemical mechanism data, which can be used to study atmospheric chemistry
  • Python wrapper
• Musica This library wraps our ODE solver, photolysis module, and provides a C interface for Fortran projects to link to; we wrap the same C interface with pybind11
  • Python wrapper

The Problem

Musica includes mechanism configuration as a C++ dependency with fetch content. This works fine for our C++ and Fortran targets.

However, the Musica Python package also depends on the Mechanism Configuration Python package, which we declare as a dependency in our pyproject.toml
Mechanism Configuration is therefore installed from pypi and includes its own pybind11 .so file and static library.

At the same time, Musica's Python package builds its own .so, statically links Mechanism Configuration from FetchContent, and installs the .a file:

/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/_musica.cpython-312-darwin.so
/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/_musica_editable.pth
/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/_musica_editable.py
/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/lib/libmechanism_configuration.a
/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/lib/libmusica.a
/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/lib/libyaml-cpp.a
/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/musica-0.10.1.dist-info/*
/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/musica/_version.py

The mechanism configuration library has its own static library.

/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/lib/libmechanism_configuration.a
/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/mechanism_configuration-0.1.1.dist-info/*
/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/mechanism_configuration.cpython-312-darwin.so
/opt/homebrew/anaconda3/envs/musica/lib/python3.12/site-packages/mechanism_configuration/_version.py

This results in two separate builds of the Mechanism Configuration library being linked into the same Python process.
I think this leads to an ABI mismatch on Linux, based on one of our failing CI tests on Ubuntu.

My Question

What is the best practice for managing shared C++ dependencies across multiple pybind11-based Python packages?

We would like to:

• Keep Mechanism Configuration as a standalone Python package installable via PyPI.
• Avoid rebuilding it in Musica if possible, but Musica still needs to link against the C++ library somehow.

Ensure that both _musica.so and mechanism_configuration.so refer to a single, consistent implementation of Mechanism Configuration.

Do we need to build Mechanism Configuration as a shared library and link both bindings against it? If so, how can we structure this correctly in terms of CMake and packaging?

Any advice or examples of similar setups would be much appreciated.

[K20shores]

Update: We are likely going to move the mechanism configuration python code into the musica library, so this won't be an issue. However, I'd still love to know an answer and I'm sure I'm not the first who has had this issue but I haven't been able to find any similar situations.