Skip to content

Python Hierarchy Implementation #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 130 commits into from
Jul 7, 2022
Merged

Conversation

taguhiM
Copy link
Collaborator

@taguhiM taguhiM commented Jun 11, 2022

Hi.
In this PR we present the implementation of the python hierarchy.
Note that, the states and hypers now are stored in std::vectors for the sake of generalization; there is an internal 'switcher' implemented for the user to change between different hierarchy implementations in python dynamically (in runtime).

*The documentation and styling of the code are not yet done.

We would like to mention that we did try using the 'make opaque' technique to pass by reference between c++ and python, but that surprisingly didn't improve the performance.

To build the library:

$ mkdir build
$ ./make_.sh

To run an example using the python implementations:
$ python3 docs/examples/test_run.py

We would like to hear your comments to improve the code.

@mberaha
Copy link
Contributor

mberaha commented Jul 4, 2022

The Reame.md file should be updated with the steps needed to build everything.

Running ./build_pybmix.sh returns ./build_pybmix.sh: line 4: cd: build/: No such file or directory

after a mkdir build and ./build_pybmix.sh, the program ends with

WARNING: --write-unchanged-files/-W implies -w.
lib2to3.main: Output in '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto' will mirror the input directory '/Users/marioberaha/dev/tests/pybmix/../pybmix' layout.
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Can't open /Users/marioberaha/dev/tests/pybmix/../pybmix/proto: [Errno 2] No such file or directory: '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto'
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't open /Users/marioberaha/dev/tests/pybmix/../pybmix/proto: [Errno 2] No such file or directory: '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto'
make[3]: *** [CMakeFiles/two_to_three] Error 1
make[2]: *** [CMakeFiles/two_to_three.dir/all] Error 2
make[1]: *** [CMakeFiles/two_to_three.dir/rule] Error 2
make: *** [two_to_three] Error 2

Also, it would be nice to have a notebook comparing the C++ implementation of the NNIG/laplace hierarchies with the Python ones, to make sure that the code is correct (same density estimate) and show the performance loss

@taguhiM
Copy link
Collaborator Author

taguhiM commented Jul 4, 2022

We've updated the README (we tried to install from 0 on a virtual machine following the updated README, so we are sure it works), please try again.

1) Clone the repository
```shell
git clone --recurse-submodule [email protected]:bayesmix-dev/pybmix.git

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the installation should be part of the main Readme

@mberaha
Copy link
Contributor

mberaha commented Jul 4, 2022

We've updated the README (we tried to install from 0 on a virtual machine following the updated README, so we are sure it works), please try again.

I still get the same exact error. Removing the last ../ in the definition of SCRIPT_DIR in convert_protos.sh does the trick for me

@taguhiM
Copy link
Collaborator Author

taguhiM commented Jul 4, 2022

We've updated the README (we tried to install from 0 on a virtual machine following the updated README, so we are sure it works), please try again.

I still get the same exact error. Removing the last ../ in the definition of SCRIPT_DIR in convert_protos.sh does the trick for me

Do you think this is OS dependent error? Because it works fine for us, also on VM's, but we're all using Linux.

@taguhiM
Copy link
Collaborator Author

taguhiM commented Jul 4, 2022

We've updated the README (we tried to install from 0 on a virtual machine following the updated README, so we are sure it works), please try again.

I still get the same exact error. Removing the last ../ in the definition of SCRIPT_DIR in convert_protos.sh does the trick for me

Do you think this is OS dependent error? Because it works fine for us, also on VM's, but we're all using Linux.

Seems its working without that also for us. We check better and update it.

@mberaha
Copy link
Contributor

mberaha commented Jul 4, 2022

Also, installation is not clear for me at the moment.
Since python3 -m pip install . isn't working (new build pipeline), how does the notebook know how to import pybmix since the folder pybmix is not visible to the notebook?

E.g.,

cd docs/examples
jupyter notebook

and then running estimate_py_density.ipynb fails at the first cell with

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Input In [1], in <cell line: 4>()
      2 import matplotlib.pyplot as plt
      3 import scipy.stats as ss
----> 4 from pybmix.core.mixing import PythonMixing
      5 from pybmix.core.hierarchy import PythonHierarchy
      6 from pybmix.core.mixture_model import MixtureModel

ModuleNotFoundError: No module named 'pybmix'

@taguhiM
Copy link
Collaborator Author

taguhiM commented Jul 4, 2022

We've updated the README (we tried to install from 0 on a virtual machine following the updated README, so we are sure it works), please try again.

I still get the same exact error. Removing the last ../ in the definition of SCRIPT_DIR in convert_protos.sh does the trick for me

Were you inside the repository directory when that error occurred?

@EnricoPaglia
Copy link
Contributor

We've updated the README (we tried to install from 0 on a virtual machine following the updated README, so we are sure it works), please try again.

I still get the same exact error. Removing the last ../ in the definition of SCRIPT_DIR in convert_protos.sh does the trick for me

Do you think this is OS dependent error? Because it works fine for us, also on VM's, but we're all using Linux.

Seems its working without that also for us. We check better and update it.

In the end it did not work.

@EnricoPaglia
Copy link
Contributor

Also, installation is not clear for me at the moment. Since python3 -m pip install . isn't working (new build pipeline), how does the notebook know how to import pybmix since the folder pybmix is not visible to the notebook?

E.g.,

cd docs/examples
jupyter notebook

and then running estimate_py_density.ipynb fails at the first cell with

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Input In [1], in <cell line: 4>()
      2 import matplotlib.pyplot as plt
      3 import scipy.stats as ss
----> 4 from pybmix.core.mixing import PythonMixing
      5 from pybmix.core.hierarchy import PythonHierarchy
      6 from pybmix.core.mixture_model import MixtureModel

ModuleNotFoundError: No module named 'pybmix'

To run the notebook we included the lines

import sys
sys.path.append("../../")

before importing anything in the pybmix folder. We did not push this since it is not present in the other notebooks and when the pip install command will work it will not be needed anymore. Should we add it? (And in case should we add it also to the other notebooks?)

@mberaha
Copy link
Contributor

mberaha commented Jul 5, 2022

We've updated the README (we tried to install from 0 on a virtual machine following the updated README, so we are sure it works), please try again.

I still get the same exact error. Removing the last ../ in the definition of SCRIPT_DIR in convert_protos.sh does the trick for me

Were you inside the repository directory when that error occurred?

I'm at the root folder of pybmix, I added an echo "$SCRIPT_DIR" in the convert_proto.sh

pwd
> /Users/marioberaha/dev/tests/pybmix

./convert_proto.sh 
> /Users/marioberaha/dev/tests/pybmix/..
WARNING: --write-unchanged-files/-W implies -w.
lib2to3.main: Output in '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto' will mirror the input directory '/Users/marioberaha/dev/tests/pybmix/../pybmix' layout.
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Can't open /Users/marioberaha/dev/tests/pybmix/../pybmix/proto: [Errno 2] No such file or directory: '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto'
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't open /Users/marioberaha/dev/tests/pybmix/../pybmix/proto: [Errno 2] No such file or directory: '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto'

the first line of the output comes from echo "$SCRIPT_DIR" --> it is clear that is pointing to the parent folder wrt to pybmix root

@EnricoPaglia
Copy link
Contributor

We've updated the README (we tried to install from 0 on a virtual machine following the updated README, so we are sure it works), please try again.

I still get the same exact error. Removing the last ../ in the definition of SCRIPT_DIR in convert_protos.sh does the trick for me

Were you inside the repository directory when that error occurred?

I'm at the root folder of pybmix, I added an echo "$SCRIPT_DIR" in the convert_proto.sh

pwd
> /Users/marioberaha/dev/tests/pybmix

./convert_proto.sh 
> /Users/marioberaha/dev/tests/pybmix/..
WARNING: --write-unchanged-files/-W implies -w.
lib2to3.main: Output in '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto' will mirror the input directory '/Users/marioberaha/dev/tests/pybmix/../pybmix' layout.
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Can't open /Users/marioberaha/dev/tests/pybmix/../pybmix/proto: [Errno 2] No such file or directory: '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto'
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't open /Users/marioberaha/dev/tests/pybmix/../pybmix/proto: [Errno 2] No such file or directory: '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto'

the first line of the output comes from echo "$SCRIPT_DIR" --> it is clear that is pointing to the parent folder wrt to pybmix root

But when conver_proto.sh is called by build_pybmix.sh, the latter goes to the folder build before calling it so it should echo /Users/marioberaha/dev/tests/pybmix/build/... If instead one runs convert_proto.sh manually the last ../ should not be needed.

@mberaha
Copy link
Contributor

mberaha commented Jul 5, 2022

But when conver_proto.sh is called by build_pybmix.sh, the latter goes to the folder build before calling it so it should echo /Users/marioberaha/dev/tests/pybmix/build/... If instead one runs convert_proto.sh manually the last ../ should not be needed.

This is the full output when calling ./build_pybmix.sh

pwd
> /Users/marioberaha/dev/tests/pybmix

./build_pybmix.sh 
CMAKE_MODULE_PATH /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp
-- Using math TBB
-- Building Intel TBB: /usr/bin/make -j8 stdver=c++17 compiler=clang tbb_build_dir=/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/lib/math/lib/tbb tbb_build_prefix=tbb
FINDPROTOBUF:CMAKE Protobuf_FOUND 0
FINDPROTOBUF:CMAKE PROTOBUF_FOUND 1
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_id.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_id.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_id.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_params.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_params.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_params.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_state.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_state.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_state.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/distribution.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/distribution.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/distribution.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_id.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/hierarchy_id.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/hierarchy_id.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_prior.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/hierarchy_prior.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/hierarchy_prior.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/ls_state.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/ls_state.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/ls_state.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/matrix.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/matrix.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/matrix.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_id.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_id.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_id.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_prior.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_prior.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_prior.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_state.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_state.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_state.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/semihdp.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/semihdp.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/semihdp.pb.cc
-- pybind11 v2.10.0 dev1
ProtoFiles: /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_id.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_params.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_state.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/distribution.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_id.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_prior.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/ls_state.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/matrix.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_id.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_prior.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_state.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/semihdp.proto
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_id.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/algorithm_id_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_params.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/algorithm_params_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_state.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/algorithm_state_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/distribution.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/distribution_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_id.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/hierarchy_id_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_prior.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/hierarchy_prior_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/ls_state.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/ls_state_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/matrix.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/matrix_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_id.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/mixing_id_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_prior.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/mixing_prior_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_state.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/mixing_state_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/semihdp.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/semihdp_pb2.py
CMake Warning (dev) at CMakeLists.txt:46 (find_package):
  Ignoring EXACT since no version is requested.
This warning is for project developers.  Use -Wno-dev to suppress it.

INCLUDE_PATHS
-- Configuring done
CMake Warning at lib/pybind11/tools/pybind11Tools.cmake:173 (add_library):
  Cannot generate a safe runtime search path for target pybmixcpp because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libomp.dylib] in /opt/homebrew/Caskroom/miniconda/base/envs/scipy-dev/lib may be hidden by files in:
      /opt/homebrew/lib

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  CMakeLists.txt:19 (pybind11_add_module)


-- Generating done
-- Build files have been written to: /Users/marioberaha/dev/tests/pybmix/build
Consolidate compiler generated dependencies of target bayesmix
[ 79%] Built target bayesmix
[ 80%] Built target bayesmixlib
Consolidate compiler generated dependencies of target pybmixcpp
[100%] Built target pybmixcpp
[100%] Built target generate_protos
[100%] Built target generate_protos
/Users/marioberaha/dev/tests/pybmix/..
WARNING: --write-unchanged-files/-W implies -w.
lib2to3.main: Output in '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto' will mirror the input directory '/Users/marioberaha/dev/tests/pybmix/../pybmix' layout.
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Can't open /Users/marioberaha/dev/tests/pybmix/../pybmix/proto: [Errno 2] No such file or directory: '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto'
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't open /Users/marioberaha/dev/tests/pybmix/../pybmix/proto: [Errno 2] No such file or directory: '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto'
make[3]: *** [CMakeFiles/two_to_three] Error 1
make[2]: *** [CMakeFiles/two_to_three.dir/all] Error 2
make[1]: *** [CMakeFiles/two_to_three.dir/rule] Error 2
make: *** [two_to_three] Error 2

It seems that SCRIPT_DIR = /Users/marioberaha/dev/tests/pybmix/..

@EnricoPaglia
Copy link
Contributor

But when conver_proto.sh is called by build_pybmix.sh, the latter goes to the folder build before calling it so it should echo /Users/marioberaha/dev/tests/pybmix/build/... If instead one runs convert_proto.sh manually the last ../ should not be needed.

This is the full output when calling ./build_pybmix.sh

pwd
> /Users/marioberaha/dev/tests/pybmix

./build_pybmix.sh 
CMAKE_MODULE_PATH /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp
-- Using math TBB
-- Building Intel TBB: /usr/bin/make -j8 stdver=c++17 compiler=clang tbb_build_dir=/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/lib/math/lib/tbb tbb_build_prefix=tbb
FINDPROTOBUF:CMAKE Protobuf_FOUND 0
FINDPROTOBUF:CMAKE PROTOBUF_FOUND 1
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_id.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_id.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_id.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_params.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_params.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_params.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_state.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_state.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/algorithm_state.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/distribution.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/distribution.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/distribution.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_id.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/hierarchy_id.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/hierarchy_id.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_prior.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/hierarchy_prior.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/hierarchy_prior.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/ls_state.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/ls_state.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/ls_state.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/matrix.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/matrix.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/matrix.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_id.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_id.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_id.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_prior.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_prior.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_prior.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_state.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_state.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/mixing_state.pb.cc
-- protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/semihdp.proto
-- protoc hdr: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/semihdp.pb.h
-- protoc src: /Users/marioberaha/dev/tests/pybmix/build/pybmix/core/pybmixcpp/semihdp.pb.cc
-- pybind11 v2.10.0 dev1
ProtoFiles: /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_id.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_params.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_state.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/distribution.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_id.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_prior.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/ls_state.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/matrix.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_id.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_prior.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_state.proto;/Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/semihdp.proto
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_id.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/algorithm_id_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_params.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/algorithm_params_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/algorithm_state.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/algorithm_state_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/distribution.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/distribution_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_id.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/hierarchy_id_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/hierarchy_prior.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/hierarchy_prior_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/ls_state.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/ls_state_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/matrix.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/matrix_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_id.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/mixing_id_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_prior.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/mixing_prior_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/mixing_state.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/mixing_state_pb2.py
-- GENERATING PYTHON protoc proto(cc): /Users/marioberaha/dev/tests/pybmix/pybmix/core/pybmixcpp/bayesmix/src/proto/semihdp.proto --> /Users/marioberaha/dev/tests/pybmix/pybmix/proto/semihdp_pb2.py
CMake Warning (dev) at CMakeLists.txt:46 (find_package):
  Ignoring EXACT since no version is requested.
This warning is for project developers.  Use -Wno-dev to suppress it.

INCLUDE_PATHS
-- Configuring done
CMake Warning at lib/pybind11/tools/pybind11Tools.cmake:173 (add_library):
  Cannot generate a safe runtime search path for target pybmixcpp because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libomp.dylib] in /opt/homebrew/Caskroom/miniconda/base/envs/scipy-dev/lib may be hidden by files in:
      /opt/homebrew/lib

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  CMakeLists.txt:19 (pybind11_add_module)


-- Generating done
-- Build files have been written to: /Users/marioberaha/dev/tests/pybmix/build
Consolidate compiler generated dependencies of target bayesmix
[ 79%] Built target bayesmix
[ 80%] Built target bayesmixlib
Consolidate compiler generated dependencies of target pybmixcpp
[100%] Built target pybmixcpp
[100%] Built target generate_protos
[100%] Built target generate_protos
/Users/marioberaha/dev/tests/pybmix/..
WARNING: --write-unchanged-files/-W implies -w.
lib2to3.main: Output in '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto' will mirror the input directory '/Users/marioberaha/dev/tests/pybmix/../pybmix' layout.
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Can't open /Users/marioberaha/dev/tests/pybmix/../pybmix/proto: [Errno 2] No such file or directory: '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto'
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't open /Users/marioberaha/dev/tests/pybmix/../pybmix/proto: [Errno 2] No such file or directory: '/Users/marioberaha/dev/tests/pybmix/../pybmix/proto'
make[3]: *** [CMakeFiles/two_to_three] Error 1
make[2]: *** [CMakeFiles/two_to_three.dir/all] Error 2
make[1]: *** [CMakeFiles/two_to_three.dir/rule] Error 2
make: *** [two_to_three] Error 2

It seems that SCRIPT_DIR = /Users/marioberaha/dev/tests/pybmix/..

Does it work if instead we define SCRIPT_DIR as

SCRIPT_DIR=$(pwd)/..

? It works for me

@mberaha
Copy link
Contributor

mberaha commented Jul 5, 2022

yes works for me as well

Copy link
Contributor

@mberaha mberaha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good. Please move the installation instruction to the main REAME.md file then merge at will

Copy link

@brunoguindani brunoguindani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've had a quick look at the C++ code. I have not followed the development closely, so this is just an external check for major mishaps. Everything looks fine, I have only noted a small amount of observations.

Also, one last thing: what's up with the empty file pybmix/core/pybmixcpp/tmp.hpp?

@mberaha
Copy link
Contributor

mberaha commented Jul 7, 2022

If everything has been addressed, you can merge the PR!

@gbpollam gbpollam merged commit 3b30c7e into bayesmix-dev:master Jul 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants