diff --git a/README.spack_devflow.md b/README.spack_devflow.md new file mode 100644 index 00000000000..4b4bf1411d7 --- /dev/null +++ b/README.spack_devflow.md @@ -0,0 +1,401 @@ +Spack has become a standard package management solution for software development HPC environments. While a powerful tool, even experienced users have found it's use to be occasionally frustrating to navigate. Furthermore, much of the downstream use of Spack tends to emphasize it's use as means to dev-ops. This is, of course, Spack's main context, but this can mislead developers looking for a flexible, extensible programming environment. + +This walkthrough is intended to provide users with experience in using Spack for a developer workflow, with the goal of demonstrating how Spack works "under the hood" by demonstrating the distinct steps taking place that may, at a glance, be unclear. + +Assume familiarity with basic Spack usage (for example, defining a package's dependencies and installing those dependencies) + +## Setup + +### Step 0 - Directory overview and getting a new Spack instance (recommended) + +This guide will focus on Spack's ability to provide a modular environment, constrained to live under a single top-level directory and be "container-like". + +NOTE: This guide is written using a Spack 1.0, which has significant and breaking differences with current in-use Spack deployments. + +First let's make the directory and retrieve a fresh instance of Spack + +```bash +$> mkdir dev_top && cd dev_top +$> git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git +``` + +Since we want to confine our working Spack instance to our top-level directory, we want Spack to not use any existing user-level cache (`${HOME}/.spack`). To tell spack to use a separate user cache that is fixed to his directory + +```bash +$> export SPACK_USER_CACHE_PATH="$(pwd)/.spack" +``` + +Load Spack and run a trivial concretize request to bootstrap the instance. This may take a few minutes to complete. + +```bash +$> source ./spack/share/spack/setup-env.sh +$> spack spec zlib +``` + +### Create an environment + +Let's start with creating our top-level environment directory. For the next few sections of the guide, all code checkouts will be placed beneath this directory. + +```bash +$ ~/dev_top > mkdir seos-dev && cd seos-dev +$ ~/dev_top > cd seos-dev +$ ~/dev_top/seos-dev > +``` + +Now, we want to create our development environment. We will make an _anonymous_ environment; the best way to think about it is that this directory will be synonymous with our development environment - that is, when we work in anything in this directory, it is being done in the environment we will construct. + +```bash +$ ~/dev_top/seos-dev > spack env create -d . +==> Created independent environment in: /home/jack/devel/seos-dev +==> Activate with: spack env activate . +$ ~/dev_top/seos-dev > spack env activate . +``` + +Let's look at what we have after the environment is created + +```bash +$ ~/dev_top/seos-dev > ls -la +drwxr-xr-x@ 4 jack dudes 128 Mar 28 10:29 . +drwxr-xr-x@ 3 jack dudes 96 Mar 28 10:29 .. +drwxr-xr-x@ 5 jack dudes 160 Mar 28 10:29 .spack-env +-rw-r--r--@ 1 jack dudes 230 Mar 28 10:29 spack.yaml +``` + +`spack.yaml` will be the configuration file for our environment. Right now it's empty, but that will soon change. `.spack-env` will be a cache space local to this environment, with the main data being a file-system view that collects our environment similar to a (very) lightweight container. + +NOTE: The next few steps will update our `spack.yaml` file using Spack commands. This is useful for reproducibility and for allowing Spack to generate the correct schema in the file. However, you may also simply edit `spack.yaml` directly. You may even do both and switch between one or the other approach at any time. + +We want to be developing `singularity-eos`, so add it to the environment + +```bash +$ ~/dev_top/seos-dev > spack add singularity-eos@main +``` + +Let's assume we want `mpi` and `hdf5` on, since we might as well. While we're at it, let's fix the `mpi` provider to `openmpi` + +```bash +$ ~/dev_top/seos-dev > spack config add "packages:all:variants: +hdf5+mpi" +$ ~/dev_top/seos-dev > spack config add "packages:mpi:require: openmpi" +``` + +Looking ahead, we want to have access to `spiner` code, so let's explicitly add this to our packages. + +NOTE: This step is not required, but included here for demonstration. In the following sections we will show that we can also get local copies of packages even if we don't have a listed spec for them in the environment. + +```bash +$ ~/dev_top/seos-dev > spack add spiner@main +``` + +We've got a basic spack environment ready. Let's take a look at our updated `spack.yaml` + +```yaml +spack: +# include: +# - ./external_packages.yaml +# - ./external_compilers.yaml + packages: + all: + variants: +hdf5+mpi + mpi: + require: openmpi + specs: + - singularity-eos@main + - spiner@main + view: true + concretizer: + unify: true +``` + +(Here, I've added some `include:` sections for external packages and external compilers. These will be platform specific. See - - below for an example to generate these files) + +Now, let's recall: +- I want to start editing and updating `singularity-eos` code in a new clone and checkout. +- `spiner` and `singularity-eos` have a tight coupling, and I also want a new clone and checkout of that package. +- Perhaps another package? + +## Getting and updating `singularity-eos` + +Let's make some updates to the core package `singularity-eos`. + +The regular pipeline of Spack is to go get a clone of repo into a temporary directory and execute that package's build procedure. What we want is to tell spack to get and use a non-temporary clone, preferably to a path we can easily `cd` to and start hacking. + +This can be done with the `spack develop` command + +```bash +$ ~/dev_top/seos-dev > spack develop singularity-eos@main +==> Cloning source code for singularity-eos@=main +``` + +We can see that spack has done a `git clone` for us! + +```bash +$ ~/dev_top/seos-dev > ls -la +drwxr-xr-x@ 4 jack dudes 128 Mar 28 10:29 . +drwxr-xr-x@ 3 jack dudes 96 Mar 28 10:29 .. +drwxr-xr-x@ 5 jack dudes 160 Mar 28 10:29 .spack-env +drwxrwxr-x@ 2 jack dudes 4.0K Mar 28 11:11 singularity-eos +-rw-rw-r--@ 1 jack dudes 24K Mar 28 11:10 spack.lock +-rw-r--r--@ 1 jack dudes 230 Mar 28 10:29 spack.yaml +``` + +NOTE: The spec version (`@main`) ensures our clone will have the `main` branch checkout, though like any `git clone`, we can (and will) change this branch as needed for development. + +From here, we can do regular Spack things. + +```bash +$ ~/dev_top/seos-dev > spack concretize -f +==> Warning: Unable to resolve the git commit for singularity-eos. An installation of this binary won't have complete binary provenance. +==> Warning: Unable to resolve the git commit for spiner. An installation of this binary won't have complete binary provenance. +==> Concretized 2 specs: +[+] 66ruf4l singularity-eos@main+closure~cuda+eospac+fortran+hdf5~ipo+kokkos~kokkos-kernels+mpi~openmp~python+spiner build_extra:=none build_system=cmake build_type=Release dev_path=/vast/home/mauneyc/develop/xcap-dev/dev-env-workflow/singularity-eos generator=make arch=linux-rhel8-haswell %c,cxx,fortran=gcc@13.2.0 +[+] cvvxqem ^cmake@3.31.6~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-rhel8-haswell %c,cxx=gcc@13.2.0 +[e] ivwo7eo ^curl@7.61.1+gssapi+ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl arch=linux-rhel8-haswell +[+] b4cxilh ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-rhel8-haswell %c,cxx=gcc@13.2.0 +[e] p3bp3e3 ^zlib@1.2.11+optimize+pic+shared build_system=makefile arch=linux-rhel8-haswell +[+] mj4ubyb ^compiler-wrapper@1.0 build_system=generic arch=linux-rhel8-haswell +[+] withyuf ^eigen@3.3.8~ipo~nightly~rocm build_system=cmake build_type=RelWithDebInfo generator=make patches:=55daee8,b8877a8 arch=linux-rhel8-haswell %c,cxx=gcc@13.2.0 +[+] il66js2 ^eospac@6.5.12~offload build_system=generic arch=linux-rhel8-haswell %c,cxx,fortran=gcc@13.2.0 +[e] lbz2bqn ^gcc@13.2.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-rhel8-haswell +[+] va64afh ^gcc-runtime@13.2.0 build_system=generic arch=linux-rhel8-haswell +[e] i5ej3xm ^glibc@2.28 build_system=autotools arch=linux-rhel8-haswell +[e] df7ehnf ^gmake@4.2.1~guile build_system=generic patches:=ca60bd9,fe5b60d arch=linux-rhel8-haswell +[+] zaqgsqx ^hdf5@1.14.6~cxx~fortran+hl~ipo~java~map+mpi+shared~subfiling~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-rhel8-haswell %c=gcc@13.2.0 +[e] lpanugo ^openmpi@5.0.2+atomics~cuda~debug+fortran~gpfs~internal-hwloc~internal-libevent~internal-pmix~ipv6~java~lustre~memchecker~openshmem~romio+rsh~static~two_level_namespace+vt~wrapper-rpath build_system=autotools fabrics:=ucx patches:=1a8ff33 romio-filesystem:=none schedulers:=none arch=linux-rhel8-haswell +[e] ojffucq ^pkgconf@1.4.2 build_system=autotools arch=linux-rhel8-haswell +[+] gjjssb7 ^kokkos@4.6.01~aggressive_vectorization~cmake_lang~compiler_warnings+complex_align~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hip_relocatable_device_code~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial~shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rhel8-haswell %c,cxx=gcc@13.2.0 +[+] e5n25ma ^mpark-variant@1.4.0~ipo build_system=cmake build_type=Release generator=make patches:=21a4f8d,4e173fe,b3501f7 arch=linux-rhel8-haswell %cxx=gcc@13.2.0 +[+] qhzavje ^ports-of-call@main~ipo~test build_system=cmake build_type=Release commit=9dad5cb335ee92d7bf4a33ca070ef93e745212d0 generator=make arch=linux-rhel8-haswell %c,cxx=gcc@13.2.0 +[+] kq4og7r spiner@main+hdf5~ipo+kokkos+mpi~python build_system=cmake build_type=Release dev_path=/vast/home/mauneyc/develop/xcap-dev/dev-env-workflow/spiner generator=make arch=linux-rhel8-haswell %c,cxx=gcc@13.2.0 +$ seos-dev > spack install +``` + +Now, let's make an update to `singularity-eos` code in a new local branch. (I use Vim, you may use your preferred editor) + +```bash +$ ~/dev_top/seos-dev > cd singularity-eos +$ ~/dev_top/seos-dev > git checkout -b new_eos_feature +$ ~/dev_top/seos-dev > vi singularity-eos/eos/eos_bash.hpp +``` + +Here, I'll add my amazing new update + +```c++ + 15 #ifndef _SINGULARITY_EOS_EOS_EOS_BASE_ + 16 #define _SINGULARITY_EOS_EOS_EOS_BASE_ + 17 + 18 #include + 19 #include + 20 #include + 21 + 22 #include + 23 #include + 24 #include + 25 #include + 26 #include + 27 #include + 28 + 29 // MY NEW CODE + 30 // THIS WILL FIX EVERYTHING! + 31 static_assert(false); +``` + +Instead of manually building, let's use Spack to check that the build works. (In the following sections, we will show how to do a more traditional build using this environment) + +```bash +$ ~/dev_top/seos-dev > spack install +[+] /usr (external glibc-2.28-i5ej3xmy3uonjbh2yn3lubq7xmue442j) +[+] /projects/opt/rhel8/x86_64/openmpi/5.0.2-gcc_13.2.0 (external openmpi-5.0.2-lpanugo33x6swa22ws2gvnpt4cyjd6p4) +[+] /usr (external pkgconf-1.4.2-ojffucqu32xqeuph3dnufdahuiqzjijl) +[+] /usr (external zlib-1.2.11-p3bp3e3qjussu7ibvpfolmjd36va7rkq) +[+] /usr (external curl-7.61.1-ivwo7eocoj4lubj2xj7t4luqhffzkng4) +[+] /usr (external gmake-4.2.1-df7ehnfeehbynnqobw5vg4ih5aojixew) +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/compiler-wrapper-1.0-mj4ubybewkayq3jjmsuyqdyskwffbre5 +[+] /projects/opt/rhel8/x86_64/gcc/13.2.0 (external gcc-13.2.0-lbz2bqno2yi74phtrhypxnc3quk3ylbu) +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/gcc-runtime-13.2.0-va64afhaljvjalhfhh66dudde5ifaib4 +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/eigen-3.3.8-withyufrtkjcrt2nng2nauy7jt6qbluw +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/eospac-6.5.12-il66js2nmooobxtng3sykhzmdhobuxv4 +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/kokkos-4.6.01-gjjssb76dkwnmwjvu3lgslvtdknk6soq +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/mpark-variant-1.4.0-e5n25mavzyoqywk4w6g3xwbx5f7kerwh +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/ports-of-call-main-qhzavjeh5tdrzm55u7uw74j4po3rd6ra +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/hdf5-1.14.6-zaqgsqxckmewfgc2wdptlmd2yrgpqp54 +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/ncurses-6.5-b4cxilhggxjrp3ly2bwkh42rogdbvg2c +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/spiner-main-kq4og7rxelnbfqef45rzntjw3lds2uen +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/cmake-3.31.6-cvvxqemwg7lkzpokbh3mt4vbkma6ohnm +==> No binary for singularity-eos-main-66ruf4le2mplqe3qiux3bowkck3gyknc found: installing from source +==> Installing singularity-eos-main-66ruf4le2mplqe3qiux3bowkck3gyknc [19/19] +==> No patches needed for singularity-eos +==> singularity-eos: Executing phase: 'cmake' +==> singularity-eos: Executing phase: 'build' +==> Error: ProcessError: Command exited with status 2: + '/usr/bin/make' '-j16' + +14 errors found in build log: + 43 /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/compiler-wrapper-1.0-mj4ubybewkayq3jjmsuyqdyskwffbre5/libexec/spack/gcc/g++ -DH5_BUILT_AS_DYNAMIC_LIB -DKOKKOS_DEPENDENCE -DPORTABILITY_STRATEGY_KOKKOS -DSINGULARITY_BUILD_CLOSURE -DSINGULARITY_USE_EOSPAC -DSINGULARITY_USE_HDF5 -DSINGULARIT + Y_USE_SPINER -DSINGULARITY_USE_SPINER_WITH_HDF5 -DSINGULARITY_VERSION=\"1.9.2\" -DSINGULARITY_VERSION_MAJOR=1 -DSINGULARITY_VERSION_MINOR=9 -DSINGULARITY_VERSION_PATCH=2 -DSPINER_USE_HDF -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200809L -I/vast/home/mauneyc/develop/xcap-de + v/dev-env-workflow/singularity-eos/eospac-wrapper -I/vast/home/mauneyc/develop/xcap-dev/dev-env-workflow/singularity-eos -I/tmp/mauneyc/spack-stage/spack-stage-singularity-eos-main-66ruf4le2mplqe3qiux3bowkck3gyknc/spack-build-66ruf4l/generated -isystem /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/l + inux-haswell/hdf5-1.14.6-zaqgsqxckmewfgc2wdptlmd2yrgpqp54/include -isystem /vast/projects/opt/rhel8/x86_64/openmpi/5.0.2-gcc_13.2.0/include -isystem /projects/opt/rhel8/x86_64/openmpi/5.0.2-gcc_13.2.0/include -isystem /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/eospac-6.5.12-il66js2n + mooobxtng3sykhzmdhobuxv4/include -isystem /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/kokkos-4.6.01-gjjssb76dkwnmwjvu3lgslvtdknk6soq/include -isystem /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/mpark-variant-1.4.0-e5n25mavzyoqywk4w6g3xwbx5f7kerwh/include -isyste + m /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/ports-of-call-main-qhzavjeh5tdrzm55u7uw74j4po3rd6ra/include -isystem /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/spiner-main-kq4og7rxelnbfqef45rzntjw3lds2uen/include -isystem /vast/home/mauneyc/develop/xcap-dev/spack + /opt/spack/linux-haswell/eigen-3.3.8-withyufrtkjcrt2nng2nauy7jt6qbluw/include/eigen3 -O3 -DNDEBUG -pthread -march=core-avx2 -mtune=core-avx2 -Wno-psabi -Wno-class-memaccess -MD -MT CMakeFiles/singularity-eos_Library.dir/singularity-eos/eos/get_sg_eos_rho_p.cpp.o -MF CMakeFiles/singularity-eos_Library.dir/ + singularity-eos/eos/get_sg_eos_rho_p.cpp.o.d -o CMakeFiles/singularity-eos_Library.dir/singularity-eos/eos/get_sg_eos_rho_p.cpp.o -c /vast/home/mauneyc/develop/xcap-dev/dev-env-workflow/singularity-eos/singularity-eos/eos/get_sg_eos_rho_p.cpp + 44 [ 80%] Building CXX object CMakeFiles/singularity-eos_Library.dir/singularity-eos/eos/get_sg_eos_rho_e.cpp.o + 45 /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/compiler-wrapper-1.0-mj4ubybewkayq3jjmsuyqdyskwffbre5/libexec/spack/gcc/g++ -DH5_BUILT_AS_DYNAMIC_LIB -DKOKKOS_DEPENDENCE -DPORTABILITY_STRATEGY_KOKKOS -DSINGULARITY_BUILD_CLOSURE -DSINGULARITY_USE_EOSPAC -DSINGULARITY_USE_HDF5 -DSINGULARIT + Y_USE_SPINER -DSINGULARITY_USE_SPINER_WITH_HDF5 -DSINGULARITY_VERSION=\"1.9.2\" -DSINGULARITY_VERSION_MAJOR=1 -DSINGULARITY_VERSION_MINOR=9 -DSINGULARITY_VERSION_PATCH=2 -DSPINER_USE_HDF -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200809L -I/vast/home/mauneyc/develop/xcap-de + v/dev-env-workflow/singularity-eos/eospac-wrapper -I/vast/home/mauneyc/develop/xcap-dev/dev-env-workflow/singularity-eos -I/tmp/mauneyc/spack-stage/spack-stage-singularity-eos-main-66ruf4le2mplqe3qiux3bowkck3gyknc/spack-build-66ruf4l/generated -isystem /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/l + inux-haswell/hdf5-1.14.6-zaqgsqxckmewfgc2wdptlmd2yrgpqp54/include -isystem /vast/projects/opt/rhel8/x86_64/openmpi/5.0.2-gcc_13.2.0/include -isystem /projects/opt/rhel8/x86_64/openmpi/5.0.2-gcc_13.2.0/include -isystem /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/eospac-6.5.12-il66js2n + mooobxtng3sykhzmdhobuxv4/include -isystem /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/kokkos-4.6.01-gjjssb76dkwnmwjvu3lgslvtdknk6soq/include -isystem /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/mpark-variant-1.4.0-e5n25mavzyoqywk4w6g3xwbx5f7kerwh/include -isyste + m /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/ports-of-call-main-qhzavjeh5tdrzm55u7uw74j4po3rd6ra/include -isystem /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/spiner-main-kq4og7rxelnbfqef45rzntjw3lds2uen/include -isystem /vast/home/mauneyc/develop/xcap-dev/spack + /opt/spack/linux-haswell/eigen-3.3.8-withyufrtkjcrt2nng2nauy7jt6qbluw/include/eigen3 -O3 -DNDEBUG -pthread -march=core-avx2 -mtune=core-avx2 -Wno-psabi -Wno-class-memaccess -MD -MT CMakeFiles/singularity-eos_Library.dir/singularity-eos/eos/get_sg_eos_rho_e.cpp.o -MF CMakeFiles/singularity-eos_Library.dir/ + singularity-eos/eos/get_sg_eos_rho_e.cpp.o.d -o CMakeFiles/singularity-eos_Library.dir/singularity-eos/eos/get_sg_eos_rho_e.cpp.o -c /vast/home/mauneyc/develop/xcap-dev/dev-env-workflow/singularity-eos/singularity-eos/eos/get_sg_eos_rho_e.cpp + 46 In file included from /vast/home/mauneyc/develop/xcap-dev/dev-env-workflow/singularity-eos/singularity-eos/eos/default_variant.hpp:27, + 47 from /tmp/mauneyc/spack-stage/spack-stage-singularity-eos-main-66ruf4le2mplqe3qiux3bowkck3gyknc/spack-build-66ruf4l/generated/singularity-eos/eos/eos.hpp:18, + 48 from /vast/home/mauneyc/develop/xcap-dev/dev-env-workflow/singularity-eos/singularity-eos/eos/singularity_eos.cpp:17: + >> 49 /vast/home/mauneyc/develop/xcap-dev/dev-env-workflow/singularity-eos/singularity-eos/eos/eos_base.hpp:31:15: error: static assertion failed + 50 31 | static_assert(false); + 51 | ^~~~~ +``` + +Oops! Looks like we messed up. Let's fix it + +```bash +$ ~/dev_top/seos-dev > vi singularity-eos/eos/eos_bash.hpp +``` + +```c++ + 15 #ifndef _SINGULARITY_EOS_EOS_EOS_BASE_ + 16 #define _SINGULARITY_EOS_EOS_EOS_BASE_ + 17 + 18 #include + 19 #include + 20 #include + 21 + 22 #include + 23 #include + 24 #include + 25 #include + 26 #include + 27 #include + 28 + 29 // MY NEW CODE + 30 // THIS WILL FIX EVERYTHING! NOW TRUE! + 31 static_assert(true); +``` + +```bash +$ seos-dev > spack install +[+] /usr (external glibc-2.28-i5ej3xmy3uonjbh2yn3lubq7xmue442j) +[+] /projects/opt/rhel8/x86_64/openmpi/5.0.2-gcc_13.2.0 (external openmpi-5.0.2-lpanugo33x6swa22ws2gvnpt4cyjd6p4) +[+] /usr (external pkgconf-1.4.2-ojffucqu32xqeuph3dnufdahuiqzjijl) +[+] /usr (external zlib-1.2.11-p3bp3e3qjussu7ibvpfolmjd36va7rkq) +[+] /usr (external curl-7.61.1-ivwo7eocoj4lubj2xj7t4luqhffzkng4) +[+] /usr (external gmake-4.2.1-df7ehnfeehbynnqobw5vg4ih5aojixew) +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/compiler-wrapper-1.0-mj4ubybewkayq3jjmsuyqdyskwffbre5 +[+] /projects/opt/rhel8/x86_64/gcc/13.2.0 (external gcc-13.2.0-lbz2bqno2yi74phtrhypxnc3quk3ylbu) +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/gcc-runtime-13.2.0-va64afhaljvjalhfhh66dudde5ifaib4 +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/hdf5-1.14.6-zaqgsqxckmewfgc2wdptlmd2yrgpqp54 +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/ports-of-call-main-qhzavjeh5tdrzm55u7uw74j4po3rd6ra +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/ncurses-6.5-b4cxilhggxjrp3ly2bwkh42rogdbvg2c +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/kokkos-4.6.01-gjjssb76dkwnmwjvu3lgslvtdknk6soq +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/eospac-6.5.12-il66js2nmooobxtng3sykhzmdhobuxv4 +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/mpark-variant-1.4.0-e5n25mavzyoqywk4w6g3xwbx5f7kerwh +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/eigen-3.3.8-withyufrtkjcrt2nng2nauy7jt6qbluw +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/cmake-3.31.6-cvvxqemwg7lkzpokbh3mt4vbkma6ohnm +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/spiner-main-kq4og7rxelnbfqef45rzntjw3lds2uen +==> No binary for singularity-eos-main-66ruf4le2mplqe3qiux3bowkck3gyknc found: installing from source +==> Installing singularity-eos-main-66ruf4le2mplqe3qiux3bowkck3gyknc [19/19] +==> No patches needed for singularity-eos +==> singularity-eos: Executing phase: 'cmake' +==> singularity-eos: Executing phase: 'build' +==> singularity-eos: Executing phase: 'install' +==> singularity-eos: Successfully installed singularity-eos-main-66ruf4le2mplqe3qiux3bowkck3gyknc + Stage: 0.00s. Cmake: 0.00s. Build: 1m 16.82s. Install: 2.69s. Post-install: 0.56s. Total: 1m 20.29s +[+] /vast/home/mauneyc/develop/xcap-dev/spack/opt/spack/linux-haswell/singularity-eos-main-66ruf4le2mplqe3qiux3bowkck3gyknc +``` + +Great! We can now commit and request a merge. + +And there we go! We've gotten a build with our expected environment, made changes and built all with spack! + +## Working with `spiner` and getting a build environment + +Now, we want to add a feature to `spiner` to enhance `singularity-eos`. + +As before, tell Spack we want a local source directory + +```bash +~/dev_top/seos-dev > spack develop spiner@main +``` + +NOTE: Here, we are still using our local clone of `singularity-eos`. At the end of this section we will "remove" these local clones from our environment. + +```bash +ls +vi +``` + +Before, we used `spack install` to run the build steps. Here, let's instead ask Spack for a pre-configured shell where we can manually build the code ourselves + +```bash +~/dev_top/seos-dev > spack build-env spiner@main -- bash +~/dev_top/seos-dev > +``` + +Spack has launched a new Bash shell and loaded it with all the build information for us. We can now do a more traditional build routine + +```bash +~/dev_top/seos-dev > cd spiner && mkdir build && cd build +~/dev_top/seos-dev > cmake .. +# ... +~/dev_top/seos-dev > cmake --build . +``` + +Once you have finished doing your work, you can even make sure your new updates are compatible with upstream `singularity-eos` + +```bash +~/dev_top/seos-dev > exit +~/dev_top/seos-dev > spack install +``` + +And there we go! We have a shared environment for `spiner` and `singularity-eos`, and `singularity-eos` is using our local, updated `spiner`! Who said spack is hard? + +Once your development cycle is complete, you may "reset" the environment to using the default Spack mechanics and not your local code: + +```bash +~/dev_top/seos-dev > spack undevelop spiner +~/dev_top/seos-dev > spack undevelop singularity-eos +``` + +NOTE: `spack undevelop` will remove your local source directory, so make sure you're pushed your changes first! Also, any changes will not be reflected until they are merged. + +## Working with other dependencies; an alternative to git submodules + +`singularity-eos` originally used (and to a limited extent still supports) Git submodules as a dependency solution. In this mode, key dependencies (such as `kokkos`) are configured and built "in-tree", allowing the developer to directly access and modify the source of the dependency. While not a scalable approach, many developers find this level of control useful to their workflow. + +This section will demonstrate using Spack as an alternative approach to that model that also provides the ability to directly access and modify dependency code. + +Let's update our Spack environment to require `kokkos` + +```bash +~/dev_top/seos-dev > spack config change packages:all:variants: +hdf5+mpi+kokkos +~/dev_top/seos-dev > spack concretize -f +``` + +The steps here follow closely from the previous section on `spiner`, and you may follow those instructions. Here, we will introduce some additional Spack functionality. + +Let's consider the case where the developer has a separate clone of `kokkos` they've been working on elsewhere on the filesystem + +```bash +# IN THE DISTANT PAST.... +~/my_code > git clone kokkos +~/my_code > cd kokkos +~/my_code/kokkos > git checkout -b new_device +# ... +``` + +Back in our Spack environment, we can tell Spack we want to use our existing clone like so + +```bash +~/dev_top/seos-dev > spack develop --path $HOME/my_code/kokkos kokkos@4: +``` + +Spack will elide the clone step, and subsequently use the provided source path when building/installing the `kokkos` package.