Use shpc to build modules

[fredjaya]

This PR makes shelley 🐢 build modules using shpc install under the hood, replacing creating a dummy modules.lua file.

Manual tests

blast:2.5.0--hc0b0e79_3

• Old version that does not exist in remote registry
• Requires multiple aliases to different binaries e.g. blastn, blastp

shelley-bio build blast:2.5.0--hc0b0e79_3
module load blast/2.5.0--hc0b0e79_3
blastn -version

WARNING: Skipping mount /opt/singularity/4.2.1/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container blastn: 2.5.0+
 Package: blast 2.5.0, build Sep 20 2018 01:34:18

blastp -version

WARNING: Skipping mount /opt/singularity/4.2.1/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
blastp: 2.5.0+
 Package: blast 2.5.0, build Sep 20 2018 01:34:18

Notes:

• creating local registry and adding new container works
• aliases are preserved this way
• /etc/resolv.conf warning

star/2.7.11a--h0033a41_0

• Latest version that exists in the registry
• Issues with bad $PATH exports due to manual module.lua build shelley-bio build: executable file not found in $PATH #9

shelley-bio build star
module load star/2.7.11b--h5ca1c30_8
STAR --version

2.7.11b # success

star-fusion:1.0.0

• Older version of star-fusion that is not in the registry
• Has many aliases in the rmeote registry

shelley-bio build star-fusion:1.0.0
module load star-fusion

STARlong --version

WARNING: Skipping mount /opt/singularity/4.2.1/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
STAR_2.5.3a

salmon

WARNING: Skipping mount /opt/singularity/4.2.1/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
FATAL:   stat /usr/local/bin/salmon: no such file or directory

• Issue founds, aliases may not reflect older builds, or are broken. Inspect.

star-fusion-exec find /usr/local/bin -type f | grep salmon # no salmon
star-fusion-exec find /usr/local/bin -type f | grep STAR

/usr/local/bin/STAR
/usr/local/bin/STAR-Fusion
/usr/local/bin/STARlong

Confirmed: registry aliases does not reflect older versions.

Possible solution is to use an automated binary detection like shpc-guts that is compatible with singularity?

[fredjaya]

Spike test: Inspecting CVMFS images for aliases during build

Utilising shpc-registry helper scripts to create accurate registries

from add_container.py:

    # Generate guts
    cli = ManifestGenerator()
    manifests = cli.diff(args.image)

    # Assemble aliases
    aliases = {}
    for path in list(manifests.values())[0]["diff"]["unique_paths"]:
        # Don't include system bin
        if "sbin" in path or "/usr/bin" in path:
            print(f"Skipping system bin {path}")
            continue

        name = os.path.basename(path)
        if name in aliases:
            print(f"Warning, duplicate alias {name}")
        aliases[name] = path

    container["aliases"] = aliases
    print(f"Writing with aliases to {container_yaml}")
    shpc.utils.write_yaml(container, container_yaml)
    print(f"container_yaml={container_yaml} >> $GITHUB_OUTPUT")

container_guts.main.ManifestGenerator() looks likes the way the shpc workflow automatically updates the aliases, but uses Docker so this is a no go for the BioShell environment

---

Using find?

module load star-fusion
star-fusion-exec find /usr/local/bin -perm +111 -maxdepth 1 | wc -l

146 # MANY binaries...

Can we diff out the utility tools (e.g. non bioinformatics binaries) from /usr/bin?

star-fusion-exec find /usr/bin -perm +111 -maxdepth 1 | head

/usr/bin/[
/usr/bin/[[
/usr/bin/ar
/usr/bin/awk
/usr/bin/basename
/usr/bin/bunzip2
/usr/bin/bzcat
/usr/bin/chrt
/usr/bin/chvt

STAR-fusion v1.0.0 conda build script:

• https://github.com/bioconda/bioconda-recipes/tree/75505efe265e4c7143e57ff4fa03506f554715d0/recipes/star-fusion

Resource for biocontainer builds:

• https://cyverse-container-camp-workshop-2018.readthedocs-hosted.com/en/latest/biocontainer/biocontainers.html#introduction-to-biocontainers