Slim conda env before packing to fit under 2GB GitHub release limit #2
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Build & Release Conda Env | |
| on: | |
| push: | |
| tags: ['envs-v*'] | |
| workflow_dispatch: | |
| inputs: | |
| tag: | |
| description: 'Release tag (e.g. envs-v1)' | |
| required: true | |
| default: 'envs-v1' | |
| jobs: | |
| build-env: | |
| runs-on: ubuntu-22.04 | |
| permissions: | |
| contents: write | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Set up Miniconda | |
| uses: conda-incubator/setup-miniconda@v3 | |
| with: | |
| python-version: '3.10' | |
| miniconda-version: latest | |
| auto-activate-base: true | |
| - name: Create conda env | |
| shell: bash -el {0} | |
| run: | | |
| conda create -p ./env python=3.10 -y | |
| - name: Install GPU PyTorch | |
| shell: bash -el {0} | |
| run: | | |
| ./env/bin/pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu118 | |
| - name: Install dependencies | |
| shell: bash -el {0} | |
| run: | | |
| ./env/bin/pip install -r requirements.txt | |
| ./env/bin/pip install git+https://github.com/facebookresearch/esm.git | |
| ./env/bin/pip install --ignore-installed fastmcp | |
| - name: Slim environment before packing | |
| shell: bash -el {0} | |
| run: | | |
| ENV=./env | |
| echo "=== Pre-cleanup env size ===" | |
| du -sh $ENV | |
| # 1. Remove duplicate CUDA libs from env/lib/ (torch uses nvidia/*/lib/ copies) | |
| rm -f $ENV/lib/libcublas* $ENV/lib/libcufft* $ENV/lib/libcusolver* \ | |
| $ENV/lib/libcusparse* $ENV/lib/libnvrtc* | |
| # 2. Remove triton (only needed for torch.compile, ESM uses eager mode) | |
| rm -rf $ENV/lib/python3.10/site-packages/triton | |
| # 3. Remove unused MKL variants (torch uses mkl_rt → core + intel_thread only) | |
| rm -f $ENV/lib/libmkl_pgi* $ENV/lib/libmkl_tbb* $ENV/lib/libmkl_gnu* \ | |
| $ENV/lib/libmkl_sequential* $ENV/lib/libmkl_scalapack* \ | |
| $ENV/lib/libmkl_blacs* $ENV/lib/libmkl*ilp64* | |
| # 4. Remove NPP libraries (image processing, unused by ESM) | |
| rm -f $ENV/lib/libnpp* | |
| rm -rf $ENV/lib/python3.10/site-packages/nvidia/npp | |
| # 5. Remove __pycache__ directories (regenerated on first import) | |
| find $ENV -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true | |
| # 6. Remove nccl (multi-GPU distributed communication, not needed) | |
| rm -rf $ENV/lib/python3.10/site-packages/nvidia/nccl | |
| rm -f $ENV/lib/libnccl* | |
| # 7. Remove CUDA targets/ directory (headers/stubs, not runtime) | |
| find $ENV -type d -name targets -path "*/nvidia/*" -exec rm -rf {} + 2>/dev/null || true | |
| # 8. Remove test directories in packages | |
| find $ENV/lib/python3.10/site-packages -type d -name tests -exec rm -rf {} + 2>/dev/null || true | |
| find $ENV/lib/python3.10/site-packages -type d -name test -exec rm -rf {} + 2>/dev/null || true | |
| # 9. Remove profiling libraries | |
| rm -rf $ENV/lib/python3.10/site-packages/nvidia/cuda_cupti | |
| rm -f $ENV/lib/libnvperf* $ENV/lib/libcupti_static* | |
| # 10. Remove torch C++ headers (only for building extensions) | |
| rm -rf $ENV/lib/python3.10/site-packages/torch/include | |
| # 11. Remove pip, setuptools, wheel (not needed in packed runtime env) | |
| rm -rf $ENV/lib/python3.10/site-packages/pip \ | |
| $ENV/lib/python3.10/site-packages/setuptools \ | |
| $ENV/lib/python3.10/site-packages/wheel \ | |
| $ENV/lib/python3.10/site-packages/_distutils_hack | |
| # 12. Remove torchaudio (not used by ESM) | |
| rm -rf $ENV/lib/python3.10/site-packages/torchaudio | |
| # 13. Remove .dist-info directories (package metadata, not runtime) | |
| find $ENV/lib/python3.10/site-packages -type d -name "*.dist-info" -exec rm -rf {} + 2>/dev/null || true | |
| # 14. Strip debug symbols from shared libraries | |
| find $ENV -name "*.so" -exec strip --strip-debug {} + 2>/dev/null || true | |
| find $ENV -name "*.so.*" -exec strip --strip-debug {} + 2>/dev/null || true | |
| echo "=== Post-cleanup env size ===" | |
| du -sh $ENV | |
| - name: Pack environment | |
| shell: bash -el {0} | |
| run: | | |
| conda install -n base conda-pack -y | |
| $(conda info --base)/bin/conda-pack -p ./env -o esm_mcp-env.tar.gz --ignore-editable-packages | |
| - name: Determine release tag | |
| id: tag | |
| run: | | |
| if [[ "${{ github.event_name }}" == "push" ]]; then | |
| echo "tag=${{ github.ref_name }}" >> $GITHUB_OUTPUT | |
| else | |
| echo "tag=${{ github.event.inputs.tag }}" >> $GITHUB_OUTPUT | |
| fi | |
| - name: Create GitHub Release | |
| uses: softprops/action-gh-release@v2 | |
| with: | |
| tag_name: ${{ steps.tag.outputs.tag }} | |
| name: "Conda Environment ${{ steps.tag.outputs.tag }}" | |
| body: | | |
| Pre-packaged conda environment for `esm_mcp`. | |
| **Usage in quick_setup.sh:** | |
| ```bash | |
| USE_PACKED_ENVS=1 bash quick_setup.sh | |
| ``` | |
| **Usage in Google Colab:** | |
| ```python | |
| import os, subprocess | |
| url = "https://github.com/<owner>/<repo>/releases/download/${{ steps.tag.outputs.tag }}/esm_mcp-env.tar.gz" | |
| subprocess.run(["wget", "-q", url]) | |
| os.makedirs("esm_mcp/env", exist_ok=True) | |
| subprocess.run(["tar", "xzf", "esm_mcp-env.tar.gz", "-C", "esm_mcp/env"]) | |
| subprocess.run(["bash", "-c", "source esm_mcp/env/bin/activate && conda-unpack"]) | |
| ``` | |
| files: esm_mcp-env.tar.gz | |
| fail_on_unmatched_files: true |