Provide wheels directly for every platform (#2)

* Directly build wheels for all platforms by changing tag

* Remove mscl_release_assets from origin

* Add mscl_release_assets/ to gitignore

* Use mscl 67.0.1, make CI work, use zipfile for extraction

Author: harshil21

.github/workflows/build_wheels.yml

@@ -1,4 +1,4 @@
-name: Build wheels
+name: Download, extract, build, and test wheels
 
 on:
   pull_request:
@@ -10,19 +10,14 @@ on:
 
 jobs:
   build-wheels:
-    name: Build wheels
+    name: Download, extract, build, and wheels
     runs-on: ${{ matrix.os }}
     timeout-minutes: 5
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-latest, windows-latest]
-        python-version: [3.9, 3.11, 3.13]
-        exclude:
-          - os: windows-latest
-            python-version: 3.9
-          - os: windows-latest
-            python-version: 3.13
+        os: [ubuntu-latest, windows-latest, ubuntu-24.04-arm]
+        python-version: [3.13]
 
     steps:
       - uses: actions/checkout@v4
@@ -39,15 +34,21 @@ jobs:
 
       - name: Install the project
         run: uv sync
+      
+      - name: Download and extract the mscl assets
+        run: |
+          uv run download_and_extract_assets.py
 
-      - name: Build the wheel
-        run: uv build --wheel
+      - name: Build the wheels
+        run: |
+          uv run run_build.py
 
-      - name: Initialize test environment
+      # There should be only one wheel so it should work:
+      - name: Initialize test environment and install wheel
         run: |
           uv init --no-workspace test
           cd test
-          uv add ../dist/*.whl
+          uv add "$(python -c 'import glob; print(glob.glob("../dist/*.whl")[0])')"
 
       - name: Verify installation
         run: uv run -- python -c "from python_mscl import mscl"

.gitignore

@@ -15,10 +15,12 @@ wheels/
 # Exclude local .deb and .zip files from mscl:
 mscl_release_assets/*.zip
 mscl_release_assets/*.deb
+mscl_release_assets/
 
 # VSCode:
 .vscode/
 
 # Exclude build downloaded mscl files:
 src/python_mscl/mscl.py
 src/python_mscl/_mscl.so
+src/python_mscl/mscl.pyd

README.md

@@ -2,10 +2,9 @@
 
 Unofficial Python package for the [Microstrain Communication Library](https://github.com/LORD-MicroStrain/MSCL/tree/master).
 
-This library just makes it so that we can install the MSCL library using pip. Wheels are not provided. This will fetch the necessary files for your architecture and python
-version, and then build the wheel for you.
+This library just makes it so that we can install the MSCL library using pip, and directly provides the wheels!
 
-It is therefore recommended to use a cache for your CI or package manager, unless you're okay with the ~20MB download every time you run your CI.
+Only Python 3.x wheels are provided. If you need Python 2.x wheels, please open an issue.
 
 ### Installation
 
@@ -21,11 +20,6 @@ from python_mscl import mscl
 # ... use the MSCL library as you normally would
 ```
 
-### Windows support:
-
-The latest mscl version (v67.0.0) only has a .zip for python 3.11. It has been confirmed that it does not work on other python versions (You would get an import error). However the build itself would still go through.
-
-
 ### Versioning system:
 
 This repository follows the same versioning system as the MSCL library. This is reflected in the tags of this repository.
@@ -44,13 +38,17 @@ The below steps assume you have [`uv`](https://docs.astral.sh/uv/) installed.
 
 1. Clone the repo and `cd` into it.
 2. Optional: Create a .env file and insert your GITHUB_TOKEN= to make requests to the GitHub API.
-3. Edit & run `uv run main.py` to fetch the latest tagged MSCL releases and extract them.
-4. Run `uv build`, which will build the source distribution and wheel for your python
-version and architecture.
+3. Edit & run `uv run download_and_extract_assets.py` to fetch the latest tagged MSCL releases and extract them.
+4. Run `uv run run_build.py`, which will build the source distribution and wheel for your python
+version and architecture. The wheels will be placed in the `dist/` directory.
 
 Notes for me, the maintainer:
-5. Optional: Run `uv publish` to publish the package to PyPI. To upload to TestPyPI, uncomment lines in `pyproject.toml`, and run `uv publish --index testpypi dist/*.tar.gz`.
-6. Optional: To check if the package worked correctly: `uv add --index https://test.pypi.org/simple/ --index-strategy unsafe-best-match python-mscl` in a new uv project directory.
+5. Make sure that the constants in `constants.py` are updated, and that the MSCL repo still follows their
+versioning system. If not, update rest of the files accordingly.
+
+6. Optional: Run `uv publish` to publish the package to PyPI. To upload to TestPyPI, uncomment lines in `pyproject.toml`, and run `uv publish --index testpypi dist/*.whl`.
+
+7. Optional: To check if the package worked correctly: `uv add --index https://test.pypi.org/simple/ --index-strategy unsafe-best-match python-mscl` in a new uv project directory.
 
 
 ## Issues:

build_helpers/release_downloader.py

@@ -8,6 +8,8 @@
 from github.GitRelease import GitRelease
 from github.GitReleaseAsset import GitReleaseAsset
 
+from constants import ASSET_DIRECTORY, ReleaseAsset
+
 
 class GithubDownloader:
     """Manages downloading the Github release assets for the mscl library, along with the
@@ -18,6 +20,7 @@ def __init__(self):
         self.mscl_repo = "LORD-MicroStrain/MSCL"
         self.python_mscl_repo = "harshil21/python-mscl"
         self.latest_release = None
+        self.asset_dir = Path(ASSET_DIRECTORY)
 
     def get_latest_release(self) -> GitRelease:
         """Returns the latest stable release for the given repo."""
@@ -33,13 +36,19 @@ def get_latest_release(self) -> GitRelease:
                 break
         return self.latest_release
 
-    def download_release_assets(self, output_dir: str):
-        """Downloads the release assets for the given repo and tag."""
+    def download_release_assets(self, only_release: ReleaseAsset | None = None) -> None:
+        """Downloads the release assets from the MSCL repository.
+
+        Args:
+            only_release: If set, only download the release asset for the given Python version and
+                architecture. If not set, download all the release assets.
+        """
         release = self.get_latest_release()
-        output_path = Path(output_dir)
+        output_path = Path(self.asset_dir)
         output_path.mkdir(parents=True, exist_ok=True)
 
         asset: GitReleaseAsset
+        print(f"Downloading release assets for {only_release=}")
         for asset in release.get_assets():
             # Don't download the "Documentation" or "Examples"
             if "Documentation" in asset.name or "Examples" in asset.name:
@@ -51,22 +60,17 @@ def download_release_assets(self, output_dir: str):
             if "3" not in asset.name:
                 continue
 
+            # Extract the python version, arch, and platform from the only_release, if set:
+            if only_release:
+                if only_release.python_version not in asset.name:
+                    continue
+                if only_release.arch not in asset.name:
+                    continue
+
             self.download_asset(output_path, asset)
+            print(f"Downloaded {asset.name}")
 
     def download_asset(self, output_path: Path, asset: GitReleaseAsset) -> None:
         response = requests.get(asset.browser_download_url, timeout=15)
         asset_path = output_path / asset.name
         asset_path.write_bytes(response.content)
-
-    def download_assets_from_folder(self, tag: str, folder_name: str) -> None:
-        """Downloads all the files under the `folder_name` for the given tag, from the
-        root of the repository."""
-
-        repo = self.github.get_repo(self.python_mscl_repo)
-        contents = repo.get_contents(folder_name, ref=tag)
-
-        for content in contents:
-            if content.type == "file":
-                response = requests.get(content.download_url, timeout=15)
-                file_path = Path(content.name)
-                file_path.write_bytes(response.content)

build_helpers/release_extractor.py

@@ -1,17 +1,19 @@
 """Extracts the .deb and .zip releases for the mscl library."""
 
 import os
+import shutil
 import subprocess
 from pathlib import Path
+from zipfile import ZipFile
+
+from constants import ASSET_DIRECTORY, MSCL_VERSION
 
-MSCL_VERSION = "v67.0.0"
-"""The mscl version to extract."""
 
 class ReleaseExtractor:
     """Will extract the .deb and .zip releases for the mscl library."""
 
     def __init__(self):
-        self.asset_dir = Path("mscl_release_assets")
+        self.asset_dir = Path(ASSET_DIRECTORY)
 
     def extract_assets(self):
         """Extracts the .deb and .zip releases into the same directory."""
@@ -77,18 +79,19 @@ def extract_zip(self, file: Path) -> None:
         # Create a directory to extract the .zip file. Syntax: mscl-<arch>-<python-ver>-<mscl-ver>
         parts = file.stem.split("_")
         arch, py_ver = parts[2], parts[3]
-        mscl_versioned_name = f"mscl-Windows-{arch}-{py_ver}-{MSCL_VERSION}"
+        mscl_versioned_name = f"mscl-Windows_{arch}-{py_ver}-{MSCL_VERSION}"
         mscl_versioned_dir = cwd / self.asset_dir / mscl_versioned_name
 
         # If output directory exists, remove it:
         if mscl_versioned_dir.exists():
-            os.system(f"rm -rf {mscl_versioned_dir}")
+            shutil.rmtree(mscl_versioned_dir)
 
         mscl_versioned_dir.mkdir(parents=True, exist_ok=True)
-        file_relative = file.absolute().relative_to(mscl_versioned_dir, walk_up=True)
 
         # Extract the .zip file
-        subprocess.run(["unzip", str(file_relative)], cwd=mscl_versioned_dir, check=True)  # noqa: S603, S607
+        with ZipFile(file, "r") as zip_ref:
+            zip_ref.extractall(mscl_versioned_dir)
+            print("Extracted the zip file.")
 
         found_mscl_py = list(mscl_versioned_dir.rglob("mscl.py"))
         found_mscl_pyd = list(mscl_versioned_dir.rglob("_mscl.pyd"))
@@ -106,8 +109,18 @@ def extract_zip(self, file: Path) -> None:
         # Delete the remaining files in mscl_versioned_dir:
         for f in mscl_versioned_dir.iterdir():
             if f.stem in (mscl_py.stem, mscl_pyd.stem):
+                print(f"Skipping deletion of {f}")
                 continue
             if f.is_dir():
-                os.system(f"rm -rf {f}")
+                print(f"Deleting the directory {f}")
+                shutil.rmtree(f)
             else:
+                print(f"Deleting {f}")
                 f.unlink()
+
+        # Confirm that the files still exist after deleting the rest:
+        found_mscl_py = list(mscl_versioned_dir.rglob("mscl.py"))
+        found_mscl_pyd = list(mscl_versioned_dir.rglob("_mscl.pyd"))
+
+        if not found_mscl_py or not found_mscl_pyd:
+            raise FileNotFoundError(f"Deleted mscl.py or _mscl.pyd in {mscl_versioned_dir}!")

constants.py

@@ -0,0 +1,38 @@
+"""File to store common constants used in the project."""
+
+from pathlib import Path
+from typing import NamedTuple
+
+
+# Named tuple to store the release asset information:
+class ReleaseAsset(NamedTuple):
+    """Named tuple to store the release asset information.
+
+    Args:
+        python_version: The Python version of the release asset. E.g. "Python3.9".
+        arch: The architecture of the release asset. E.g. "amd64".
+    """
+
+    python_version: str
+    arch: str
+
+
+ASSET_DIRECTORY = Path("mscl_release_assets")
+"""The directory to store the downloaded release assets."""
+
+
+# Keep this the same as the one in `hatch_build.py`!
+MSCL_VERSION = "v67.0.1"
+"""The mscl version to extract from the `ASSET_DIRECTORY`. The
+downloader will download the latest version despite this version number."""
+
+
+MACHINE_MAPPING_TO_ARCH = {
+    # Linux:
+    "x86_64": "amd64",
+    "aarch64": "arm64",
+    "armv7l": "armhf",
+    # Windows:
+    "AMD64": "Windows_x64",
+    "x86": "Windows_x86",
+}

download_and_extract_assets.py

@@ -0,0 +1,41 @@
+"""Entry point for downloading and extracting mscl release assets."""
+
+import os
+import platform
+import sys
+
+from build_helpers.release_downloader import GithubDownloader, ReleaseAsset
+from build_helpers.release_extractor import ReleaseExtractor
+from constants import MACHINE_MAPPING_TO_ARCH
+
+
+def main(github_actions: bool = False) -> None:
+    """Entry point to fetch the latest release assets from the Github repository & extract them.
+
+    :param github_actions: If the script is running in a Github Actions environment. If true,
+        the script will download and extract the release asset of only the python version and
+        architecture and OS type detected.
+    """
+    gh = GithubDownloader()
+    if github_actions:
+        print(f"Downloading on {sys.version_info=}, {platform.machine()=}")
+        gh.download_release_assets(
+            ReleaseAsset(
+                python_version=f"Python{sys.version_info.major}.{sys.version_info.minor}",
+                arch=MACHINE_MAPPING_TO_ARCH.get(platform.machine()),
+            )
+        )
+    else:
+        gh.download_release_assets()
+
+    re = ReleaseExtractor()
+    re.extract_assets()
+
+
+if __name__ == "__main__":
+    if os.getenv("GITHUB_ACTIONS", "false") == "true":
+        print("Running in Github Actions environment.")
+        main(github_actions=True)
+    else:
+        print("Running locally")
+        main()