Skip to content

Commit

Permalink
fix: Native builds for Linux/Windows (#70)
Browse files Browse the repository at this point in the history
* chore: Performance wrong caption

* Info about benchmarking

* fix: Setting target arch for Linux/Windows to skylake
  • Loading branch information
mwiewior authored Jan 13, 2025
1 parent 309e98f commit 249044f
Show file tree
Hide file tree
Showing 6 changed files with 19 additions and 25 deletions.
19 changes: 12 additions & 7 deletions .github/workflows/publish_to_pypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ permissions:

# Make sure CI fails on all warnings, including Clippy lints
env:
RUSTFLAGS: "-Dwarnings -Ctarget-cpu=native"
POETRY_VERSION: 1.8.4
TARGET_CPU: skylake

jobs:
linux_tests:
Expand Down Expand Up @@ -74,10 +74,11 @@ jobs:
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.target }}

args: --release --out dist --find-interpreter
sccache: 'true'
manylinux: auto
env:
RUSTFLAGS: "-Dwarnings -Ctarget-cpu=${{ env.TARGET_CPU }}"

- name: Upload wheels
uses: actions/upload-artifact@v4
Expand All @@ -101,12 +102,11 @@ jobs:
- name: Build wheels
uses: PyO3/maturin-action@v1
with:

target: ${{ matrix.target }}

args: --release --out dist --find-interpreter
sccache: 'true'

env:
RUSTFLAGS: "-Dwarnings -Ctarget-cpu=${{ env.TARGET_CPU }}"
- name: Upload wheels
uses: actions/upload-artifact@v4
with:
Expand All @@ -118,6 +118,11 @@ jobs:
strategy:
matrix:
target: [x86_64, aarch64]
include:
- target: x86_64
apple_cpu_target: "skylake"
- target: aarch64
apple_cpu_target: "apple-m1"
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
Expand All @@ -137,8 +142,8 @@ jobs:
with:
name: wheels-macos-${{ matrix.target }}
path: dist
env:
RUSTFLAGS: "-Clink-arg=-undefined -Clink-arg=dynamic_lookup"
env:
RUSTFLAGS: "-Dwarnings -Clink-arg=-undefined -Clink-arg=dynamic_lookup -Ctarget-cpu=${{ matrix.apple_cpu_target }}"

sdist:
runs-on: ubuntu-latest
Expand Down
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "polars_bio"
version = "0.5.2"
version = "0.5.3"
edition = "2021"

[lib]
Expand Down
12 changes: 2 additions & 10 deletions docs/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -1138,7 +1138,7 @@ the `parallel` dataset was used (see [Test datasets](#test-datasets))
###### Memory characteristic polars_bio_pandas_pd
![polars-bio-overlap-pd-mem.png](assets/memory/dataframes/polars-bio-overlap-pd-mem.png)

###### Memory characteristic polars_bio_pandas_eager
###### Memory characteristic polars_bio_polars_eager

![polars-bio-overlap-pl-mem.png](assets/memory/dataframes/polars-bio-overlap-pl-mem.png)

Expand Down Expand Up @@ -1319,15 +1319,7 @@ pyranges1 | 35,940 | 2.45x |
![pyranges](assets/memory/pyranges1_sink.png)

## How to run the benchmarks
```bash
poetry env use python3.12
poetry update
poetry shell
RUSTFLAGS="-Ctarget-cpu=native" maturin develop --release -m Cargo.toml
## export path with benchmark datasets, e.g.
export BENCH_DATA_ROOT=/Users/mwiewior/research/databio/
## other test also in benchmark/src directory
python benchmark/src/bench_overlap.py
Check the [repository](https://github.com/biodatageeks/polars-bio-bench) for more details on how to run the benchmarks.
```
!!! todo
Expand Down
7 changes: 2 additions & 5 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,15 @@
pip install polars-bio
```
There are binary versions for Linux (x86_64), MacOS (x86_64 and arm64) and Windows (x86_64).
!!! Note
In case of Xeon processors, you are advised to build the package from source to avoid `Illegal instruction (core dumped)` errors.

In case of other platforms (or errors indicating incompatibilites between Python's ABI), it is fairly easy to build polars-bio from source with [maturin](https://github.com/PyO3/maturin):
```shell
RUSTFLAGS="-Ctarget-cpu=native" maturin build --release -m Cargo.toml
```
and you should see the following output:
```shell
Compiling polars_bio v0.2.11 (/Users/mwiewior/research/git/polars-bio)
Compiling polars_bio v0.5.2 (/Users/mwiewior/research/git/polars-bio)
Finished `release` profile [optimized] target(s) in 1m 25s
📦 Built wheel for abi3 Python ≥ 3.8 to /Users/mwiewior/research/git/polars-bio/target/wheels/polars_bio-0.2.11-cp38-abi3-macosx_11_0_arm64.whl
📦 Built wheel for abi3 Python ≥ 3.8 to /Users/mwiewior/research/git/polars-bio/target/wheels/polars_bio-0.5.2-cp38-abi3-macosx_11_0_arm64.whl
```
and finally install the package with pip:
```bash
Expand Down
2 changes: 1 addition & 1 deletion polars_bio/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
POLARS_BIO_MAX_THREADS = "datafusion.execution.target_partitions"


__version__ = "0.5.2"
__version__ = "0.5.3"
__all__ = [
"overlap",
"nearest",
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "maturin"

[project]
name = "polars-bio"
version = "0.5.2"
version = "0.5.3"
description = "Blazing fast genomic operations on large Python dataframes"
authors = []
requires-python = ">=3.9"
Expand Down

0 comments on commit 249044f

Please sign in to comment.