Merge pull request #18 from pycompression/release_0.1.0

Release 0.1.0

Author: rhpvorderman

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,3 @@
+
+### Checklist
+- [ ] Pull request details were added to CHANGELOG.rst

.github/release_checklist.md

@@ -0,0 +1,21 @@
+Release checklist
+- [ ] Check outstanding issues on JIRA and Github.
+- [ ] Check [latest documentation](https://python-isal.readthedocs.io/en/latest/) looks fine.
+- [ ] Create a release branch.
+  - [ ] Set version to a stable number.
+  - [ ] Change current development version in `CHANGELOG.rst` to stable version.
+- [ ] Merge the release branch into `main`.
+- [ ] Create a test pypi package from the main branch. ([Instructions.](
+https://packaging.python.org/tutorials/packaging-projects/#generating-distribution-archives
+))
+- [ ] Install the packages from the test pypi repository to see if they work.
+- [ ] Created an annotated tag with the stable version number. Include changes 
+from CHANGELOG.rst.
+- [ ] Push tag to remote.
+- [ ] Push tested packages to pypi.
+- [ ] merge `main` branch back into `develop`.
+- [ ] Add updated version number to develop.
+- [ ] Build the new tag on readthedocs. Only build the last patch version of
+each minor version. So `1.1.1` and `1.2.0` but not `1.1.0`, `1.1.1` and `1.2.0`.
+- [ ] Create a new release on github.
+- [ ] Update the package on conda-forge.

.gitignore

@@ -6,6 +6,10 @@ __pycache__/
 # C extensions
 *.so
 
+# Cython generated files
+*.c
+*.html
+
 # Distribution / packaging
 .Python
 build/

.readthedocs.yml

@@ -0,0 +1,9 @@
+version: 2
+formats: []  # Do not build epub and pdf
+
+python:
+  install:
+    - method: pip
+      path: .
+conda:
+  environment: docs/conda-environment.yml

.travis.yml

@@ -0,0 +1,35 @@
+language: python
+
+before_install:
+  # Install conda
+  - export MINICONDA=${HOME}/miniconda
+  - export PATH=${MINICONDA}/bin:${PATH}
+  - wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
+  - bash miniconda.sh -b -f -p ${MINICONDA}
+  - conda config --set always_yes yes
+  - conda config --add channels defaults
+  - conda config --add channels conda-forge
+
+install:
+  - conda create -n python-isal python=$TRAVIS_PYTHON_VERSION tox isa-l
+  - source activate python-isal
+
+python: 3.6  # Use the oldest supported version of python as default.
+script:
+    - tox -e $TOX_ENV
+matrix:
+  include:
+    # TEST DOCS AND LINTING
+    # Use default python3 version here.
+    - env: TOX_ENV=docs
+    - env: TOX_ENV=lint
+    # UNIT TESTS
+    # On most recent versions of python.
+    - env: TOX_ENV=py36
+      after_success:
+        - pip install codecov
+        - codecov -v  # -v to make sure coverage upload works.
+    - python: 3.7
+      env: TOX_ENV=py37
+    - python: 3.8
+      env: TOX_ENV=py38

CHANGELOG.rst

@@ -0,0 +1,21 @@
+==========
+Changelog
+==========
+
+.. Newest changes should be on top.
+
+.. This document is user facing. Please word the changes in such a way
+.. that users understand how the changes affect the new version.
+
+version 0.1.0
+-----------------
++ Publish API documentation on readthedocs.
++ Add API documentation.
++ Ensure the igzip module is fully compatible with the gzip stdlib module.
++ Add compliance tests from CPython to ensure isal_zlib and igzip are validated
+  to the same standards as the zlib and gzip modules.
++ Added a working gzip app using ``python -m isal.igzip``
++ Add test suite that tests all possible settings for functions on the
+  isal_zlib module.
++ Create igzip module which implements all gzip functions and methods.
++ Create isal_zlib module which implements all zlib functions and methods.

README.md

@@ -0,0 +1,91 @@
+.. image:: https://readthedocs.org/projects/python-isal/badge
+   :target: https://python-isal.readthedocs.io
+   :alt:
+
+
+python-isal
+===========
+
+Faster zlib and gzip compatible compression and decompression
+by providing python bindings for the isa-l library.
+
+This package provides Python bindings for the `isa-l
+<https://github.com/intel/isa-l>`_ library. The Intel Infrastructure Storage
+Acceleration Library (isa-l) implements several key algorithms in `assembly
+language <https://en.wikipedia.org/wiki/Assembly_language>`_. This includes
+a variety of functions to provide zlib/gzip-compatible compression.
+
+``python-isal`` provides the bindings by offering an ``isal_zlib`` and
+``igzip`` module which are usable as drop-in replacements for the ``zlib``
+and ``gzip`` modules from the stdlib (with some minor exceptions, see below).
+
+Installation
+------------
+
+isa-l version 2.26.0 or higher is needed. This includes bindings for the
+adler32 function.
+
+isa-l is available in numerous Linux distro's as well as on conda via the
+conda-forge channel. Checkout the `ports documentation
+<https://github.com/intel/isa-l/wiki/Ports--Repos>`_ on the isa-l project wiki
+to find out how to install it.
+
+The latest development version of python-isal can be installed with
+
+.. code-block::
+
+    pip install git+https://github.com/rhpvorderman/python-isal.git
+
+Usage
+-----
+
+Python-isal has faster versions of the stdlib's ``zlib`` and ``gzip`` module
+these are called ``isal_zlib`` and ``igzip`` respectively.
+
+They can be imported as follows
+
+.. code-block:: python
+
+    from isal import isal_zlib
+    from isal import igzip
+
+``isal_zlib`` and ``igzip`` were meant to be used as drop in replacements so
+their api and functions are the same as the stdlib's modules. Except where
+isa-l does not support the same calls as zlib (See differences below).
+
+A full API documentation can be found on `our readthedocs page
+<https://python-isal.readthedocs.io>`_.
+
+``python -m isal.igzip`` implements a simple gzip-like command line
+application (just like ``python -m gzip``).
+
+Differences with zlib and gzip modules
+--------------------------------------
+
++ Compression level 0 in ``zlib`` and ``gzip`` means **no compression**, while
+  in ``isal_zlib`` and ``igzip`` this is the **lowest compression level**.
+  This is a design choice that was inherited from the isa-l library.
++ Compression levels range from 0 to 3, not 1 to 9.
++ ``isal_zlib.crc32`` and ``isal_zlib.adler32`` do not support negative
+  numbers for the value parameter.
++ ``zlib.Z_DEFAULT_STRATEGY``, ``zlib.Z_RLE`` etc. are exposed as
+  ``isal_zlib.Z_DEFAULT_STRATEGY``, ``isal_zlib.Z_RLE`` etc. for compatibility
+  reasons. However, ``isal_zlib`` only supports a default strategy and will
+  give warnings when other strategies are used.
++ ``zlib`` supports different memory levels from 1 to 9 (with 8 default).
+  ``isal_zlib`` supports memory levels smallest, small, medium, large and
+  largest. These have been mapped to levels 1, 2-3, 4-6, 7-8 and 9. So
+  ``isal_zlib`` can be used with zlib compatible memory levels.
++ ``isal_zlib`` only supports ``FLUSH``, ``SYNC_FLUSH`` and ``FULL_FLUSH``
+  ``FINISH`` is aliased to ``FULL_FLUSH`` (and works correctly as such).
++ ``isal_zlib`` has a ``compressobj`` and ``decompressobj`` implementation.
+  However, the unused_data and unconsumed_tail for the Decompress object, only
+  work properly when using gzip compatible compression. (25 <= wbits <= 31).
++ The flush implementation for the Compress object behavious differently from
+  the zlib equivalent.
+
+Contributing
+------------
+Please make a PR or issue if you feel anything can be improved. Bug reports
+are also very welcome. Please report them on the `github issue tracker
+<https://github.com/rhpvorderman/python-isal/issues>`_.

README.rst

@@ -0,0 +1,114 @@
+import argparse
+import gzip
+import timeit
+import zlib
+from pathlib import Path
+from typing import Dict
+
+from isal import igzip, isal_zlib  # noqa: F401 used in timeit strings
+
+DATA_DIR = Path(__file__).parent / "tests" / "data"
+COMPRESSED_FILE = DATA_DIR / "test.fastq.gz"
+with gzip.open(str(COMPRESSED_FILE), mode="rb") as file_h:
+    data = file_h.read()
+
+sizes: Dict[str, bytes] = {
+    "0b": b"",
+    "8b": data[:8],
+    "128b": data[:128],
+    "1kb": data[:1024],
+    "8kb": data[:8 * 1024],
+    "16kb": data[:16 * 1024],
+    "32kb": data[:32 * 1024],
+    "64kb": data[:64 * 1024],
+    # "128kb": data[:128*1024],
+    # "512kb": data[:512*1024]
+}
+compressed_sizes = {name: zlib.compress(data_block)
+                    for name, data_block in sizes.items()}
+
+compressed_sizes_gzip = {name: gzip.compress(data_block)
+                         for name, data_block in sizes.items()}
+
+
+def show_sizes():
+    print("zlib sizes")
+    print("name\t" + "\t".join(str(level) for level in range(-1, 10)))
+    for name, data_block in sizes.items():
+        orig_size = max(len(data_block), 1)
+        rel_sizes = (
+            str(round(len(zlib.compress(data_block, level)) / orig_size, 3))
+            for level in range(-1, 10))
+        print(name + "\t" + "\t".join(rel_sizes))
+
+    print("isal sizes")
+    print("name\t" + "\t".join(str(level) for level in range(0, 4)))
+    for name, data_block in sizes.items():
+        orig_size = max(len(data_block), 1)
+        rel_sizes = (
+            str(round(len(isal_zlib.compress(data_block, level)) / orig_size,
+                      3))
+            for level in range(0, 4))
+        print(name + "\t" + "\t".join(rel_sizes))
+
+
+def benchmark(name: str,
+              names_and_data: Dict[str, bytes],
+              isal_string: str,
+              zlib_string: str,
+              number: int = 10_000,
+              **kwargs):
+    print(name)
+    print("name\tisal\tzlib\tratio")
+    for name, data_block in names_and_data.items():
+        timeit_kwargs = dict(globals=dict(**globals(), **locals()),
+                             number=number, **kwargs)
+        isal_time = timeit.timeit(isal_string, **timeit_kwargs)
+        zlib_time = timeit.timeit(zlib_string, **timeit_kwargs)
+        isal_nanosecs = round(isal_time * (1_000_000 / number), 2)
+        zlib_nanosecs = round(zlib_time * (1_000_000 / number), 2)
+        ratio = round(isal_time / zlib_time, 2)
+        print("{0}\t{1}\t{2}\t{3}".format(name,
+                                          isal_nanosecs,
+                                          zlib_nanosecs,
+                                          ratio))
+
+
+# show_sizes()
+
+def argument_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--all", action="store_true")
+    parser.add_argument("--checksums", action="store_true")
+    parser.add_argument("--functions", action="store_true")
+    parser.add_argument("--gzip", action="store_true")
+    return parser
+
+
+if __name__ == "__main__":
+    args = argument_parser().parse_args()
+    if args.checksums or args.all:
+        benchmark("CRC32", sizes,
+                  "isal_zlib.crc32(data_block)",
+                  "zlib.crc32(data_block)")
+
+        benchmark("Adler32", sizes,
+                  "isal_zlib.adler32(data_block)",
+                  "zlib.adler32(data_block)")
+    if args.functions or args.all:
+        benchmark("Compression", sizes,
+                  "isal_zlib.compress(data_block, 1)",
+                  "zlib.compress(data_block, 1)")
+
+        benchmark("Decompression", compressed_sizes,
+                  "isal_zlib.decompress(data_block)",
+                  "zlib.decompress(data_block)")
+
+    if args.gzip or args.all:
+        benchmark("Compression", sizes,
+                  "igzip.compress(data_block, 1)",
+                  "gzip.compress(data_block, 1)")
+
+        benchmark("Decompression", compressed_sizes_gzip,
+                  "igzip.decompress(data_block)",
+                  "gzip.decompress(data_block)")

benchmark.py

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)