Skip to content

Commit 9d74007

Browse files
committed
dev: faiss-cpu>=1.12.0 + remove monkeypatch (#16) [skip ci]
Signed-off-by: Élie Goudout <[email protected]>
1 parent 6205fa9 commit 9d74007

File tree

4 files changed

+24
-74
lines changed

4 files changed

+24
-74
lines changed

docs/src/user_guide/installation_compatibility.rst

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,21 +18,18 @@ If you wish to install from source or wheels manually, you can download `release
1818
1919
OS Compatibility
2020
----------------
21-
The library is available and functional on Ubuntu, Windows and MacOS. However, note the following regarding ``faiss``-related features.
22-
23-
- They are **partially** supported on **Windows** and their use will trigger a warning: due to non-Python dependency conflicts, there might be uncontrolled and silent concurrency issues, potentially altering results. For more information, see this nice ressource about `native dependencies <https://pypackaging-native.github.io/key-issues/native-dependencies>`_. Note that we still run full CI tests on Windows.
24-
- They are **not officially supported** on **MacOS** and might generate random crashes. We should match the current Windows support status *relatively soon*.
21+
The library is available and functional on **Ubuntu**, **Windows** and *MacOS* (minus ``faiss``-related features ─ feel free to show your interest `here <https://github.com/ThalesGroup/scio/issues/2>`_).
2522

2623
Framework Compatiblity
2724
----------------------
2825
Although confidence scores are not a particularity of Neural Networks, ``scio`` is precisely developed to handle such models. More precisely, only *PyTorch* models (inheriting from ``torch.nn.Module``) are currently supported.
2926

3027
GPU Compatibility
3128
-----------------
32-
Our library is **fully compatible** and tested with CUDA devices for native use of GPU acceleration! Note however that features using ``faiss`` currently use CPU-bound indexes, introducing a potential GPU > CPU > GPU bottleneck. This may be improved in future versions. Do not hesitate to express interest by opening a related issue.
29+
Our library is **fully compatible** and tested with CUDA devices for native use of GPU acceleration! Note however that features using ``faiss`` currently use CPU-bound indexes, introducing a potential GPU > CPU > GPU bottleneck. This may be improved in future versions ─ feel free to show your interest `here <https://github.com/ThalesGroup/scio/issues/18>`_.
3330

3431
Supported Data Types
3532
--------------------
3633
The package is **fully compatible** and tested with ``torch.half`` (float16), ``torch.float`` (float32) and ``torch.double`` (float64) data types. However, we **discourage** the use of gradient-based algorithms with ``torch.half`` data, as they can easily generate irrelevant results due to potential ``nan`` values. Finally, note that ``faiss``-related operations temporarily convert to 32 bits data, regardless of the input type.
3734

38-
There is currently no official support for ``torch.bfloat16`` even though many features may be compatible. Do not hesitate to express interest by opening a related issue.
35+
There is currently no official support for ``torch.bfloat16`` even though many features may be compatible. Feel free to show your interest by opening a related issue.

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ classifiers = [
2929
]
3030
dependencies = [
3131
"dill>=0.3.8",
32-
"faiss-cpu>=1.11",
32+
"faiss-cpu>=1.12",
3333
"lazy-loader>=0.4",
3434
"matplotlib>=3.10",
3535
"numpy>=2.2.2",

scio/__init__.py

Lines changed: 0 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
"""scio package."""
22

3-
import os
4-
import platform
53
from importlib.metadata import PackageNotFoundError, version
64

75
try:
@@ -13,48 +11,3 @@
1311

1412
# Lazily load from adjacent `.pyi` file
1513
__getattr__, __dir__, __all__ = lazy.attach_stub(__name__, __file__)
16-
17-
# Monkeypatch the use of ``faiss`` on non-Linux platforms
18-
if (
19-
platform.system() != "Linux"
20-
and os.environ.get("KMP_DUPLICATE_LIB_OK", None) != "TRUE"
21-
): # pragma: no cover
22-
import importlib
23-
import sys
24-
import types
25-
from warnings import warn
26-
27-
null = object()
28-
29-
class DummyFaiss(types.ModuleType):
30-
"""Dummy faiss to intercept faiss loading."""
31-
32-
__faiss = null
33-
34-
def __getattribute__(self, attr: str) -> object:
35-
"""Intercept loading, warn & enable KMP_DUPLICATE_LIB_OK."""
36-
__faiss = super().__getattribute__(f"_{type(self).__name__}__faiss")
37-
38-
# If necessary, warn and load faiss
39-
if __faiss is null:
40-
msg = (
41-
"On non-Linux platforms, the use of any `faiss`-dependant feature "
42-
"is partially supported by `scio`: you might experience slowdowns "
43-
"or even incorrect results! This is due to a non-Python dependency "
44-
"conflict regarding OpenMP (see https://pypackaging-native.github"
45-
".io/key-issues/native-dependencies for interesting ressources). "
46-
"As a monkeypatch, we enable the `KMP_DUPLICATE_LIB_OK`."
47-
)
48-
warn(msg, stacklevel=2)
49-
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
50-
51-
if sys.modules.pop("faiss") is not self:
52-
msg = "Dummy faiss module should only be defined during scio init"
53-
raise RuntimeError(msg)
54-
55-
self.__faiss = __faiss = importlib.import_module("faiss")
56-
sys.modules["faiss"] = __faiss
57-
58-
return __faiss.__getattribute__(attr)
59-
60-
sys.modules["faiss"] = DummyFaiss("faiss")

uv.lock

Lines changed: 20 additions & 20 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)