Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEST: enabled test_memory_usage for array API inputs #2041

Draft
wants to merge 113 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
bec3ebb
BUG: fixing circular import in daal4py/sklearnex device_offloading
samir-nasibli May 13, 2024
9052a73
removed onedal4py sklearnex dependence
samir-nasibli May 15, 2024
b070c89
minor fix
samir-nasibli May 15, 2024
0fbc680
Merge branch 'intel:main' into fix/device_offload
samir-nasibli May 16, 2024
8a6c4da
Merge branch 'intel:main' into fix/device_offload
samir-nasibli Jun 1, 2024
89ac903
Merge branch 'intel:main' into fix/device_offload
samir-nasibli Jun 6, 2024
01d73da
minor update
samir-nasibli Jun 8, 2024
d4587aa
Merge branch 'intel:main' into fix/device_offload
samir-nasibli Jun 8, 2024
b5f8921
added daal4py.sklearn._config for exposing sklearnex settings
samir-nasibli Jun 8, 2024
46dea79
Merge branch 'intel:main' into fix/device_offload
samir-nasibli Jun 11, 2024
8ce81ea
removed daal4py device_offloading
samir-nasibli Jun 11, 2024
fc01bae
integrating changes of device offloading for sklearnex primitives/est…
samir-nasibli Jun 11, 2024
18e6599
minor fixes
samir-nasibli Jun 11, 2024
4318064
minor fix for daal4py/sklearn/linear_model/_coordinate_descent.py
samir-nasibli Jun 11, 2024
864c1a1
minor fix for daal4py/sklearn/linear_model/_linear.py
samir-nasibli Jun 11, 2024
4142bf1
fix for sklearnex/_device_offload.py
samir-nasibli Jun 11, 2024
25d87bc
fix for onedal._config
samir-nasibli Jun 11, 2024
778b88d
wrapping daal4py.sklearne Kmeans with onedal4py's support_usm_ndarray
samir-nasibli Jun 11, 2024
ecd731d
ENH: functional support for Array API
samir-nasibli Jun 12, 2024
f18070f
minor update for support_usm_ndarray decorator
samir-nasibli Jun 14, 2024
df682e6
update sklearnex/dispatcher.py
samir-nasibli Jun 14, 2024
23c84f4
fixed dispatcher
samir-nasibli Jun 14, 2024
0483c5c
fixed decorator name
samir-nasibli Jun 14, 2024
459e638
minor update for onedal/_device_offload.py
samir-nasibli Jun 15, 2024
75f9c4b
minor updates
samir-nasibli Jun 18, 2024
27b7a7c
update docstrings for onedal._config._get_config
samir-nasibli Jun 18, 2024
dec939e
reverted changes for LogReg refactoring
samir-nasibli Jun 19, 2024
38cc61e
using _get_config instead of _get_onedal_threadlocal_config
samir-nasibli Jun 19, 2024
cea22ac
minor fix of _get_config
samir-nasibli Jun 19, 2024
1bb3697
removed TODO, that covered by ticket
samir-nasibli Jun 20, 2024
9ec11b9
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Jun 20, 2024
4179f22
added todo comments
samir-nasibli Jun 21, 2024
694d94b
moved out from _DataParallelInteropPolicy init import of DummySyclQueue
samir-nasibli Jun 20, 2024
8bf2585
removed outdated comment; will be covered on #1813
samir-nasibli Jun 20, 2024
e903fbb
removed TODO comment from ridge.py
samir-nasibli Jun 20, 2024
1491b38
Added ElasticNet, Lasso, Ridge into sklearnex patching map
samir-nasibli Jun 21, 2024
81de19e
Merge branch 'main' into enh/functional_array_api
samir-nasibli Jun 27, 2024
3dd5000
lint
samir-nasibli Jun 27, 2024
adc6e68
removed debug print
samir-nasibli Jun 27, 2024
652d014
enabled more array api test
samir-nasibli Jun 27, 2024
d9f2ef8
Merge branch 'main' into enh/functional_array_api
samir-nasibli Jul 5, 2024
85c279c
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Jul 9, 2024
14c675d
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Jul 11, 2024
669d68f
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Jul 12, 2024
c6e158c
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Jul 15, 2024
ad42750
created seperate array api module
samir-nasibli Jul 16, 2024
7bef6c4
currently disabled array_api for test_memory_leaks
samir-nasibli Jul 17, 2024
88e2a22
update _convert_to_dataframe
samir-nasibli Jul 17, 2024
383ce0d
Merge branch 'main' into enh/functional_array_api
samir-nasibli Jul 17, 2024
d8d0dc4
linting
samir-nasibli Jul 17, 2024
45b920f
update condition for _transfer_to_host
samir-nasibli Jul 18, 2024
84da80a
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Jul 22, 2024
eda98a4
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Jul 26, 2024
f74ff13
fixed for bs and ridge
samir-nasibli Jul 26, 2024
7e83e4a
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Aug 5, 2024
4b86143
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Aug 6, 2024
b8348ee
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Aug 7, 2024
0300f72
update fallback when array_api_dispatch enabled
samir-nasibli Aug 8, 2024
5cae80b
refactor sklearnex get_namespace usage
samir-nasibli Aug 8, 2024
ac3aacb
updated array apiconditions for get_dataframes_and_queues
samir-nasibli Aug 8, 2024
80da711
fix import in test_memory_usage.py
samir-nasibli Aug 8, 2024
1de8fc9
first temp commit address py312 fails
samir-nasibli Aug 9, 2024
621dfc2
small tmp workaround for py312
samir-nasibli Aug 9, 2024
d2f4383
removed from_dlpack from the non-zero support logic
samir-nasibli Aug 10, 2024
16ddb69
fixing tests for incremenatal estimators
samir-nasibli Aug 11, 2024
9be33c2
using asarray instead of dlpack conversions
samir-nasibli Aug 12, 2024
e2fa37a
FIX: fixing spmd tests utilities for dpctl inputs
samir-nasibli Aug 12, 2024
016f550
Deselect LOF stability test with array api
samir-nasibli Aug 12, 2024
3e833d7
Revert "Deselect LOF stability test with array api"
samir-nasibli Aug 12, 2024
d0fc95d
MAINT: minor refactoring and docstrings added
samir-nasibli Aug 12, 2024
2ff418f
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Aug 26, 2024
df1efc6
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Aug 30, 2024
2094fad
refactoring
samir-nasibli Aug 30, 2024
3004faa
Merge branch 'main' into enh/functional_array_api
samir-nasibli Sep 2, 2024
0fcaafe
minor update for _extract_array_attr
samir-nasibli Sep 2, 2024
5521720
minor update
samir-nasibli Sep 2, 2024
7bd3df0
update conditions for support_array_api
samir-nasibli Sep 2, 2024
19df8f1
update dispatch for array api
samir-nasibli Sep 2, 2024
442826e
covered by tickets TODOs
samir-nasibli Sep 2, 2024
fc34a6c
Merge branch 'main' into enh/functional_array_api
samir-nasibli Sep 4, 2024
ec634c1
refactor _asarray
samir-nasibli Sep 4, 2024
844745f
update a bit logic
samir-nasibli Sep 4, 2024
752df22
addressed test failes
samir-nasibli Sep 4, 2024
fe38790
update docstring
samir-nasibli Sep 4, 2024
3dd9521
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Sep 4, 2024
6989049
Update _array_api.py
samir-nasibli Sep 4, 2024
047d698
minor update
samir-nasibli Sep 4, 2024
c5f8281
minor updatte try
samir-nasibli Sep 4, 2024
1b45a24
renamed wrapper for inputs support
samir-nasibli Sep 4, 2024
ac7288f
minor refactoring
samir-nasibli Sep 5, 2024
d5f3a35
Merge branch 'main' into enh/functional_array_api
samir-nasibli Sep 5, 2024
3915066
fix refactoring
samir-nasibli Sep 5, 2024
95fa66f
addressed test fails
samir-nasibli Sep 5, 2024
8a9e497
remove unnecessary comments
samir-nasibli Sep 5, 2024
c66836f
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Sep 5, 2024
3ac9c82
enabled transform_output check
samir-nasibli Sep 6, 2024
819aa8f
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Sep 6, 2024
aa2bf82
update use of transform_output flag for input handlers
samir-nasibli Sep 6, 2024
3539ee3
reverted changes for test_incremental_pca.py
samir-nasibli Sep 6, 2024
ad71cd0
Revert "update use of transform_output flag for input handlers"
samir-nasibli Sep 6, 2024
4634c73
Merge branch 'main' into enh/functional_array_api
samir-nasibli Sep 6, 2024
41d1efc
minor refactoring
samir-nasibli Sep 9, 2024
52e9257
fixing
samir-nasibli Sep 9, 2024
dd78da4
Merge branch 'intel:main' into enh/functional_array_api
samir-nasibli Sep 10, 2024
4a44e12
minor refactoring
samir-nasibli Sep 10, 2024
defcecd
TEST: enabled test_memory_usage for array API inputs
samir-nasibli Sep 10, 2024
55a9471
Merge branch 'intel:main' into test/array_api_testing
samir-nasibli Sep 10, 2024
d114886
backup some development
samir-nasibli Sep 10, 2024
c9ff599
Merge branch 'intel:main' into test/array_api_testing
samir-nasibli Sep 10, 2024
7a15963
Merge branch 'intel:main' into test/array_api_testing
samir-nasibli Oct 1, 2024
cd6739c
Merge branch 'main' into test/array_api_testing
samir-nasibli Oct 18, 2024
f2d8d36
update;draft
samir-nasibli Oct 18, 2024
d4ea9d5
Merge branch 'main' into test/array_api_testing
samir-nasibli Dec 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions sklearnex/tests/test_memory_usage.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@
from onedal import _is_dpc_backend
from onedal.tests.utils._dataframes_support import (
_convert_to_dataframe,
dpctl_available,
dpnp_available,
get_dataframes_and_queues,
)
from onedal.tests.utils._device_selection import get_queues, is_dpctl_device_available
Expand Down Expand Up @@ -167,10 +169,13 @@ def take(x, index, axis=0, queue=None):
return xp.take(
x, xp.asarray(index, usm_type="device", sycl_queue=x.sycl_queue), axis=axis
)
# TODO:
# re-impl _is_numpy_namespace
# elif array_api and not isinstance(x, np.ndarray):
elif array_api:
return xp.take(x, xp.asarray(index, device=x.device), axis=axis)
else:
return x.take(index, axis=axis)
return xp.take(x, xp.asarray(index), axis=axis)


def split_train_inference(kf, x, y, estimator, queue=None):
Expand Down Expand Up @@ -288,7 +293,7 @@ def _kfold_function_template(

@pytest.mark.parametrize("order", ["F", "C"])
@pytest.mark.parametrize(
"dataframe,queue", get_dataframes_and_queues("numpy,pandas,dpctl", "cpu")
"dataframe,queue", get_dataframes_and_queues("numpy,pandas,dpctl,array_api", "cpu")
)
@pytest.mark.parametrize("estimator", CPU_ESTIMATORS.keys())
@pytest.mark.parametrize("data_shape", data_shapes)
Expand Down
23 changes: 22 additions & 1 deletion sklearnex/utils/_array_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,20 @@

"""Tools to support array_api."""

try:
import array_api_compat

array_api_available = True
except ImportError:
array_api_available = False

import numpy as np

from daal4py.sklearn._utils import sklearn_check_version
from onedal.utils._array_api import _get_sycl_namespace

from .._config import get_config

if sklearn_check_version("1.2"):
from sklearn.utils._array_api import get_namespace as sklearn_get_namespace

Expand Down Expand Up @@ -76,7 +85,19 @@ def get_namespace(*arrays):

if sycl_type:
return xp, is_array_api_compliant
# TODO:
# correct condition.
# scikit-learn's get_namespace require config_contex(array_api_dispatch=True).
# elif sklearn_check_version("1.2") and get_config()["array_api_dispatch"]:
elif sklearn_check_version("1.2"):
return sklearn_get_namespace(*arrays)
namespace, is_array_api_compliant = sklearn_get_namespace(*arrays)
return namespace, is_array_api_compliant
# TODO:
# on PR 2096.
# elif array_api_available:
# namespace, is_array_api_compliant = array_api_compat.get_namespace(*arrays), True
# return namespace, is_array_api_compliant
# TODO:
# should be removed.
else:
return np, False
Loading