Adopt the array api #885

mwcraig · 2025-03-19T15:17:48Z

This PR makes the necessary changes to adopt the array API. There are still some things to sort out:

Run tests mostly against numpy, but do one run against jax (or some other non-numpy array library)
Check for remaining invocations of numpy in the code.
Check whether changing .data_arr.mask to .data_arr_mask constitutes an API change. If it does (and I think it does) then patch things up so that .data_arr.mask points to .data_arr_mask
Clean up any remaining handling of immutable arrays (array_api_extra makes handling those easy)

Once those are done the next step will be to find an outside reviewer for this...

codecov · 2025-03-20T15:04:20Z

Codecov Report

Attention: Patch coverage is 98.91304% with 2 lines in your changes missing coverage. Please review.

Project coverage is 97.82%. Comparing base (aca218e) to head (c8f8da3).
Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
ccdproc/core.py	97.70%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #885      +/-   ##
==========================================
- Coverage   97.87%   97.82%   -0.05%     
==========================================
  Files           7        7              
  Lines        1271     1334      +63     
==========================================
+ Hits         1244     1305      +61     
- Misses         27       29       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

These are necessary because upstream dependencies (astropy) explicitly check for numpy arrays rather than using the array API.

This is really only needed for numpy 1.X since numpy 2.X is array API compliant.

Copilot

Pull Request Overview

This PR transitions the codebase to use the Array API via array_api_compat and array_api_extra, updates CI and test environments to run against non-NumPy backends (JAX, Dask), and renames the combiner’s mask attribute from .data_arr.mask to .data_arr_mask.

Swap direct NumPy calls for array_api_compat calls and inject xp namespaces.
Extend tox.ini, CI workflows, and pyproject.toml to test with JAX, Dask, and bump supported Python/NumPy versions.
Rename .data_arr.mask to .data_arr_mask in code and documentation.

Reviewed Changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tox.ini	Added JAX/Dask `CCDPROC_ARRAY_LIBRARY` settings for array tests.
pyproject.toml	Bumped `requires-python` and NumPy version; added Array API deps.
docs/image_combination.rst	Updated examples to use `.data_arr_mask` instead of `.mask`.
ccdproc/combiner.py	Introduced `data_arr_mask`, removed masked array, use Array API.
ccdproc/core.py	Replaced NumPy calls with `array_api_compat` namespace calls.

Comments suppressed due to low confidence (2)

ccdproc/combiner.py:190

For backward compatibility with code expecting self.data_arr.mask, add an alias or property so that self.data_arr.mask returns self.data_arr_mask.

        self.data_arr_mask = xp.array(mask_list, dtype=bool)

docs/image_combination.rst:107

The documentation now references data_arr_mask instead of data_arr.mask. Ensure the API change is documented and consider providing a deprecation note or alias for data_arr.mask.

    >>> new_n_masked = combiner.data_arr_mask.sum()

Copilot · 2025-06-24T16:45:23Z

ccdproc/tests/test_cosmicray.py

-    crflux = 10 * scale * rng.random(NCRAYS) + (threshold + 15) * scale
+    crflux = xp.asarray(10 * scale * rng.random(ncrays) + (threshold + 15) * scale)
+
+    # Some array libraris (Dask) do not support setting individual elements,


Fix typo in comment: "libraris" should be "libraries".

Suggested change

# Some array libraris (Dask) do not support setting individual elements,

# Some array libraries (Dask) do not support setting individual elements,

Copilot · 2025-06-24T16:45:24Z

ccdproc/tests/test_cosmicray.py

+    crflux = xp.asarray(10 * scale * rng.random(ncrays) + (threshold + 15) * scale)
+
+    # Some array libraris (Dask) do not support setting individual elements,
+    # so use nuympy.


Fix typo in comment: "nuympy" should be "NumPy".

Suggested change

# so use nuympy.

# so use NumPy.

eteq · 2025-06-24T21:51:22Z

As a quick FYI I ran the test suite in this PR on a machine with a NVidia GPU with CCDPROC_ARRAY_LIBRARY=cupy and got a whole pile of errors (sent the log directly to @mwcraig but can also share snippets here if someone else wants to see them)

nstarman · 2025-06-24T18:15:42Z

pyproject.toml

@@ -145,6 +148,7 @@ filterwarnings= [
    "ignore:numpy\\.ufunc size changed:RuntimeWarning",
    "ignore:numpy.ndarray size changed:RuntimeWarning",
    "ignore:`np.bool` is a deprecated alias for the builtin `bool`:DeprecationWarning",
+    "ignore:invalid value encountered in sqrt:RuntimeWarning",


I'd suggest adding a comment recording why the ignore is necessary, which helps figuring out when it isn't.

nstarman · 2025-06-25T18:17:18Z

ccdproc/combiner.py

 try:
    import bottleneck as bn
 except ImportError:
    HAS_BOTTLENECK = False  # pragma: no cover
 else:
    HAS_BOTTLENECK = True

+import array_api_compat
+import array_api_extra as xpx


I like the alias.

nstarman · 2025-06-25T18:20:07Z

ccdproc/combiner.py

+        mask_list = []
+        for ccd in ccd_list:
            if ccd.mask is not None:
-                self.data_arr.mask[i] = ccd.mask
+                mask_list.append(ccd.mask)
            else:
-                self.data_arr.mask[i] = ma.zeros(default_shape)
+                mask_list.append(xp.zeros(default_shape))


I think this can be sped up through list comprehension and made a one liner.
mask_list = [ccd.mask if ccd.mask is not None else xp.zeros(default_shape) for ccd in ccd_list]

nstarman · 2025-06-25T18:23:58Z

pyproject.toml

@@ -8,17 +8,19 @@ dynamic = ["version"]
 description = "Astropy affiliated package"
 readme = "README.rst"
 license = { text = "BSD-3-Clause" }
-requires-python = ">=3.8"
+requires-python = ">=3.11"


Adopt Spec 0? Has that been adopted?

mwcraig added this to the 2.5.0 milestone Mar 19, 2025

mwcraig added the enhancement label Mar 19, 2025

mwcraig added 27 commits March 27, 2025 13:04

Initial attempt to adopt array API

9b1be4c

A couple more changes

4ee3449

Add function for check if something is an array

978d102

Remove numpy from core

99c4343

WIP rewrite of a boolean index

dce098d

WIP testing updates

63d90f0

Change more almost_equal to allclose

1e1101f

Use the random number generator from numpy

ba53cbf

Remove more in-place array operations

7c86586

Set a reasonable tolerance value in float comparisons

c36830c

Continue to use numpy arrays in a few places

e27a120

These are necessary because upstream dependencies (astropy) explicitly check for numpy arrays rather than using the array API.

Refactor warning handling for numerical warnings

01f5e36

Avoid explicit use of numpy masked arrays

a911a0a

Rewrite test to not modify array in-place

57709e2

Use assert_allclose instead of older alternatives

85dcf1a

Initial attempt to adopt array API in combiner

4a031f2

Write a mask-aware sum function that is array API compatible

1482c14

Eliminate most use of numpy in tests

5bc8173

Fix a couple calls to numpy masked array

a1eeb50

One more workaround for immutable arrays

48899cd

Fix up dependencies and test environment setup

12422b6

Update minimum python to 3.10

a0f3a8a

Fix mask access

6425b96

Ignore warnings about negative values in square root

64894c3

Update several minimum dependencies

ab80450

Fix linting errors

48cca23

Drop unnecessary import

d776c1a

mwcraig added 11 commits March 27, 2025 13:04

Drop unneeded test

3daabc0

Skip memory tests if jax is installed

5b947ea

Explain why numpy is still used in image_collection

3e87408

Drop numpy import in combiner

7386936

Use array_api_extra to handle immutable arrays

98df4b7

Use a consistent namespace for arrays

fc1c11f

Clean up a couple more cases to use array_api_extra

76082d5

Change where bottleneck is test on GitHub Actions

6b96410

Skip coverage of one function

ed066ba

Remove unused argument and logic

6acc0df

Add a test

04293b7

mwcraig force-pushed the explore-array-api branch from 85625f0 to 04293b7 Compare March 27, 2025 18:05

mwcraig added 7 commits June 18, 2025 16:42

Use tox environment to handle testing of different array libraries

9fbf49c

Convert combiner tests to use Array API

fa9bd6b

Remove unnecessary copy argument

83c6aec

Use the array_api_compt numpy namespace instead of numpy

d61486d

This is really only needed for numpy 1.X since numpy 2.X is array API compliant.

Add test against dask and fix bugs uncovered by tests

c043ed1

Add dask test to CI

d0e5b9a

Fix some errors introduced when changing the tests for dask

85850a5

mwcraig marked this pull request as ready for review June 24, 2025 14:55

mwcraig changed the title ~~WIP: Adopt the array api~~ Adopt the array api Jun 24, 2025

mwcraig requested a review from Copilot June 24, 2025 16:42

Copilot AI reviewed Jun 24, 2025

View reviewed changes

Allow cupy for testing

c8f8da3

nstarman reviewed Jun 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Adopt the array api #885

Adopt the array api #885

Uh oh!

mwcraig commented Mar 19, 2025 •

edited

Loading

Uh oh!

codecov bot commented Mar 20, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jun 24, 2025

Uh oh!

Copilot AI Jun 24, 2025

Uh oh!

eteq commented Jun 24, 2025

Uh oh!

nstarman Jun 24, 2025

Uh oh!

nstarman Jun 25, 2025

Uh oh!

nstarman Jun 25, 2025

Uh oh!

nstarman Jun 25, 2025

Uh oh!

Uh oh!

	# Some array libraris (Dask) do not support setting individual elements,
	# Some array libraries (Dask) do not support setting individual elements,

Uh oh!

Adopt the array api #885

Are you sure you want to change the base?

Adopt the array api #885

Uh oh!

Conversation

mwcraig commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

eteq commented Jun 24, 2025

Uh oh!

nstarman Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

nstarman Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

nstarman Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

nstarman Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mwcraig commented Mar 19, 2025 •

edited

Loading

codecov bot commented Mar 20, 2025 •

edited

Loading