Speed up categorical regressor with numba #3353
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
❌ 7 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
ilan-gold
requested changes
Nov 11, 2024
tests/test_preprocessing.py
Outdated
| np.testing.assert_array_almost_equal(adata.X, tester) | ||
|
|
||
|
|
||
| def test_regressor_categorical(): |
There was a problem hiding this comment.
I would
- explain why this test exists (to test against a previous implementation? I am impartial whether it's necessary TBH since we are already testing for reproducibility, could see getting rid of this)
- refactor the "Create org regressors" into a helper function like
create_original
There was a problem hiding this comment.
I can see your point here
There was a problem hiding this comment.
Do you have an an opinion on the first point? Is this test necessary? If so, perhaps a comment then?
ilan-gold
requested changes
Nov 12, 2024
There was a problem hiding this comment.
I think this is missing: #3353 (comment) and the first part of https://github.com/scverse/scanpy/pull/3353/files#r1836830351
ilan-gold
requested changes
Nov 12, 2024
src/scanpy/preprocessing/_simple.py
Outdated
| @@ -722,13 +737,13 @@ def regress_out( | |||
| "we regress on the mean for each category." | |||
| ) | |||
| logg.debug("... regressing on per-gene means within categories") | |||
| regressors = np.zeros(X.shape, dtype="float32") | |||
| # Create numpy array's from categorical variable | |||
| cats = np.int64(len(adata.obs[keys[0]].cat.categories)) | |||
There was a problem hiding this comment.
Also comment why np.int64
There was a problem hiding this comment.
because it has be done because of weird typing from pandas. So this ensures that it works within the kernel
There was a problem hiding this comment.
so len doesn’t return a Python int? That’s a pandas bug.
Co-authored-by: Ilan Gold <[email protected]>
Co-authored-by: Ilan Gold <[email protected]>
Co-authored-by: Ilan Gold <[email protected]>
ilan-gold
requested changes
Nov 12, 2024
tests/test_preprocessing.py
Outdated
| np.testing.assert_array_almost_equal(adata.X, tester) | ||
|
|
||
|
|
||
| def test_regressor_categorical(): |
There was a problem hiding this comment.
Do you have an an opinion on the first point? Is this test necessary? If so, perhaps a comment then?
flying-sheep
requested changes
Nov 14, 2024
|
I renamed one variable to make is clearer what it is. Added some comments that should add more context what the code is doing. |
ilan-gold
reviewed
Feb 11, 2025
Co-authored-by: Ilan Gold <[email protected]>
ilan-gold
requested changes
Feb 11, 2025
src/scanpy/preprocessing/_simple.py
Outdated
| X: np.ndarray, number_categories: int, cat_array: np.ndarray | ||
| ) -> np.ndarray: | ||
| # create regressor matrix for categorical variables | ||
| regressors = np.zeros(X.shape, dtype=X.dtype) |
There was a problem hiding this comment.
check dtype for behavior with integer dtype i.e., need to ensure this is a floating point matrix
There was a problem hiding this comment.
Yes! discoverered a bug! regressors needs to be float32 to match old behavior, not sure why it was hardcoded
flying-sheep
changed the title
Co-authored-by: Philipp A. <[email protected]>
Co-authored-by: Ilan Gold <[email protected]>
… create_cat_regressor
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Use numba to create the regressor for categorical regression