feat(asr): add Artifact Subspace Reconstruction module (standard, rASR, adaptive, Juggler)#36
Open
snesmaeili wants to merge 42 commits into
Open
feat(asr): add Artifact Subspace Reconstruction module (standard, rASR, adaptive, Juggler)#36snesmaeili wants to merge 42 commits into
snesmaeili wants to merge 42 commits into
Conversation
Honor max_mem_mb in calibrate_asr, process_asr, and AdaptiveASR processing by streaming covariances in chunks when the full covariance stack would exceed the requested budget. The full-memory path is preserved for small datasets, and diagnostics now expose memory_mode (full / rolling / chunked / riemannian), estimated_full_cov_bytes, peak_cov_buffer_bytes, and chunk_samples so downstream pipelines can audit the choice. Adds unit tests confirming that low-memory rolling/chunked paths reproduce the full-memory output bit-for-bit at atol=1e-10, plus the ASRpy blocksize-remainder=2 edge case.
Two reproducible validation harnesses for users with local EEG datasets: - scripts/run_asr_real_data_validation.py: runs every public ASR variant on local recordings, injects known burst artifacts, and reports CPU / RSS / memory-mode / metadata-preservation / artifact-reduction metrics as JSON, CSV, and Markdown. - scripts/run_asr_paper_validation.py: sweeps the ASR cutoff and reproduces the headline figures from Chang et al. 2020 (cutoff sweep, % windows-with-rejection, % background-variance reduced), Blum et al. 2019 (per-variant timing, frontal blink amplitude reduction), and Kim et al. 2025 (reference-data fraction, 1/f-bump PSD parameter). Documents the scripts and their outputs in docs/asr.rst.
Walks the user through ASR's calibration + burst-repair pipeline on a 32-channel, 60 s synthetic recording with eight injected bursts, then sweeps the cutoff parameter k from 1 to 100 to reproduce the Chang 2020 Figure 2a trade-off (% windows touched vs % background-variance reduced). Includes the canonical reconstruction equation X_clean = M (V_clean^T M)^+ V^T X, lists the Kothe pre-processing prerequisites, and points readers to the Riemannian, Juggler, and Adaptive variant examples.
meegkit.asr imports pyriemann at module load time; skipping with a clear message keeps the parity suite green on environments that have the python-meegkit checkout but not pyriemann installed.
Pure formatting changes from `ruff format`. No behavioral changes.
Adds `.serena/`, `.claude/`, `reports/`, `docs/asr/papers_md/`, the fir-specific sbatch wrapper, the local paper-conversion utility, and WIP tutorial notebooks staged on other branches. Prevents accidental commits of agent metadata, large validation outputs, and paper full-text mirrors.
…ule, harmonized API Complete the ASR estimator family and make it merge-ready: - New variants: JugglerASR (DBSCAN/GEV reference selection), AdaptiveASR moving-window mode (final_state + sliding), and the riemannian_windowed backend (per-window eigendecomposition that restores cutoff sensitivity while keeping the Riemannian robust calibration covariance). - Harmonize the public API across ASR/AdaptiveASR/JugglerASR: one fit/transform/fit_transform/partial_fit surface, unified get_diagnostics(), get_calibration_mask() (+ calibration_mask_kind_), and to_annotations(kind="repair"|"rejection"|"calibration"); consistent calibration_* parameter names. Removes the redundant update/reconstruct aliases. - New mne_denoise.viz.asr module with 10 plot_asr_* helpers following the package viz contract (return (fig, ax), MNE or ndarray inputs, ax=/show=/fname=); wired into mne_denoise.viz. - Docs (api.rst, asr.rst) and a new visualization gallery example. - Tests: variant parity/self-consistency, robustness sweeps, viz contract, and targeted coverage (90% statement coverage on the asr package and viz/asr).
… tooling Reproducibility scripts that validate the ASR variants against their source papers and datasets (Chang 2020, Mullen 2013, Blum 2019, Tsai, Kim 2025) and study robustness across substrates (real-EOG, long recordings, channel-count extremes, per-trial ICA). Developer tooling under scripts/; not imported by the shipped package.
…ripts PEP 604 isinstance unions (UP038) and minor lint cleanups in non-ASR modules surfaced by a repo-wide ruff pass. No functional changes.
…e clash examples/asr/plot_01_basic_usage.py shared a basename with examples/zapline/plot_01_basic_usage.py. sphinx-gallery flattens generated output by basename, so the duplicate triggered a warning that failed the docs build under `-W`. Renamed to plot_01_asr_basics.py (unique across the gallery).
The CI lint job installs the latest ruff, whose formatter differs slightly from the locally pinned version. Reformat the affected files; no functional changes.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #36 +/- ##
==========================================
+ Coverage 95.93% 96.84% +0.90%
==========================================
Files 41 58 +17
Lines 4599 6901 +2302
Branches 849 1179 +330
==========================================
+ Hits 4412 6683 +2271
- Misses 88 103 +15
- Partials 99 115 +16
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Add coverage for the viz/asr.py error/edge branches (empty cutoff sweep, missing sfreq in PSD comparison, unfitted/degenerate component-reconstruction inputs, repair-span edge cases, epochs concatenation, ax reuse) and the JugglerASR constructor-level parameter guards.
`patch: target: auto` required new code to match the mature package's ~96% per-line coverage, which is an unrealistic bar for a large feature's defensive guards, plotting branches, and deep numerical fallbacks. Use a fixed 85% (with a 2% tolerance) instead -- well above Codecov's 70% default, and comfortably met by the new, well-tested code. The project gate (80%) is unchanged.
BabaSanfour
requested changes
Jun 2, 2026
Resolve the viz/spectra.py conflict by taking main's refactored version, and adapt the ASR module to main's centralized channel-picking API. main's extract_data_from_mne now auto-picks a homogeneous channel type by default and returns (data, sfreq, mne_type, orig_inst, picks, ch_names); the ASR estimators do their own channel selection, so every call site passes auto_pick=False (full data) and unpacks the new 6-tuple. Full test suite green (693 passed).
… split) - Remove planning/scaffolding files (docs/asr/ASR_IMPLEMENTATION_PLAN.md, the paper4 tutorial notebook, scripts/_fill_aasr_readme_numbers.py, the MATLAB rASR shims + ASR_REFERENCE_README) and consolidate the per-PR changelog fragments into 36.feature.rst / 36.bugfix.rst. - utils: drop the UP038 noqa in favour of a PEP 604 union. - adaptive: expand the module docstring, align the mne import with juggler.py, and move the Yule-Walker statistics-filter design into _aasr_filter.py. - core: support MEG via picks="mag"/"grad"/"meg" with a scale-invariant variance check (MEG works like EEG), and split the SPD/Riemannian primitives into _spd.py.
- viz: keep the three ASR-specific diagnostics (repair timeline, calibration fraction, component reconstruction); reuse generic plot_signal_overlay / plot_psd_comparison / plot_power_ratio_map for before/after, PSD and topomap. - examples: plot_01/02 use plot_signal_overlay for the before/after trace; plot_05 showcases the generic + ASR-specific split. - docs: api.rst + asr.rst list the three diagnostics and point to the generics. - tests: drop dead viz tests, rewrite test_asr_viz for the kept three, and reformat test_aasr_mw to the zapline/dss/icanclean house style.
Move the ASRState dataclass out of core.py into a dedicated _types.py so every variant (standard/adaptive/Juggler/Riemannian) can depend on the fitted-state container without importing the calibration pipeline. core.py re-imports it; public API and behavior are unchanged.
Move parameter/array validation and dimension resolvers to _validation.py and the statistics-only IIR filter + reflected lookahead helpers to _filters.py. core.py re-exports them (listed in __all__) so adaptive/juggler/tests/scripts keep importing from mne_denoise.asr.core unchanged. No behavior change.
Move fit_eeg_distribution (+ histogram / robust-stats helpers) to _distribution.py and the block/rolling covariance builders, robust aggregation, and memory-budgeting helpers to _covariance.py. core.py re-exports them via __all__; no behavior change.
Move window start/weight/RMS helpers, clean_rawdata-style clean-window selection + grid diagnostics, and the sample-mask/span utilities to _windows.py. core.py re-exports them via __all__; no behavior change.
Move calibrate_asr and _fit_component_thresholds to _calibration.py, importing the window/distribution/covariance/filter/SPD helpers from their new homes. core.py re-exports via __all__; no behavior change.
Move process_asr + the Riemannian processing backends to _reconstruction.py and compute_asr_qa_metrics / compute_asr_rejection_mask to _qa.py (the latter type-imports ASR under TYPE_CHECKING to avoid a runtime cycle). core.py re-exports everything via __all__; no behavior change. core.py is now ~1045 lines with only the ASR estimator class left to move.
The ASR BaseEstimator/TransformerMixin class now lives in _estimator.py. core.py is reduced from ~2900 lines to a ~130-line re-export facade (public API + the private helpers that tests/scripts import as mne_denoise.asr.core.*), kept for backward compatibility. No behavior change.
adaptive.py, juggler.py and the package __init__ now import shared helpers and the public API from their canonical modules (_calibration, _covariance, _distribution, _estimator, _filters, _reconstruction, _types, _validation, _windows) instead of from .core. The 'import hub' is gone: core.py is a pure backward-compat facade no longer loaded on normal import. No behavior change.
Move the ASR unit tests into tests/asr/ mirroring the source package (test_estimator, test_coverage, test_robustness, test_viz, test_adaptive, test_juggler). Parity tests stay under tests/parity/. File-level relocation only; test bodies unchanged.
…t ref Add a 'Typical preprocessing pipeline' section showing where ASR sits between high-pass filtering and ICA, and make 'Choosing a variant' self-contained (removed the reference to the gitignored reports/ decision guide).
Add a mne_denoise.asr package logger and set_log_level_from_verbose() helper (MNE-style: True->INFO, False->WARNING, level name/int direct, None->inherit), following the mne-denoise per-module logger convention. ASR/AdaptiveASR/ JugglerASR now apply verbose at fit/transform/partial_fit and ASR emits a calibration summary at INFO. Previously verbose was an inert placeholder.
Resolve the 26 pre-existing mypy errors with proper typing (no behavior change): @overload on fit_eeg_distribution for the return_info flag, explicit dict[str, Any] annotations on the combined-diagnostics dicts, float() around np.linalg.norm in the SPD geometric medians, and is-not-None / isinstance narrowing asserts. mypy now reports no issues for mne_denoise/asr.
Extend the ASR gallery from 5 to 13 examples so every variant and common use-case is shown, each method example grounded in and citing its source paper: - plot_06 cutoff tuning (Chang 2020) - plot_07 Riemannian vs standard ASR on real blinks (Blum 2019) - plot_08 adaptive psp/psw/mw + moving-window trajectory (Tsai) - plot_09 Juggler dbscan vs gev reference selection (Kim 2025) - plot_10 choosing a variant (runnable decision guide) - plot_11 ASR on Epochs and on MEG magnetometers - plot_12 diagnostics + QC tour - plot_13 filter -> ASR -> ICA pipeline (capstone) Synthetic data for concept examples; plot_07/plot_13 (and the MEG part of plot_11) use mne.datasets.sample (DSS-gallery precedent). Reuses existing viz + QA helpers; README regrouped. Each runs headless, ruff + format clean.
Convert the References footnote markup (.. [N]) to bullet lists (sphinx-gallery renders example headers as plain RST, where unreferenced footnotes are warnings) and extend two over-long section-title underlines. No content/behavior change.
Use a continuously-rotating artifact subspace (genuine non-stationarity), add value labels so the small variant differences are legible, and reframe honestly: on cleanly-separable synthetic data the variants clean comparably (mw marginally best); the MW adaptation trajectory is the variant-specific highlight.
BabaSanfour
requested changes
Jun 9, 2026
BabaSanfour
left a comment
BabaSanfour
left a comment
Member
There was a problem hiding this comment.
The work is shaping nicely; one round of modifications before I take it from there
refactor(asr): split core.py into responsibility modules + completeness polish
docs(asr): complete, paper-grounded example gallery (5 -> 13)
- plot_signal_overlay: add optional reference / highlight_mask / highlight_spans kwargs and fold the manual overlay code out of plot_01 and plot_02 examples - tests/asr: redistribute test_coverage.py and test_robustness.py into the existing per-module suites (test_estimator/adaptive/juggler) by the module each test exercises, and remove the purpose-named files - remove docs/changes/devel/36.bugfix.rst (memory-bounding note folded into 36.feature.rst, since it is part of the new feature) and the dev-only tests/parity/matlab_reference/generate_rasr_reference.m
BabaSanfour
approved these changes
Jun 10, 2026
Member
|
nice ! @snesmaeili I will take it from here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an Artifact Subspace Reconstruction (ASR) module to
mne-denoise. ASRrepairs short, high-amplitude, spatially-structured EEG artifacts by calibrating
a clean-data covariance model and reconstructing burst-contaminated subspaces
(clean_rawdata lookahead + moving-covariance + raised-cosine blending).
The estimators follow the package conventions: scikit-learn
BaseEstimator/TransformerMixinwithfit/transform/fit_transform(+
partial_fitfor streaming), accepting MNERaw/Epochsor NumPy arrays,and exposing fitted
*_attributes.What's included
Variants
ASR(method="standard")— Euclidean clean_rawdata ASR (the default).ASR(method="riemannian")— MATLABrASRMatlab-faithful backend (kept behindexperimental=True).ASR(method="riemannian_windowed")— Riemannian robust calibration with aper-window eigendecomposition, so the
cutoffknob behaves like standard ASR.AdaptiveASR(variant="psp"|"psw"|"mw")— Hebbian/anti-Hebbian adaptive ASR,with a moving-window mode (
mw_mode="final_state"|"sliding").JugglerASR(strategy="dbscan"|"gev")— pointwise-amplitude referenceselection for high-motion / MoBI EEG.
Unified API across all three estimators:
get_diagnostics(),get_calibration_mask()(+calibration_mask_kind_), andto_annotations(kind="repair"|"rejection"|"calibration").Visualization —
mne_denoise.vizgains 10plot_asr_*helpers (overlay,cutoff sweep, PSD comparison, variance topomap, repair timeline, calibration
fraction, component reconstruction, blink reduction, grand average, method
comparison), following the existing viz contract (
(fig, ax), MNE or ndarrayinputs,
ax=/show=/fname=).Docs & examples —
docs/asr.rstnarrative + API reference, and 5 runnablegallery examples under
examples/asr/.Validation
Each variant is validated against its source paper and reference implementation
(Mullen 2013, Chang 2020, Blum 2019, Tsai et al., Kim 2025), with MATLAB
cross-check fixtures for the standard, rASR, adaptive, and riemannian_windowed
backends. Reproducibility tooling lives under
scripts/.Testing
parity tests).
mne_denoise/asr+mne_denoise/viz/asr.py.ruff checkandruff format --checkclean across the repo.Notes
.matfixtures are generated locally and gitignored; the paritytests skip when the fixtures (or MATLAB) are unavailable.
riemannianbackend is cutoff-invariant on real EEG by construction andstays behind
experimental=True;riemannian_windowedis the recommendedRiemannian path.