Skip to content

feat: Adaptive DSS with segmentation, auto-selection, and smoothing#28

Open
snesmaeili wants to merge 7 commits into
mainfrom
feature/adaptive-dss
Open

feat: Adaptive DSS with segmentation, auto-selection, and smoothing#28
snesmaeili wants to merge 7 commits into
mainfrom
feature/adaptive-dss

Conversation

@snesmaeili

@snesmaeili snesmaeili commented Mar 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Implements adaptive / continuous DSS for handling non-stationary artifacts, and unifies segmentation across DSS and ZapLine.

Closes #27

Changes

Bug fixes (on top of original adaptive DSS commit)

  • transform() smooth preservation: data_smooth was being discarded after decomposition — now captured and added back to reconstruction
  • _clean_segment() centering: data wasn't centered before DSS projection (violating zero-mean assumption) — now centers residual before projecting
  • n_selected_ semantics: was reporting sum across all segments (misleading) — now reports max
  • Empty segments guard: _run_segmented() now raises ValueError if segmenter returns no segments
  • narrowband_scan guard: raises ValueError if segmented=True is passed (unsupported combination)

ZapLine / DSS segmentation unification

  • segment_data() in adaptive.py: replaced 99 lines of duplicate code with thin backward-compatible wrapper delegating to shared CovarianceSegmenter
  • core.py _run_adaptive(): now uses CovarianceSegmenter directly instead of segment_data()
  • Single segmentation pathway: both DSS (segmented mode) and ZapLine (adaptive mode) now use the same CovarianceSegmenter from dss.utils.segmentation

New features

  • segmented=True mode with FixedWindowSegmenter and CovarianceSegmenter
  • Auto-selection via n_select='outlier'|'ratio'|'max_gap'|'combined'
  • Smoothing decomposition via smooth=int
  • Adaptive caps (max_prop_remove) and floors (min_select)

Tests

  • 29 new tests across 3 files covering segmented DSS, auto-select, smoothing, caps/floors, segmenters, and selection utilities
  • tests/utils/test_segmentation.py (11 tests)
  • tests/utils/test_selection.py (18 tests)
  • tests/test_linear_dss.py extended with TestSegmentedDSS, TestAutoSelect, TestSmoothingDecomposition, TestCapAndFloor, and test_narrowband_scan_rejects_segmented
  • 3 new unification tests in tests/test_zapline_adaptive.py verifying segment_data delegates to CovarianceSegmenter
  • Fixed 19 broken DSS tests (corrected API usage: bias param, segmenter instances, param names)
  • All 359 tests pass (11 skipped: MATLAB parity)

snesmaeili and others added 6 commits March 4, 2026 08:34
@snesmaeili
feat(dss): Add adaptive DSS with segmentation, auto-selection, smooth…
552f207 
…ing, and pattern normalization
@snesmaeili
fix(dss): bug fixes for adaptive DSS + comprehensive tests
b43d1d5 
Bug fixes:
- transform() now preserves smooth component (was discarded after decomposition)
- _clean_segment() centers data before DSS projection (zero-mean assumption)
- n_selected_ reports max across segments (was misleading sum)
- Empty segmenter results now raise ValueError
- narrowband_scan rejects segmented=True with clear error

Tests:
- 29 new tests for segmentation, selection, segmented DSS, auto-select,
  smoothing decomposition, cap/floor, and narrowband guard
- tests/utils/test_segmentation.py (11 tests)
- tests/utils/test_selection.py (18 tests)
- tests/test_linear_dss.py extended with adaptive DSS test classes
@snesmaeili
chore: export adaptive DSS symbols + add tests/utils/__init__.py
2523d80 
- Add eigenvalue_ratio_selection, max_gap_selection to utils/__init__.py
- Re-export CovarianceSegmenter, FixedWindowSegmenter, and selection
  utilities from dss/__init__.py for public API access
- Add empty tests/utils/__init__.py for test discovery
@snesmaeili
Unify ZapLine/DSS segmentation: use shared CovarianceSegmenter
1409e95 
- Replace duplicate segment_data() in adaptive.py with thin wrapper
  delegating to CovarianceSegmenter from dss.utils.segmentation
- Update core.py _run_adaptive() to use CovarianceSegmenter directly
- Update plot_04_adaptive_mode.py example to use CovarianceSegmenter
- Fix 19 broken tests in test_linear_dss.py:
  * Add required bias=LineNoiseBias() to all DSS constructors
  * Replace string segmenter names with segmenter instances
  * Fix LineNoiseBias param names (freq, sfreq)
  * Fix test_invalid_method to test selection_method param
  * Fix test_fit_then_transform_preserves_smooth return_type

All 359 tests pass (11 skipped: MATLAB parity).
@snesmaeili
Add cross-fade overlap-add for segment boundaries
3806482 
- New 'crossfade' parameter on DSS (float, seconds, default 0.0)
- When crossfade > 0 and segmented=True, adjacent segments are
  extended by crossfade seconds, cleaned independently, then
  blended via raised-cosine (Hann) overlap-add window
- Eliminates discontinuities at segment boundaries
- Overlap auto-clamped to half the smallest segment length
- crossfade=0.0 preserves original hard-concatenation behaviour
- New _crossfade_combine() helper method
- 6 new tests in TestCrossfade: output shape, no boundary jump,
  single-segment identity, backward compat, energy preservation,
  overlap clamping

All 365 tests pass (11 skipped: MATLAB parity).
@snesmaeili
docs(dss): attribute adaptive-DSS methods to ZapLine-plus (Klug & Klo…
d21edb6 
…osterman 2022)

Validate PR #28's adaptive-DSS machinery against Klug & Kloosterman 2022 (ZapLine-plus) and de Cheveigné 2020 (ZapLine), and correct the docstring attributions:

- iterative_outlier_removal: cite Klug 2022 §2.4 -- this IS the ZapLine-plus automatic component detector (mean+SD threshold, iterate until none remain, sigma=3 default, reported more robust than MAD), alongside NoiseTools
- max_prop_remove / min_select: note they mirror ZapLine-plus's one-fifth cap and fixed-removal floor
- crossfade: note ZapLine-plus concatenates chunks; the Hann cross-fade is an mne-denoise addition
- SmoothingBias: cite de Cheveigné 2020 for the period-matched smooth/residual decomposition
- eigenvalue_ratio_selection / max_gap_selection: label as mne-denoise convenience heuristics (not in ZapLine-plus or the DSS literature)

Docstring-only; no behavior change.
@snesmaeili

snesmaeili commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

Literature validation: adaptive DSS ↔ ZapLine-plus

I cross-checked the adaptive-DSS machinery in this PR against the source papers — Klug & Kloosterman 2022 (ZapLine-plus) and de Cheveigné 2020 (ZapLine), plus Särelä & Valpola 2005, de Cheveigné & Parra 2014, de Cheveigné & Arzounian 2018, and Almeida 2005. Net result: the implementation is faithful to ZapLine-plus; the main gap was attribution, which d21edb6 fixes (docstring-only, no behavior change).

Faithful to the literature (now cited in the docstrings):

  • iterative_outlier_removal — exactly the ZapLine-plus §2.4 component detector: mean + σ·SD threshold, iterate until none remain, default σ=3, reported "more robust than the median absolute deviation."
  • max_prop_remove / min_select — §2.4's "cap at one-fifth of the components" and the fixed-removal floor (fixedNremove).
  • CovarianceSegmenter — §2.2: ±3 Hz narrowband, 1 s covariances, successive distance, peak detection, 30 s minimum chunk.
  • adaptive σ / QA loop — §2.5: 0.5% proportion thresholds, ±0.25 σ steps, min 2.5 / max 4, too-strong-takes-precedence.
  • smoothing / residual decomposition — de Cheveigné 2020 (period-matched smoother; now cited on SmoothingBias).

Honestly flagged as mne-denoise additions (sensible engineering, just not from the papers — labelled as such in the docstrings rather than over-claimed):

  • eigenvalue_ratio_selection (scree, threshold 2.0) and max_gap_selection (1.2) — ZapLine-plus uses only the outlier detector; these are convenience fallbacks with empirical defaults.
  • the Hann cross-fade — ZapLine-plus concatenates cleaned chunks directly; the overlap-add blend is an improvement for boundary continuity.

d21edb6 is docstring-only (citations + honest labels): no behavior change, ruff check clean.

@snesmaeili
Merge origin/main into feature/adaptive-dss
c7fab2f 
Resolve conflicts from main's advancement (ZapLine auto-mode fix #34, centralized MNE channel picking, icanclean, docs/CI):

- dss/utils/selection.py: union the branch's eigenvalue_ratio_selection / max_gap_selection with main's detect_eigenvalue_knee / auto_select_components_robust (keeps the Klug 2022 attribution)
- dss/utils/__init__.py: export both selection sets
- zapline/core.py: use main's auto_select_components_robust for auto n_remove
- dss/linear.py: adopt main's 6-value extract_data_from_mne signature in the segmented paths; restore main's get_normalized_patterns method
- examples/zapline/plot_04: adopt main's moved viz import path
- tests: union both selection/linear suites; adopt main's compute_dss no-variance / scale-invariant tests (compute_dss now warns-not-raises on heavy rank reduction)

Full local suite: 575 passed, 11 skipped (MATLAB parity).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: Adaptive DSS with segmentation, auto-selection, and smoothing

1 participant

@snesmaeili