[Feature] Implement SpectralSphere optimizer (#485)

* build(deps): update

* docs: README

* docs: v3.10.0 changelog

* feature: SpectralSphere optimizer

* chore: update-deps recipe

* update: codes

* build(ci): update pyright version to 1.1.408

* build(deps): update the deps

* refactor: remove unused params

* refactor: type stuff

* update: codes

* docs: update documentation

* update: codes

* docs: update index.md

* fix: typo

* update: codes

* update: recipe

* docs: docstring

* refactor: remove unused codes

* update: recipe

* docs: docstring

* update: codes

* update: codes

* update: codes

* update: codes

---------

Co-authored-by: kozistr <15344796+kozistr@users.noreply.github.com>

Author: kozistr

.github/workflows/ci.yml

@@ -51,7 +51,7 @@ jobs:
 
       - uses: jakebailey/pyright-action@v2
         with:
-          version: 1.1.407
+          version: 1.1.408
 
   test:
     runs-on: ubuntu-latest

Makefile

@@ -1,4 +1,4 @@
-.PHONY: format lint test check requirements visualize docs update-docs
+.PHONY: format lint test check requirements visualize docs update-docs update-deps
 
 FILES := pytorch_optimizer examples tests scripts hubconf.py
 BLACK_FLAGS := -S -l 119
@@ -29,3 +29,6 @@ docs:
 
 update-docs:
 	python scripts/update_docs.py
+
+update-deps:
+	uv sync --upgrade

README.md

@@ -226,6 +226,7 @@ get_supported_optimizers(['adam*', 'ranger*'])
 | BCOS                  | *Stochastic Approximation with Block Coordinate Optimal Stepsizes*                                             | [github](https://github.com/facebookresearch/bcos)                                                             | <https://arxiv.org/abs/2507.08963>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250708963J/exportcitation)                                                        |
 | Cautious WD           | *Cautious Weight Decay*                                                                                        |                                                                                                                | <https://arxiv.org/abs/2510.12402>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv251012402C/exportcitation)                                                        |
 | Ano                   | *Faster is Better in Noisy Landscape*                                                                          | [github](https://github.com/Adrienkgz/ano-optimizer)                                                           | <https://arxiv.org/abs/2508.18258>                                                                                   | [cite](https://github.com/Adrienkgz/ano-optimizer?tab=readme-ov-file#citation)                                                      |
+| Spectral Sphere       | *Controlled LLM Training on Spectral Sphere*                                                                   | [github](https://github.com/Unakar/Spectral-Sphere-Optimizer)                                                  | <https://arxiv.org/abs/2601.08393>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2026arXiv260108393X/exportcitation)                                                        |
 
 ## Supported LR Scheduler
 

docs/changelogs/v3.10.0.md

@@ -9,6 +9,8 @@
 * Update the Emo-series optimizers. (#472, #478)
   * Update `EmoNavi`, `EmoFact`, and `EmoLynx`. 
   * Fade out `EmoNeco` and `EmoZeal` optimizers.
+* Implement `SpectralSphere` optimizer. (#483, #485)
+  * [Controlled LLM Training on Spectral Sphere](https://arxiv.org/abs/2601.08393)
 
 ### Fix
 

docs/index.md

@@ -226,6 +226,7 @@ get_supported_optimizers(['adam*', 'ranger*'])
 | BCOS                  | *Stochastic Approximation with Block Coordinate Optimal Stepsizes*                                             | [github](https://github.com/facebookresearch/bcos)                                                             | <https://arxiv.org/abs/2507.08963>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250708963J/exportcitation)                                                        |
 | Cautious WD           | *Cautious Weight Decay*                                                                                        |                                                                                                                | <https://arxiv.org/abs/2510.12402>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv251012402C/exportcitation)                                                        |
 | Ano                   | *Faster is Better in Noisy Landscape*                                                                          | [github](https://github.com/Adrienkgz/ano-optimizer)                                                           | <https://arxiv.org/abs/2508.18258>                                                                                   | [cite](https://github.com/Adrienkgz/ano-optimizer?tab=readme-ov-file#citation)                                                      |
+| Spectral Sphere       | *Controlled LLM Training on Spectral Sphere*                                                                   | [github](https://github.com/Unakar/Spectral-Sphere-Optimizer)                                                  | <https://arxiv.org/abs/2601.08393>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2026arXiv260108393X/exportcitation)                                                        |
 
 ## Supported LR Scheduler
 

docs/optimizer.md

@@ -496,6 +496,10 @@
     :docstring:
     :members:
 
+::: pytorch_optimizer.SpectralSphere
+    :docstring:
+    :members:
+
 ::: pytorch_optimizer.SPlus
     :docstring:
     :members:

pyproject.toml

@@ -58,9 +58,9 @@ dependencies = [
 [dependency-groups]
 dev = [
     'black<25; python_version < "3.9"',
-    'black>=25,<26; python_version >= "3.9"',
+    'black<26; python_version >= "3.9"',
     'ruff>=0.14,<0.15',
-    'pyright==1.1.407',
+    'pyright==1.1.408',
     'pytest<9; python_version < "3.10"',
     'pytest>=9,<10; python_version >= "3.10"',
     'pytest-cov<6; python_version < "3.9"',

pytorch_optimizer/__init__.py

@@ -122,6 +122,7 @@
     DiffGrad,
     DistributedMuon,
     DynamicLossScaler,
+    SpectralSphere,
     EmoFact,
     EmoLynx,
     EmoNavi,

pytorch_optimizer/optimizer/__init__.py

@@ -104,6 +104,7 @@
 from pytorch_optimizer.optimizer.spam import SPAM, StableSPAM
 from pytorch_optimizer.optimizer.splus import SPlus
 from pytorch_optimizer.optimizer.srmm import SRMM
+from pytorch_optimizer.optimizer.sso import SpectralSphere
 from pytorch_optimizer.optimizer.swats import SWATS
 from pytorch_optimizer.optimizer.tam import TAM, AdaTAM
 from pytorch_optimizer.optimizer.tiger import Tiger
@@ -234,6 +235,7 @@
     Tiger,
     VSGD,
     Yogi,
+    SpectralSphere,
 ]
 OPTIMIZERS: Dict[str, OPTIMIZER] = {str(optimizer.__name__).lower(): optimizer for optimizer in OPTIMIZER_LIST}