Question about weight decay in Muon

[jootanehorror]

Hi!
I noticed that in nanochat the Muon optimizer is used without weight decay.

In recent work on Muon (e.g. “Muon is Scalable for LLM Training”), decoupled weight decay is sometimes applied as a post-update shrinkage and reported to improve stability and final quality.

Is there a specific reason why weight decay is omitted for Muon here?
Is it mainly for simplicity, or did you observe no benefit (or negative effects) in practice?

Thanks!

[karpathy]

I observed very minor improvements, and very slightly better with the cautious weight decay form. I'll push it to master shortly.