Skip to content

Decay OneCycle momentum instead of growing it past 1.0 - #8589

Open
vineethsaivs wants to merge 1 commit into
deepspeedai:masterfrom
vineethsaivs:fix-onecycle-momentum-decay-20260918
Open

vineethsaivs wants to merge 1 commit into
deepspeedai:masterfrom
vineethsaivs:fix-onecycle-momentum-decay-20260918

Conversation

@vineethsaivs

Copy link
Copy Markdown
Contributor

With cycle_momentum on and any decay_mom_rate, OneCycle grows the first Adam beta without bound in its decay phase and pushes it past 1.0.

_get_decay_mom multiplies cycle_max_mom by (1 + rate * interval) while _get_decay_lr divides the learning rate by the same factor. Both the docstring and the command line help call the parameter a decay rate. Divide in _get_decay_mom so momentum decays from the cycle maximum, mirroring _get_decay_lr.

Test: TORCHDYNAMO_DISABLE=1 DS_ACCELERATOR=cpu PYTHONPATH=. python -m pytest tests/unit/runtime/test_lr_schedulers.py -k 'one_cycle or momentum' -q. A regression fails before, betas[0] reaching 1.045 at step 30 and rising, a value torch.optim.Adam rejects in its own constructor. Fifteen CPU tests pass after against hand derived values. An existing assertion that pinned the growth now expects the decay. Changed-file pre-commit passes. Executed on Apple M2 Pro CPU; no CUDA or distributed training run. The DistributedTest failures in this file are pre-existing on unpatched source here.

Divide in `_get_decay_mom` so momentum decays from the cycle maximum, mirroring `_get_decay_lr`.

Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
@vineethsaivs

Copy link
Copy Markdown
Contributor Author

Could you approve the CI run? A regression fails before, betas[0] reaching 1.045 at step 30 and rising, a value torch.optim.Adam rejects in its own constructor. Fifteen CPU tests pass after against hand derived values. An existing assertion that pinned the growth now expects the decay. Changed-file pre-commit passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant