mirror of
https://github.com/zebrajr/pytorch.git
synced 2025-12-07 00:21:07 +01:00
18a58f0bd6
Fixes #107282 ## Overview - basic design decision was followed as they made on #103881 (tensor operation, test cases, order & position of argument etc.) - for the algorithm for decoupled weight decay, I referred to [1, 2] ## backwards-incompatible changes - positional argument `decoupled_weight_decay` is added to: - `torch.optim.radam` The existing code which refers to these APIs can be affected. Note: Positional argument `decoupled_weight_decay` is added to `torch.optim.RAdam`. However, since it was added to the last position and with default value, it is not affected. ## Reference - [1] [Decoupled Weight Decay Regularization](https://arxiv.org/abs/1711.05101) - [2] https://github.com/LiyuanLucasLiu/RAdam/blob/master/radam/radam.py#L5-L94 ## TODO - [x] implement tensor operation - [x] implement test cases - [x] modify doc-string - [x] pass unit test code locally `python test/test_optim.py -k test_radam` Pull Request resolved: https://github.com/pytorch/pytorch/pull/107507 Approved by: https://github.com/janeyx99
15 lines
343 B
Python
15 lines
343 B
Python
from typing import Tuple
|
|
|
|
from .optimizer import Optimizer, params_t
|
|
|
|
class RAdam(Optimizer):
|
|
def __init__(
|
|
self,
|
|
params: params_t,
|
|
lr: float = ...,
|
|
betas: Tuple[float, float] = ...,
|
|
eps: float = ...,
|
|
weight_decay: float = ...,
|
|
decoupled_weight_decay: bool = ...,
|
|
) -> None: ...
|