Jane Xu
de0375e79d
[optim][foreach] Do NOT inplace modify gradients ( #92706 )
...
SGD and ASGD already had out-of-place grads.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/92706
Approved by: https://github.com/ngimel , https://github.com/albanD
2023-01-21 00:12:28 +00:00
Jane Xu
de459bdfaa
[optim][rmsprop] group tensors in foreach to maximize perf ( #92369 )
...
Test plan:
CI
Pull Request resolved: https://github.com/pytorch/pytorch/pull/92369
Approved by: https://github.com/albanD
2023-01-18 22:28:52 +00:00
Jane Xu
0070c546b5
[BE][optim] abstract out docstrings, add differentiable docs ( #92336 )
...
1. abstract out common doc strings --> I'm sure there are more, but let this be a first step.
2. Add differentiable docs to those who are actually differentiable
Pull Request resolved: https://github.com/pytorch/pytorch/pull/92336
Approved by: https://github.com/albanD
2023-01-18 15:09:28 +00:00
Michael Lazos
c63afb283c
Disable dynamo on optimizer lazy initialization ( #89902 )
...
Helps with https://github.com/pytorch/torchdynamo/issues/1803
Separate out the group initialization and disable dynamo on it
Pull Request resolved: https://github.com/pytorch/pytorch/pull/89902
Approved by: https://github.com/soumith , https://github.com/albanD
2022-12-02 01:15:11 +00:00
Michael Lazos
3d47c74cfe
Update code style for optimizer code ( #89862 )
...
Separating out whitespace-only changes
Pull Request resolved: https://github.com/pytorch/pytorch/pull/89862
Approved by: https://github.com/albanD , https://github.com/soumith
2022-11-30 00:53:05 +00:00
Kshiteej K
09331c947c
[optim] rmsprop: handle complex params as independent real params ( #83860 )
...
Ref: #65711
Pull Request resolved: https://github.com/pytorch/pytorch/pull/83860
Approved by: https://github.com/albanD
2022-08-22 21:55:01 +00:00
Emilio Castillo
f0eb841d20
Make torch.optim.RMSprop differentiable ( #83578 )
...
Blocked by #82205
Pull Request resolved: https://github.com/pytorch/pytorch/pull/83578
Approved by: https://github.com/albanD
2022-08-22 03:37:10 +00:00
ProGamerGov
71d50f4f89
Change docstring type callable to Callable for consistency ( #82487 )
...
### Description
Across PyTorch's docstrings, both `callable` and `Callable` for variable types. The Callable should be capitalized as we are referring to the `Callable` type, and not the Python `callable()` function.
### Testing
There shouldn't be any testing required.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/82487
Approved by: https://github.com/albanD
2022-08-01 17:26:09 +00:00
Rob Zinkov
a1fd5b4273
Adding maximize to RMSprop ( #80326 )
...
Added the maximize flag #68052 to RMSprop optimizer and updates the respective tests.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80326
Approved by: https://github.com/albanD
2022-07-08 08:04:26 +00:00
anjali411
bda04e9f5e
Add __all__ for torch.optim and torch.nn.modules modules ( #80237 )
...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80237
Approved by: https://github.com/albanD
2022-06-24 21:34:10 +00:00
Mikayla Gawarecki
ce3094f5f6
Optim foreach cleanup for Rmsprop ( #70482 )
...
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/70482
Test Plan: Imported from OSS
Reviewed By: anjali411
Differential Revision: D33767862
Pulled By: mikaylagawarecki
fbshipit-source-id: 8e2e9c986d5a3774093a79755940372945f1b3a9
(cherry picked from commit baea537277
2022-02-15 18:02:08 +00:00
Ilqar Ramazanli
aefa2f3e64
To add RMSProp algorithm documentation ( #63721 )
...
Summary:
It has been discussed before that adding description of Optimization algorithms to PyTorch Core documentation may result in a nice Optimization research tutorial. In the following tracking issue we mentioned about all the necessary algorithms and links to the originally published paper https://github.com/pytorch/pytorch/issues/63236 .
In this PR we are adding description of RMSProp to the documentation. For more details, we refer to the paper https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
<img width="464" alt="RMSProp" src="https://user-images.githubusercontent.com/73658284/131179226-3fb6fe5a-5301-4948-afbe-f38bf57f24ff.png ">
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63721
Reviewed By: albanD
Differential Revision: D30612426
Pulled By: iramazanli
fbshipit-source-id: c3ac630a9658d1282866b53c86023ac10cf95398
2021-08-28 15:55:56 -07:00
Wanchao Liang
4611387608
[optim] take kw-only argument for functional optim APIs ( #56185 )
...
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56185
ghstack-source-id: 126670123
Reviewed By: albanD
Differential Revision: D27802169
fbshipit-source-id: f5e1cb2046dcdeecf5f6b0f70892828bf0adb22f
2021-04-15 20:08:04 -07:00
Vincent Quenneville-Belair
50d903f19f
[optim] make functional api be private ( #51316 ) ( #51665 )
...
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51665
This reverts commit 896f82aa92
2021-02-03 17:59:05 -08:00
Vincent Quenneville-Belair
896f82aa92
[optim] make functional api be private ( #51316 )
...
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51316
Make optim functional API be private until we release with beta
Test Plan: Imported from OSS
Reviewed By: albanD
Differential Revision: D26213469
fbshipit-source-id: b0fd001a8362ec1c152250bcd57c7205ed893107
2021-02-03 09:29:33 -08:00
Wanchao Liang
ce1781d8db
[optimizer] refactor RMSProp to use functional API ( #50410 )
...
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/50410
Test Plan: Imported from OSS
Reviewed By: izdeby
Differential Revision: D25932779
Pulled By: wanchaol
fbshipit-source-id: b0d6007ea83d77e2d70d04681163ea7e4632c5cd
2021-01-21 11:00:41 -08:00
Samuel Marks
e6779d4357
[*.py] Rename "Arguments:" to "Args:" ( #49736 )
...
Summary:
I've written custom parsers and emitters for everything from docstrings to classes and functions. However, I recently came across an issue when I was parsing/generating from the TensorFlow codebase: inconsistent use of `Args:` and `Arguments:` in its docstrings.
```sh
(pytorch#c348fae)$ for name in 'Args:' 'Arguments:'; do
printf '%-10s %04d\n' "$name" "$(rg -IFtpy --count-matches "$name" | paste -s -d+ -- | bc)"; done
Args: 1095
Arguments: 0336
```
It is easy enough to extend my parsers to support both variants, however it looks like `Arguments:` is wrong anyway, as per:
- https://google.github.io/styleguide/pyguide.html#doc-function-args @ [`ddccc0f`](https://github.com/google/styleguide/blob/ddccc0f/pyguide.md )
- https://chromium.googlesource.com/chromiumos/docs/+/master/styleguide/python.md#describing-arguments-in-docstrings @ [`9fc0fc0`](https://chromium.googlesource.com/chromiumos/docs/+/9fc0fc0/styleguide/python.md )
- https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html @ [`c0ae8e3`](https://github.com/sphinx-contrib/napoleon/blob/c0ae8e3/docs/source/example_google.rst )
Therefore, only `Args:` is valid. This PR replaces them throughout the codebase.
PS: For related PRs, see tensorflow/tensorflow/pull/45420
PPS: The trackbacks automatically appearing below are sending the same changes to other repositories in the [PyTorch](https://github.com/pytorch ) organisation.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/49736
Reviewed By: albanD
Differential Revision: D25710534
Pulled By: soumith
fbshipit-source-id: 61e8ff01abb433e9f78185c2d1d0cbd7c22c1619
2020-12-28 09:34:47 -08:00
Edward Leardi
6b50874cb7
Fix HTTP links in documentation to HTTPS ( #40878 )
...
Summary:
I ran `make linkcheck` using `sphinx.builders.linkcheck` on the documentation and noticed a few links weren't using HTTPS so I quickly updated them all.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/40878
Differential Revision: D22404647
Pulled By: ngimel
fbshipit-source-id: 9c9756db59197304023fddc28f252314f6cf4af3
2020-07-06 20:05:21 -07:00
albanD
6e2bb1c054
End of the .data removal in torch/optim ( #34211 )
...
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/34211
Test Plan: Imported from OSS
Differential Revision: D20248684
Pulled By: albanD
fbshipit-source-id: 2294bfa41b82ff47f000bc98860780f59d7d4421
2020-03-09 06:40:39 -07:00
Eleanor Dwight Holland
6a97777f72
Remove use of .data from optimizers ( #33640 )
...
Summary:
Removes all uses of `.data` from optimizers.
Or tries to.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/33640
Reviewed By: vincentqb
Differential Revision: D20203216
Pulled By: albanD
fbshipit-source-id: 9bfe78bbed00fd4aaa690801cff0201f0bd680a0
2020-03-03 13:21:55 -08:00
Xiao Wang
c1dd70688a
Fix deprecated python "add" calls ( #33428 )
...
Summary:
This PR fixed those python "add" calls using deprecated signature `add(Scalar, Tensor)`. The alternative signature `add(Tensor, alpha = Scalar)` is used.
cc csarofeen zasdfgbnm ptrblck ngimel
Pull Request resolved: https://github.com/pytorch/pytorch/pull/33428
Differential Revision: D20002534
Pulled By: vincentqb
fbshipit-source-id: 81f2dd6170a47a9b53a17e5817c26e70d8afa130
2020-02-26 09:02:31 -08:00
Vitaly Fedyunin
877c96cddf
explicitly provide memory format when calling to *_like operators
...
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/30008
Test Plan: Imported from OSS
Differential Revision: D18575981
Pulled By: VitalyFedyunin
fbshipit-source-id: ec3418257089ad57913932be1a8608cd20ce054c
2019-11-19 16:19:29 -08:00
Vincent Quenneville-Belair
660d9e24dd
Highlighting in the doc that square root comes before adding epsilon
...
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/26735
Test Plan: Imported from OSS
Differential Revision: D17558505
Pulled By: vincentqb
fbshipit-source-id: 36449c501f3ab3bc7cadd1f580258904b39369d4
2019-09-25 15:52:28 -07:00
meijieru
bd054e7cef
reduce memory usage for centered rmsprop ( #24170 )
...
Summary:
Reduce gpu memory usage by using in-place operation
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24170
Differential Revision: D16784495
Pulled By: vincentqb
fbshipit-source-id: 03820cdc9a3952b95b9af0f87d3a9bb0f21e9b4d
2019-08-13 12:18:31 -07:00
lazypanda1
063946d2b3
Added parameter range checks for all optimizers ( #6000 )
2018-03-28 11:22:23 +02:00
Ozan Çağlayan
dd6d04ddf2
doc: Normalize all true/false in docstrings to `True|False` ( #3593 )
...
* doc: Normalize all true/false in docstrings to ``True|False``
This makes them more apparent in the documentation.
* doc: fix flake8
2017-11-09 08:12:29 -05:00
SsnL
f76d6c029c
Sparse Adam optimizer for sparse gradients ( #3137 )
...
* sparse adam
* Favor dense addition over sparse_mask
2017-11-06 14:20:51 -05:00
Leonid Vlasenkov
46a868dab7
[Ready] Limit docs line length ( #1900 )
...
* some docs are ready
* docs
* docs
* fix some more
* fix some more
2017-07-10 10:24:54 -04:00
Edouard Delasalles
b1c2714ad5
Add momentum and centered options to RMSProp ( #810 )
...
* add momentum and centered options
Add two options :
- Momentum (like SGD's momentum)
- Centered RMSprop, as in Graves 2013 ( https://arxiv.org/abs/1308.0850 ) : grad is normalized by running estimation of its variance
* somme PEP8
* bug in default
* bug2
* sign mistake
* alloc of momentum & centered only if needed
* add link to docstring
* some pep8 on docstring
* implement __setstate__() for backward compatibilty
* correct grammar mistake
* multiply by lr when adding delta to params
* rename momentum variables
* change __init__ params order
2017-03-09 10:04:32 +01:00
Martin Raison
f17cfe4293
sparse tensor operations ( #735 )
2017-03-03 18:37:03 +01:00
Luke Yeager
e7c1e6a8e3
[pep8] Fix most lint automatically with autopep8
...
Here's the command I used to invoke autopep8 (in parallel!):
git ls-files | grep '\.py$' | xargs -n1 -P`nproc` autopep8 -i
Several rules are ignored in setup.cfg. The goal is to let autopep8
handle everything which it can handle safely, and to disable any rules
which are tricky or controversial to address. We may want to come back
and re-enable some of these rules later, but I'm trying to make this
patch as safe as possible.
Also configures flake8 to match pep8's behavior.
Also configures TravisCI to check the whole project for lint.
2017-01-28 01:15:51 +01:00
Adam Paszke
ecfcf39f30
Improve optimizer serialization
...
Also, add optimizer.load_state_dict
2017-01-24 17:30:50 -05:00
glample
99f4864674
fixed RMSprop initialization ( #485 )
...
* fixed RMSprop initialization
2017-01-18 17:05:53 -05:00
Adam Paszke
95f0fa8a92
Change .grad attribute of Variables to be a Variable
2017-01-16 12:59:47 -05:00
Adam Paszke
604e13775f
Add optim docs
2017-01-16 12:59:47 -05:00
Adam Paszke
09493603f6
Change optimizer API
2016-11-08 18:12:56 +01:00
Adam Paszke
df59b89fbb
Add more optimizers
2016-11-07 22:50:56 +01:00
Adam Paszke
2f342af22f
Move optim to legacy
2016-08-01 12:01:46 -04:00
Adam Paszke
554a1d8336
Add optim
2016-07-21 16:42:06 -04:00
