This updates ruff to 0.285 which is faster, better, and have fixes a bunch of false negatives with regards to fstrings.
I also enabled RUF017 which looks for accidental quadratic list summation. Luckily, seems like there are no instances of it in our codebase, so enabling it so that it stays like that. :)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/107519
Approved by: https://github.com/ezyang
Add similar semantics for creating a buffer object similar to creating a parameter. This is done by introducing a new `Buffer` class that can be used for type disambiguation. The underlying functionality of registering a buffer remains the same as the `register_buffer` method has not been changed. The `persistent` parameter in the `Buffer` type is to indicate whether a buffer object should be persistent or not. Other non-test changes have to do with getting the new `Buffer` type recognized by inductor and dynamo. Remaining changes are test changes to make sure that the `Buffer` type can be used as a drop in replacement for `register_buffer` as it just leads to `register_buffer` being called. The addition of this new functionality still allows for normal tensors to be used as buffers so these changes are intended to be backwards compatible.
Fixes#35735
Pull Request resolved: https://github.com/pytorch/pytorch/pull/104069
Approved by: https://github.com/mikaylagawarecki
Summary: UninitializedParameter overrides `__new__` method thus the parent class's `__deepcopy__` method doesn't work anymore, causing models using LazyModule cannot be instantiated.
Test Plan:
locally copied lazy module.
After change:
```
shenxiu@devbig1109:fbcode (5c57dd833)$ bento console --kernel pytorch --local
/data/users/shenxiu/fbsource/buck-out/v2/gen/fbcode/26f2c80c27f9e71d/bento/kernels/__bento_kernel_pytorch__/bento_kernel_pytorch#link-tree/scribeutil/lib.py:9: DeprecationWarning: The "thrift" clients in libfb.py.thrift_clients are not proper thrift clients, and often have unexpected or incorrect behaviour. They are also completely unsupported. Please use a supported client from https://fburl.com/srpy or a supported raw thrift client if you cannot use ServiceRouter.
from libfb.py.thrift_clients.scribe_thrift_client import ScribeThriftClient
/data/users/shenxiu/fbsource/buck-out/v2/gen/fbcode/26f2c80c27f9e71d/bento/kernels/__bento_kernel_pytorch__/bento_kernel_pytorch#link-tree/ipykernel/iostream.py:14: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import lock_held as import_lock_held
Python 3.8.6 (default, Jun 10 2022, 04:32:13)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import copy
...: import torch
...:
...: class LazyModule(torch.nn.Module):
...: def __init__(self):
...: super().__init__()
...: self.m = torch.nn.LazyLinear(10)
...:
...: def forward(self, input):
...: x = self.m(input)
...: return x
...:
...: m = LazyModule()
...: print(m.state_dict())
copy.deepcopy(m)
/data/users/shenxiu/fbsource/buck-out/v2/gen/fbcode/26f2c80c27f9e71d/bento/kernels/__bento_kernel_pytorch__/bento_kernel_pytorch#link-tree/mpmath/ctx_mp_python.py:892: SyntaxWarning: "is" with a literal. Did you mean "=="?
if other is 0:
/data/users/shenxiu/fbsource/buck-out/v2/gen/fbcode/26f2c80c27f9e71d/bento/kernels/__bento_kernel_pytorch__/bento_kernel_pytorch#link-tree/mpmath/ctx_mp_python.py:986: SyntaxWarning: "is" with a literal. Did you mean "=="?
if other is 0:
/data/users/shenxiu/fbsource/buck-out/v2/gen/fbcode/26f2c80c27f9e71d/bento/kernels/__bento_kernel_pytorch__/bento_kernel_pytorch#link-tree/sympy/solvers/diophantine.py:3188: SyntaxWarning: "is" with a literal. Did you mean "=="?
if feasible is 1: # it's prime and k == 2
/data/users/shenxiu/fbsource/buck-out/v2/gen/fbcode/26f2c80c27f9e71d/bento/kernels/__bento_kernel_pytorch__/bento_kernel_pytorch#link-tree/sympy/plotting/plot.py:520: SyntaxWarning: "is" with a literal. Did you mean "=="?
if self.xscale is 'log':
/data/users/shenxiu/fbsource/buck-out/v2/gen/fbcode/26f2c80c27f9e71d/bento/kernels/__bento_kernel_pytorch__/bento_kernel_pytorch#link-tree/sympy/plotting/plot.py:540: SyntaxWarning: "is" with a literal. Did you mean "=="?
if self.xscale is 'log':
/data/users/shenxiu/fbsource/buck-out/v2/gen/fbcode/26f2c80c27f9e71d/bento/kernels/__bento_kernel_pytorch__/bento_kernel_pytorch#link-tree/sympy/plotting/plot.py:553: SyntaxWarning: "is" with a literal. Did you mean "=="?
if self.xscale is 'log':
/data/users/shenxiu/fbsource/buck-out/v2/gen/fbcode/26f2c80c27f9e71d/bento/kernels/__bento_kernel_pytorch__/bento_kernel_pytorch#link-tree/sympy/plotting/plot.py:560: SyntaxWarning: "is" with a literal. Did you mean "=="?
if self.xscale is 'log':
OrderedDict([('m.weight', <UninitializedParameter>), ('m.bias', <UninitializedParameter>)])
In [2]: copy.deepcopy(m)
Out[2]:
LazyModule(
(m): LazyLinear(in_features=0, out_features=10, bias=True)
)
```
Before change, above code will give
```
TypeError: empty() received an invalid combination of arguments - got (int, dtype=NoneType, device=bool), but expected one of:
* (tuple of ints size, *, tuple of names names, torch.memory_format memory_format, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
* (tuple of ints size, *, torch.memory_format memory_format, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
* (tuple of SymInts size, *, torch.memory_format memory_format, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
```
Cloned n2369721 locally and successful (thru console not notebook because somehow bento notebook doesn't work with buck2 well).
Reviewed By: avilay
Differential Revision: D38866072
Pull Request resolved: https://github.com/pytorch/pytorch/pull/83809
Approved by: https://github.com/ngimel
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/66486
The newly-introduced Python dispatcher mode (`__torch_dispatch__`) does not have support for `torch.tensor()` (see #64360) and this causes friction in the user experience if some `nn.Modules` use `torch.tensor()` either implicitly or explicitly.
This PR replaces calls to `torch.tensor()` in `Parameter`, `UninitializedParameter`, and `UninitializedBuffer` with an equivalent call to `torch.empty()` which serves the same purpose and is syntactically more readable.
ghstack-source-id: 140520931
Test Plan: Since no behavioral change, run the existing unit and integration tests.
Reviewed By: pbelevich
Differential Revision: D31575587
fbshipit-source-id: bd7bdeea54370f3e53dc13bd182b97d0f67146f5
Summary:
Adds a note explaining the difference between several often conflated mechanisms in the autograd note
Also adds a link to this note from the docs in `grad_mode` and `nn.module`.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/58513
Reviewed By: gchanan
Differential Revision: D28651129
Pulled By: soulitzer
fbshipit-source-id: af9eb1749b641fc1b632815634eea36bf7979156
Summary:
Retake on https://github.com/pytorch/pytorch/issues/40493 after all the feedback from albanD
This PR implements the generic Lazy mechanism and a sample `LazyLinear` layer with the `UninitializedParameter`.
The main differences with the previous PR are two;
Now `torch.nn.Module` remains untouched.
We don't require an explicit initialization or a dummy forward pass before starting the training or inference of the actual module. Making this much simpler to use from the user side.
As we discussed offline, there was the suggestion of not using a mixin, but changing the `__class__` attribute of `LazyLinear` to become `Linear` once it's completely initialized. While this can be useful, by the time being we need `LazyLinear` to be a `torch.nn.Module` subclass since there are many checks that rely on the modules being instances of `torch.nn.Module`.
This can cause problems when we create complex modules such as
```
class MyNetwork(torch.nn.Module):
def __init__(self):
super(MyNetwork, self).__init__()
self.conv = torch.nn.Conv2d(20, 4, 2)
self.linear = torch.nn.LazyLinear(10)
def forward(self, x):
y = self.conv(x).clamp(min=0)
return self.linear(y)
```
Here, when the __setattr__ function is called at the time LazyLinear is registered, it won't be added to the child modules of `MyNetwork`, so we have to manually do it later, but currently there is no way to do such thing as we can't access the parent module from LazyLinear once it becomes the Linear module. (We can add a workaround to this if needed).
TODO:
Add convolutions once the design is OK
Fix docstrings
Pull Request resolved: https://github.com/pytorch/pytorch/pull/44538
Reviewed By: ngimel
Differential Revision: D24162854
Pulled By: albanD
fbshipit-source-id: 6d58dfe5d43bfb05b6ee506e266db3cf4b885f0c
Summary:
According to pytorch/rfcs#3
From the goals in the RFC:
1. Support subclassing `torch.Tensor` in Python (done here)
2. Preserve `torch.Tensor` subclasses when calling `torch` functions on them (done here)
3. Use the PyTorch API with `torch.Tensor`-like objects that are _not_ `torch.Tensor`
subclasses (done in https://github.com/pytorch/pytorch/issues/30730)
4. Preserve `torch.Tensor` subclasses when calling `torch.Tensor` methods. (done here)
5. Propagating subclass instances correctly also with operators, using
views/slices/indexing/etc. (done here)
6. Preserve subclass attributes when using methods or views/slices/indexing. (done here)
7. A way to insert code that operates on both functions and methods uniformly
(so we can write a single function that overrides all operators). (done here)
8. The ability to give external libraries a way to also define
functions/methods that follow the `__torch_function__` protocol. (will be addressed in a separate PR)
This PR makes the following changes:
1. Adds the `self` argument to the arg parser.
2. Dispatches on `self` as well if `self` is not `nullptr`.
3. Adds a `torch._C.DisableTorchFunction` context manager to disable `__torch_function__`.
4. Adds a `torch::torch_function_enabled()` and `torch._C._torch_function_enabled()` to check the state of `__torch_function__`.
5. Dispatches all `torch._C.TensorBase` and `torch.Tensor` methods via `__torch_function__`.
TODO:
- [x] Sequence Methods
- [x] Docs
- [x] Tests
Closes https://github.com/pytorch/pytorch/issues/28361
Benchmarks in https://github.com/pytorch/pytorch/pull/37091#issuecomment-633657778
Pull Request resolved: https://github.com/pytorch/pytorch/pull/37091
Reviewed By: ngimel
Differential Revision: D22765678
Pulled By: ezyang
fbshipit-source-id: 53f8aa17ddb8b1108c0997f6a7aa13cb5be73de0
* Codemod to update our codebase to 0.4 standard
* Update some of the test scri[ts
* remove Variable in test_clip_grad_value
* fix _symbolic_override_wrapper_maker
This changes type(tensor) to return `torch.Tensor` instead of
`torch.autograd.Variable`.
This requires a few implementation changes:
- torch.Tensor is now a regular Python class instead of a
pseudo-factory like torch.FloatTensor/torch.DoubleTensor
- torch.autograd.Variable is just a shell with a __new__ function.
Since no instanes are constructed it doesn't have any methods.
- Adds torch.get_default_dtype() since torch.Tensor.dtype returns
<attribute 'dtype' of 'torch._C._TensorBase' objects>
This removes volatile from Variable. The functionality is mostly
replaced by a global (thread-local) flag, which is controlled by
torch.set_grad_enabled() and the context manager torch.no_grad().
In C++, the flag is exposed through GradMode::is_enabled() and GradMode::set_enabled()
Fixes#3627
The core autograd Variable, Function, and Engine no longer depend on the
Python API. This let's us implement functions in C++. In the future, we
can also multithread engine and release the GIL for most of the
non-Python backwards.
