This adds overrides in VariableType for the xxx_out ATen functions and
implements Python bindings. There is no support for automatic
differentiation. If any of the inputs (or outputs) requires grad, then the
function will throw an exception unless it's running in "no-grad" mode.
The bindings for calling torch.xxx functions on Variables are moved to a
different object. Previously, they were static method on VariableBase.
This change prevents users from accidentally calling static methods as if
they were instance methods.
This removes volatile from Variable. The functionality is mostly
replaced by a global (thread-local) flag, which is controlled by
torch.set_grad_enabled() and the context manager torch.no_grad().
In C++, the flag is exposed through GradMode::is_enabled() and GradMode::set_enabled()
Fixes#3627
* Implement Variable.cuda using ATen
This adds an optional async flag to Tensor::copy_, which attempts to do
a non-blocking copy if the one of the tensors is in pinned memory and
the other is a CUDA tensor.
* Perform cross-device copy in CopyBackwards
Also call torch.cuda._lazy_init() from Variable.cuda()
* Implement Variable.type via ATen
* Changes from review:
- remove copy_out
- remove unnecessary include
- fix default device for .cuda()
* Combine if statements in dispatch_type
* Implement remaining random methods through ATen
* Change test_bernoulli on Tensor to avoid broadcasting
The new ATen-dispatched bernoulli_ supports broadcasting. The old
Tensor.bernoulli_ bindings instead require the tensors to have the same
number of elements. I haven't change the old code because it will be
deleted soon.
Implements from_numpy using ATen tensors. Variable.from_numpy is a
convenient placeholder for the variant that returns Variables until we
merge Tensor and Variable.
The behavior is slightly changed:
- from_numpy() on an empty array now returns an empty tensor instead of
throwing an exception. The shape may not be preserved.
- CharTensor(ndarray) used to throw an exception. It now copies the
ndarray. Copying is implemented via ATen toType.
* Implement matmul as a native function; use it for Variable impl.
This also includes an (inefficient) version of allclose, which was necessary for testing.
A more efficient version would use some apply logic to fuse the ops and exit early (coming in future PR).
On small tensors [(2, 5, 5) @ (5,5)], this yields ~2.5x speedup over the python implementation.
* Make maybeSqueeze static.
* Have localScalar work with all 1 element tensors, not just scalars.
Also have toCFloat, etc. call localScalar so 1 element tensors work as well.
* Implement python number conversions.
* Implement __bool__, __nonzero__ as ATen functions.
* Remove merge artifacts.
* Simplify by dispatching to toCDouble.
Implements basic and advanced indexing using ATen tensors/variables.
Basic indexing is translated at the Python-binding level
(python_variable_indexing.cpp) to slice/squeeze/unsqueeze/select calls.
Advanced indexing is implemented in ATen in terms of take() and put()
calls.
* Move Variable conversion methods to ATen.
* Add a test to ensure type conversions work through backwards.
* Fix VariableType copy for type conversions.
* Add comment about needing to handle device movement.
* Move back to opposite order for copy function params -- inplace views depend on it.
* Use is_available() rather than is_available.
* Use aten version of is_signed.
* Define is_cuda native function and use it for variable.
* Use ATen dim for Variable dim/ndimension.
* Get rid of dim, ndimension fallthroughs in variable.py.
* Move size/stride Variable methods to use ATen.
* Implement shape property on Variable via ATen.
* Remove the _getattr__ function from Variable.
* Get rid of dispatch functions and avoid cast.
* Add THPUtils_packInt64Array.
* Throw python errors.
* Use fallthrough and fix fallthrough generation for native functions.
* is_cuda is a property, not a method.
Allow in-place operations on views
Adds VariableViewImpl, a subclass of VariableImpl which has a pointer to
the base Variable on which it is a view. In-place operations on views
change the grad_fn of the base.
Note that in-place operations only work on views that are the first output of the function that created them. All C++/ATen implemented functions have this behavior, but it's possible to write Python-implemented autograd functions that do not. In-place operations on these view will raise an exception.
Fixes#3313
This includes some changes to the dispatch code for torch.xxx functions:
- Since Variable.addmm is an instance-method, the self argument has to
come first. The dispatch code swaps the first two arguments if
necessary to suppor the deprecated signatures where 'alpha' or 'beta'
comes before the 'self' tensor.
- Delete IMPLEMENT_STATELESS_REVERSED. These functions require output
arguments to be passed in using the keyword 'out'. They were meant to
handle torch.gt(out, a, b), but we haven't allowed that for a while.
This removes the StochasticFunctions for bernoulli, multinomial, and
normal and replaces them with classes in the torch.distributions
package. Each distribution supports the differentiable log_prob function
that returns the log of the pdf/pmf of the samples.
The current StochasticFunction implementation has a few problems: it can
be painful to use when there are multiple stochastic outputs which need
to be back-propagated through. It also requires that we store grad_fns
on Variables that have requires_grad=False in order to find stochastic
nodes.
