mirror of
https://github.com/zebrajr/pytorch.git
synced 2025-12-07 00:21:07 +01:00
7342251281
Happy to split this PR more if it helps. This PR adds functorch.grad support for autograd.Function. There's a lot going on; here is the high level picture and there are more details as comments in the code. Mechanism (PyOperator) - Somehow, autograd.Function needs to dispatch with functorch. This is necessary because every layer of functorch needs to see the autograd.Function; grad layers need to preserve the backward pass. - The mechanism for this is via PyOperator. If functorch transforms are active, then we wrap the autograd.Function in a `custom_function_call` PyOperator where we are able to define various rules for functorch transforms. - `custom_function_call` has a rule for the functorch grad transform. autograd.Function changes - I needed to make some changes to autograd.Function to make this work. - First, this PR splits autograd.Function into a _SingleLevelFunction (that works with a single level of functorch transform) and autograd.Function (which works with multiple levels). This is necessary because functorch's grad rule needs some way of specifying a backward pass for that level only. - This PR changes autograd.Function's apply to eitehr call `custom_function_call` (if functorch is active) or super().apply (if functorch isn't active). Testing - Most of this PR is just testing. It creates an autograd.Function OpInfo database that then gets passed to the functorch grad-based tests (grad, vjp, vjpvjp). - Since functorch transform tests are autogenerated from OpInfo tests, this is the easiest way to test various autograd.Function with functorch. Future - jvp and vmap support coming next - better error message (functorch only supports autograd.Function that have the optional setup_context staticmethod) - documentation to come when we remove the feature flag Pull Request resolved: https://github.com/pytorch/pytorch/pull/89860 Approved by: https://github.com/soulitzer |
||
|---|---|---|
| .. | ||
| _dynamo | ||
| __init__.pyi.in | ||
| _autograd.pyi | ||
| _cudnn.pyi | ||
| _distributed_autograd.pyi | ||
| _distributed_c10d.pyi | ||
| _distributed_rpc_testing.pyi | ||
| _distributed_rpc.pyi | ||
| _functions.pyi | ||
| _functorch.pyi | ||
| _itt.pyi | ||
| _lazy_ts_backend.pyi | ||
| _lazy.pyi | ||
| _monitor.pyi | ||
| _nn.pyi.in | ||
| _nvtx.pyi | ||
| _onnx.pyi | ||
| _profiler.pyi | ||
| _VariableFunctions.pyi.in | ||
| _verbose.pyi | ||
| build.bzl | ||
| return_types.pyi.in | ||