Ryan Spring f499ab9cef Implement Tanh Gelu Approximation (#61439) ... Summary: 1. Implements https://github.com/pytorch/pytorch/issues/39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - https://github.com/pytorch/xla/pull/3039 Pull Request resolved: https://github.com/pytorch/pytorch/pull/61439 Reviewed By: mikaylagawarecki Differential Revision: D33744717 Pulled By: jbschlosser fbshipit-source-id: d64532a562ed53247bb4fa52bb16722634d5c187 (cherry picked from commit 4713dd9cca)		2022-01-28 16:59:09 +00:00
..
api	Implement Tanh Gelu Approximation (#61439)	2022-01-28 16:59:09 +00:00
c10d	Add support for deleteKey for FileStore (#69953)	2022-01-07 06:20:59 -08:00
common
dist_autograd	Fix distributed autograd gradients synchronization (#57792)	2021-05-09 17:32:59 -07:00
jit	Fix bug where frozen mod not used for OFI #68903 (#71436)	2022-01-27 23:37:50 +00:00
lazy	[LT] Remove torch::lazy::convertShapes (#71291)	2022-01-14 12:06:39 -08:00
lite_interpreter_runtime	Back out "Revert D30710710: [Pytorch Edge] Support profiling kineto events from external source" (#66421)	2021-10-12 10:55:29 -07:00
monitor	torch/monitor: add pybind (#69567)	2022-01-12 13:35:11 -08:00
rpc	Remove ProcessGroup from TensorPipeAgent initialization (#68128)	2021-11-11 12:28:55 -08:00
tensorexpr	[TensorExpr] Support scalar intermediate and output values. (#71186)	2022-01-26 06:32:51 +00:00
__init__.py	remediation of S205607	2020-07-17 17:19:47 -07:00