Ryan Spring 4f8b986e28 Implement Tanh Gelu Approximation (#61439) ... Summary: 1. Implements https://github.com/pytorch/pytorch/issues/39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - https://github.com/pytorch/xla/pull/3039 Pull Request resolved: https://github.com/pytorch/pytorch/pull/61439 Reviewed By: VitalyFedyunin Differential Revision: D33894937 Pulled By: jbschlosser fbshipit-source-id: b65e8fb6ea66168af8f34f45ed50e92737a33851 (cherry picked from commit 6e986f91a9)		2022-02-14 03:40:32 +00:00
..
cuda	Implement Tanh Gelu Approximation (#61439)	2022-02-14 03:40:32 +00:00
fuser	[AutoAccept][Codemod][FBSourceClangFormatLinter] Daily arc lint --take CLANGFORMAT	2022-01-14 06:10:57 -08:00