fduwjj
983fd5ba79
[2D][TP] Enable DDP TP integration with unit test ( #106583 )
...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/106583
Approved by: https://github.com/kumpera , https://github.com/fegin , https://github.com/wanchaol
ghstack dependencies: #107313
2023-08-17 02:54:17 +00:00
fduwjj
487ebcac3b
Clean up unsed MHA code to avoid confusion ( #105956 )
...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/105956
Approved by: https://github.com/wz337 , https://github.com/ezyang , https://github.com/wanchaol
2023-07-27 17:10:17 +00:00
fduwjj
953aa6d90e
[TP] Enable more generic attn in Tensor Parallelism ( #100508 )
...
To make TP more generic for Attention module, we come up with this new col/rowwise parallel style.
Basically, the idea behind is that:
We only do DTensor op for Col/Rowwise sharded part. For the rest of ATen ops, we will leave it to Tensor ops.
And we set this behavior as default for Colwise and Rowwise parallel style. If people want to customize it, they can always pass in different prepare_input or prepare_output
Pull Request resolved: https://github.com/pytorch/pytorch/pull/100508
Approved by: https://github.com/wanchaol
2023-05-07 18:15:49 +00:00
Svetlana Karslioglu
d425da8bf3
Replace master with main in links and docs/conf.py ( #100176 )
...
Fixes #ISSUE_NUMBER
Pull Request resolved: https://github.com/pytorch/pytorch/pull/100176
Approved by: https://github.com/albanD , https://github.com/malfet
2023-05-02 18:20:32 +00:00
fduwjj
b209d8fa0d
[PT-D][Sequence Parallelism] Enable DTensor based Naive sequence parallelism ( #94369 )
...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94369
Approved by: https://github.com/wanchaol
2023-02-16 21:21:00 +00:00
Wanchao Liang
cd9ca4c73f
[tp] additional doc fixes ( #94786 )
...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94786
Approved by: https://github.com/fduwjj
2023-02-15 21:25:26 +00:00
fduwjj
39511697d4
[PT-D][BE] Update 2D parallelism API name and docs ( #94771 )
...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94771
Approved by: https://github.com/wanchaol
2023-02-14 08:13:15 +00:00
PyTorch MergeBot
28ed0bdb37
Revert "[tp] additional doc fixes ( #94786 )"
...
This reverts commit 7522ca55f1https://github.com/pytorch/pytorch/pull/94786 on behalf of https://github.com/huydhn due to Sorry for reverting your PR, but the doc failure looks related and they are also failing in trunk 7522ca55f1
2023-02-14 05:43:37 +00:00
Wanchao Liang
7522ca55f1
[tp] additional doc fixes ( #94786 )
...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94786
Approved by: https://github.com/fduwjj
2023-02-14 04:52:04 +00:00
Wanchao Liang
2db12e3844
[tp] minor update to TP docs ( #94748 )
...
minor update to TP docs for beta release
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94748
Approved by: https://github.com/fduwjj
2023-02-13 21:54:19 +00:00
fduwjj
41e3189222
[PT-D][Tensor parallelism] Add documentations for TP ( #94421 )
...
This is far from completed and we will definitely polish it down the road.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94421
Approved by: https://github.com/wz337
2023-02-09 02:31:06 +00:00
Wanchao Liang
4451eb24e6
Move tensor_parallel out to distributed.tensor folder ( #89878 )
...
This PR moves tensor parallel from torch.distributed._tensor.parallel
to torch.distributed.tensor.parallel, to prepare for beta release
Pull Request resolved: https://github.com/pytorch/pytorch/pull/89878
Approved by: https://github.com/fduwjj
2022-11-30 22:13:10 +00:00
