Wanchao Liang
599ae95d1a
[dtensor] use stack to manage mesh resources ( #101202 )
...
This PR changes the context manager behavior of device mesh, now we use
a mesh env to track the current mesh and save the mesh to a stack so
that we can allow nested context manager
Pull Request resolved: https://github.com/pytorch/pytorch/pull/101202
Approved by: https://github.com/wz337
2023-05-11 23:48:36 +00:00
Wanchao Liang
55a1dc7f88
[dtensor] redistributed by default take self mesh instead ( #99060 )
...
This PR switches redistribute to default use self mesh instead of
the global mesh, which is more user friendly
Pull Request resolved: https://github.com/pytorch/pytorch/pull/99060
Approved by: https://github.com/mrshenli
2023-04-14 05:14:28 +00:00
Shen Li
02179827cb
[Easy] Include SPMD and DTensor files in UFMT checks ( #98148 )
...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/98148
Approved by: https://github.com/fegin
2023-04-02 15:34:49 +00:00
Wanchao Liang
e9c4904915
[dtensor] remove custom dispatch op ( #95629 )
...
Since we removed all custom dispatch ops, we can safely delete this
table as we won't use it for other purposes
Pull Request resolved: https://github.com/pytorch/pytorch/pull/95629
Approved by: https://github.com/XilunWu
2023-03-28 02:25:45 +00:00
Kazuaki Ishizaki
35fd5c548e
Fix typos under torch/distributed directory ( #95638 )
...
This PR fixes typos in comments and messages of `.py` files under torch/distributed directory
Pull Request resolved: https://github.com/pytorch/pytorch/pull/95638
Approved by: https://github.com/usamah1 , https://github.com/H-Huang , https://github.com/kit1980
2023-03-27 21:13:44 +00:00
Wanchao Liang
2a1cb9640c
[dtensor] support creating DTensor in submesh ( #95458 )
...
This PR supports creating DTensor in a submesh, if the rank is not
participating in the mesh, we assign the local tensor to be empty
tensor, and do nothing in the operator dispatch
Differential Revision: [D43643577](https://our.internmc.facebook.com/intern/diff/D43643577 )
Pull Request resolved: https://github.com/pytorch/pytorch/pull/95458
Approved by: https://github.com/XilunWu
2023-02-28 17:54:26 +00:00
Wanchao Liang
bb9a05b116
[dtensor] use tracing for metadata prop ( #95456 )
...
This PR uses tracing for metadata prop, so that we can get correct
shape/stride metadata without manual calculation by ourselves.
The follow up PR on this would be adopt tracing for the sharding
prop itself
Differential Revision: [D43643578](https://our.internmc.facebook.com/intern/diff/D43643578 )
Pull Request resolved: https://github.com/pytorch/pytorch/pull/95456
Approved by: https://github.com/XilunWu
2023-02-28 17:54:22 +00:00
Wanchao Liang
680fc84e7b
[dtensor] group public APIs together ( #94524 )
...
This PR groups distribute_tensor/module to api.py
rename some to non-public (ToTensor/FromTensor)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94524
Approved by: https://github.com/XilunWu
2023-02-10 23:40:34 +00:00
Wanchao Liang
b072245178
[dtensor][4/N] refactor dispatching logic and add propagator ( #90733 )
...
This PR refactors the dispatching logic to make it more clean, and
isolate the sharding propagation logic out to a separate class.
This is so that we can implement more complicated propagation features
later.
Differential Revision: [D42876251](https://our.internmc.facebook.com/intern/diff/D42876251 )
Pull Request resolved: https://github.com/pytorch/pytorch/pull/90733
Approved by: https://github.com/XilunWu , https://github.com/fduwjj
2023-02-01 05:02:11 +00:00
fduwjj
77f336600a
[PT-D] Enable Meta Tensor Support for DTensor ( #92652 )
...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/92652
Approved by: https://github.com/XilunWu , https://github.com/wanchaol
2023-01-26 04:54:57 +00:00
Wanchao Liang
7afba50508
[dtensor] delete unused torch_function ( #90449 )
...
torch_function is not actually getting used yet today, deleting
it first and we can revisit once we really need it
Pull Request resolved: https://github.com/pytorch/pytorch/pull/90449
Approved by: https://github.com/fduwjj
2022-12-10 01:29:02 +00:00
Wanchao Liang
bf23e0bdbd
[dtensor] ufmt distributed._tensor ( #89967 )
...
cmd: `ufmt format torch/distributed/_tensor`
copy from Andrew:
Notes
For VSCode users,
Install ufmt: https://pypi.org/project/ufmt/
Install VSCode ufmt extension: https://marketplace.visualstudio.com/items?itemName=omnilib.ufmt
Include in settings.json:
```
{
"[python]": {
"editor.defaultFormatter": "omnilib.ufmt",
"editor.formatOnSave": true,
},
}
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/89967
Approved by: https://github.com/fduwjj
2022-12-01 20:58:13 +00:00
Wanchao Liang
4b945967de
[dtensor] PART 2: move DTensor abstraction and APIs to core distributed ( #88176 )
...
This PR moves the core DTensor abstraction and high level APIs to
torch.distributed._tensor folder, which includes the following:
1. DTensor class
2. high level APIs (distribute_tensor/module)
3. dispatching logic
4. redistribute logic
part of https://github.com/pytorch/pytorch/issues/88838
Pull Request resolved: https://github.com/pytorch/pytorch/pull/88176
Approved by: https://github.com/fduwjj
2022-11-16 08:07:41 +00:00
