mirror of
https://github.com/zebrajr/pytorch.git
synced 2025-12-07 12:21:27 +01:00
23a3cef5d9
This PR implements _allgather_base, reduce_scatter, and _reduce_scatter_base in the MPI backend (ProcessGroupMPI), enabling support for Fully Sharded Data Parallel (FSDP) in environments that use MPI for distributed communication. ### Context As noted in https://github.com/pytorch/pytorch/issues/85628, FSDP currently supports only the NCCL backend. Due to this limitation, FSDP cannot run on legacy HPC environments or clusters that rely on MPI. By implementing just these three collective operations, we can enable FSDP to work with the MPI backend. These collectives are implemented in a similar manner to existing operations such as allgather. ### Testing We validated this PR using pytorch/build/bin/ProcessGroupMPITest with OpenMPI, and all tests passed successfully. Pull Request resolved: https://github.com/pytorch/pytorch/pull/150162 Approved by: https://github.com/H-Huang |
||
|---|---|---|
| .. | ||
| example | ||
| BackoffTest.cpp | ||
| CMakeLists.txt | ||
| CUDATest.cu | ||
| CUDATest.hpp | ||
| FileStoreTest.cpp | ||
| HashStoreTest.cpp | ||
| ProcessGroupGlooAsyncTest.cpp | ||
| ProcessGroupGlooTest.cpp | ||
| ProcessGroupMPITest.cpp | ||
| ProcessGroupNCCLErrorsTest.cpp | ||
| ProcessGroupNCCLTest.cpp | ||
| ProcessGroupUCCTest.cpp | ||
| StoreTestCommon.hpp | ||
| TCPStoreTest.cpp | ||
| TestUtils.hpp | ||