Shen Li 30e45bb133 Enable GPU-to-GPU comm in TensorPipeAgent (#44418) ... Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/44418 This commit uses TensorPipe's cuda_ipc channel to conduct cross-process same-machine GPU-to-GPU communication. On the sender side, `TensorPipeAgent` grabs a stream to each device used by the message, let these streams wait for current streams, and passes the streams to TensorPipe `CudaBuffer`. On the receiver side, it also grabs a stream for each device used in the message, and uses these streams to receive tensors and run user functions. After that, these streams are then used for sending the response back to the sender. When receiving the response, the sender will grab a new set of streams and use them for TensorPipe's `CudaBuffer`. If device maps are provided, `TensorPipeAgent::send` will return a derived class of `CUDAFuture`, which is specifically tailored for RPC Messages. TODOs: 1. Enable sending CUDA RPC to the same process. 2. Add a custom CUDA stream pool. 3. When TensorPipe addressed the error for `cudaPointerGetAttributes()`, remove `cuda:0` context initialization code in `backend_registry.py`. 4. When TensorPipe can detect availability of peer access, enable all tests on platforms without peer access. Differential Revision: D23626207 Test Plan: Imported from OSS Reviewed By: lw Pulled By: mrshenli fbshipit-source-id: d30e89e8a98bc44b8d237807b84e78475c2763f0		2021-01-14 13:55:41 -08:00
..
_internal	Enable GPU-to-GPU comm in TensorPipeAgent (#44418)	2021-01-14 13:55:41 -08:00
__init__.py	fix assert_allclose doesnt check shape (#47580)	2020-11-10 15:03:25 -08:00
check_kernel_launches.py	Fix check_kernel_launches.py for macros and provide extended context (#49365)	2020-12-14 22:09:33 -08:00