Summary:
This is intended for use with torchft when we need to do a streaming state dict transfer. This is strictly superior to the prior streaming method in torchft as this supports all tensor subclasses such as DTensor.
This supports 100% of the inputs to torch.save/load but is not wire compatible nor intended to have any backwards compatibility.
Security wise this fully supports weights_only and defaults to True. It does use pickle for some metadata but uses weights_only for the metadata.
Adapted from:
https://github.com/pytorch/torchft/pull/101https://github.com/pytorch/torchft/pull/54
Test Plan:
pytest test/distributed/test_serialization.py
Pull Request resolved: https://github.com/pytorch/pytorch/pull/146555
Approved by: https://github.com/fegin, https://github.com/mikaylagawarecki
Co-authored-by: Krishn Parasar <76171905+Krishn1412@users.noreply.github.com>
