[torch][cuda] fix race condition in cuda initialization (#143238)

The access to lazy init callbacks (`_lazy_seed_tracker` and `_queued_calls`) is not synchronized with the initialization lock.

This exposes us to the following race:
1. start `_lazy_init`
2. take `_initialization_lock`
3. flush `_queued_calls` and run them all
4. another thread comes in and uses `_lazy_call` to put something on the queue (in our case, the `manual_seed`)
5. original thread finishes initializing, but never runs that call

Pull Request resolved: https://github.com/pytorch/pytorch/pull/143238
Approved by: https://github.com/ngimel
This commit is contained in:
Michael Suo 2024-12-14 07:41:22 +00:00 committed by PyTorch MergeBot
parent 28d8297712
commit 9933e59c2b

View File

@ -245,6 +245,7 @@ def is_initialized():
def _lazy_call(callable, **kwargs):
with _initialization_lock:
if is_initialized():
callable()
else: