OSSForks/pytorch
1
0
Fork 0
mirror of https://github.com/zebrajr/pytorch.git synced 2025-12-07 12:21:27 +01:00
Code Issues Actions 37 Packages Projects Releases Wiki Activity
fe3ecfe0cf
pytorch/torch/utils/data
History
Kevin Tse f04bb519f5 [DataPipe] Change DataPipe display name in profiler (#100042) 
Script:
```python
from torchdata.datapipes.iter import IterableWrapper
from torchdata.dataloader2 import DataLoader2, MultiProcessingReadingService

ls = range(16)
dp = IterableWrapper(ls).map(fn_2).map(fn_3).map(fn_4)

rs = MultiProcessingReadingService(num_workers=0, main_prefetch_cnt=0, worker_prefetch_cnt=0)
dl2 = DataLoader2(dp, reading_service=rs)

with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU]) as prof:
    for x in dl2:
        pass
```

Output before:
```
---------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------
                                               Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls
---------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------
             enumerate(DataPipe)#MapperIterDataPipe        76.37%       1.419ms       213.08%       3.959ms      80.796us            49
    enumerate(DataPipe)#IterableWrapperIterDataPipe        12.70%     236.000us        12.70%     236.000us      13.882us            17
...
```

Output after:
```
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------
Mapper(datapipe=Mapper, fn=fn_4, input_col=None, out...        29.79%     645.000us        99.17%       2.147ms     126.294us            17
Mapper(datapipe=IterableWrapper, fn=fn_2, input_col=...        29.24%     633.000us        42.96%     930.000us      54.706us            17
Mapper(datapipe=Mapper, fn=fn_3, input_col=None, out...        24.76%     536.000us        68.59%       1.485ms      87.353us            17
IterableWrapper(deepcopy=True, iterable=range(0, 16)...        10.58%     229.000us        10.58%     229.000us      13.471us            17
...
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/100042
Approved by: https://github.com/ejguan
2023-05-03 21:36:13 +00:00
..
_utils Add support for pin memory on custom device. (#97621) 2023-03-29 23:45:52 +00:00
datapipes [DataPipe] Change DataPipe display name in profiler (#100042) 2023-05-03 21:36:13 +00:00
__init__.py [DataLoader] Removing DataLoader2 related code (#88848) 2022-11-11 22:27:01 +00:00
backward_compatibility.py
dataframes_pipes.ipynb
dataloader.py [DataLoader] Follow-up Fix: TypeVars of Sampler (#100409) 2023-05-03 17:38:31 +00:00
dataset.py Simplify by using yield from in torch/utils/data (#97839) 2023-03-29 04:51:26 +00:00
distributed.py [DataLoader] Follow-up Fix: TypeVars of Sampler (#100409) 2023-05-03 17:38:31 +00:00
graph_settings.py Forward fix for DataLoader to accept custom Sharding DataPipe (#97287) 2023-04-05 22:33:37 +00:00
graph.py [DataLoader] Replacing traverse function with traverse_datapipes (#85667) 2022-09-27 19:58:15 +00:00
sampler.py [DataLoader] Follow-up Fix: TypeVars of Sampler (#100409) 2023-05-03 17:38:31 +00:00
standard_pipes.ipynb
typing.ipynb