pytorch/docs/source
Emilio Castillo 5472426b9f Reset DataLoader workers instead of creating new ones (#35795)
Summary:
This PR needs discussion as it changes the behavior of `DataLoader`. It can be closed if its not considered a good practice.

Currently, the `DataLoader` spawns a new `_BaseDataLoaderIter` object every epoch,
In the case of the multiprocess DataLoader, every epoch the worker processes are re-created and they make a copy of the original `Dataset` object.
If users want to cache data or do some tracking on their datasets, all their data will be wiped out every epoch. Notice that this doesn't happen when the number of workers is 0. giving some inconsistencies with the multiprocess and serial data loaders.

This PR keeps the `_BaseDataLoaderIter` object alive and just resets it within epochs, so the workers remain active and so their own `Dataset` objects. People seem to file issues about this often.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/35795

Reviewed By: ailzhang

Differential Revision: D23426612

Pulled By: VitalyFedyunin

fbshipit-source-id: e16950036bae35548cd0cfa78faa06b6c232a2ea
2020-09-01 11:48:00 -07:00
..
_static Improve documentation around builtin functions (#30347) 2019-12-04 13:50:40 -08:00
_templates Revert "Revert D21337640: [pytorch][PR] Split up documentation into subpages and clean up some warnings" (#37778) 2020-05-04 14:32:35 -07:00
_templates-stable Add layout.html to template for stable docs (#33770) 2020-03-04 03:14:52 -08:00
community Add MSFT Owners to the Windows Maintainership (#42280) 2020-07-30 08:22:13 -07:00
notes Update determinism documentation (#41692) 2020-08-31 21:06:24 -07:00
rpc Revert "Revert D21337640: [pytorch][PR] Split up documentation into subpages and clean up some warnings" (#37778) 2020-05-04 14:32:35 -07:00
scripts Optimize SiLU (Swish) op in PyTorch (#42976) 2020-08-16 13:21:57 -07:00
__config__.rst
amp.rst Added index_put to promotelist (#41035) 2020-07-07 20:36:55 -07:00
autograd.rst Update docs feature classifications (#39966) 2020-06-24 15:35:59 -07:00
bottleneck.rst
checkpoint.rst
complex_numbers.rst Doc note for complex (#41252) 2020-07-16 08:53:27 -07:00
conf.py restore old documentation references (#39086) 2020-07-09 15:20:10 -07:00
cpp_extension.rst correct some cpp extension code usages and documents (#39766) 2020-06-10 08:31:22 -07:00
cpp_index.rst Add C++ Landing Page (#38450) 2020-05-14 16:02:01 -07:00
cuda.rst Fix most documentation warnings (#27782) 2019-10-13 10:34:01 -07:00
cudnn_persistent_rnn.rst Revert "Revert D21337640: [pytorch][PR] Split up documentation into subpages and clean up some warnings" (#37778) 2020-05-04 14:32:35 -07:00
cudnn_rnn_determinism.rst [doc] Add LSTM non-deterministic workaround (#40893) 2020-07-21 16:20:02 -07:00
data.rst Reset DataLoader workers instead of creating new ones (#35795) 2020-09-01 11:48:00 -07:00
distributed.rst Publish all_gather_object and gather_object docs (#43772) 2020-08-31 13:28:00 -07:00
distributions.rst Revert "Revert D21337640: [pytorch][PR] Split up documentation into subpages and clean up some warnings" (#37778) 2020-05-04 14:32:35 -07:00
dlpack.rst
docutils.conf Revert "Revert D21337640: [pytorch][PR] Split up documentation into subpages and clean up some warnings" (#37778) 2020-05-04 14:32:35 -07:00
fft.rst Adds fft namespace (#41911) 2020-08-06 00:20:50 -07:00
futures.rst [JIT x RPC] Consolidate Future type class and Future impl class (#40406) 2020-06-24 01:44:49 -07:00
hub.rst Fix torch.hub.hub_dir inconsistencies (#38969) 2020-05-26 21:06:52 -07:00
index.rst Adds torch.linalg namespace (#42664) 2020-08-07 10:18:30 -07:00
jit_builtin_functions.rst
jit_language_reference.rst Error printing extension support for multiline errors (#43807) 2020-09-01 10:02:43 -07:00
jit_python_reference.rst [JIT] Add support for with statements (#34705) 2020-06-18 16:57:18 -07:00
jit_unsupported.rst [JIT] Make torch.unique_consecutive compatible (#39339) 2020-06-02 14:54:29 -07:00
jit.rst Grammatical corrections (#43473) 2020-08-25 12:09:14 -07:00
linalg.rst Add torch.linalg.norm (#42749) 2020-08-28 18:28:33 -07:00
math-quantizer-equation.png adding quantization.rst file for quantization feature (#27559) 2019-10-09 16:45:09 -07:00
mobile_optimizer.rst optimize_for_mobile: bring packed params to root module (#42740) 2020-08-08 15:53:20 -07:00
model_zoo.rst
multiprocessing.rst Update docs for master to remove Python 2 references (#36336) 2020-04-16 10:15:48 -07:00
name_inference.rst Add arcosh, arcsinh and arctanh to unary ops (#38388) 2020-06-04 11:40:55 -07:00
named_tensor.rst Easier english updated tech docs (#42016) 2020-07-24 14:36:17 -07:00
nn.functional.rst Added SiLU activation function (#41034) 2020-07-10 07:37:30 -07:00
nn.init.rst Bag of documentation fixes; fix more sphinx warnings (#27850) 2019-10-15 07:31:14 -07:00
nn.rst Add Unflatten Module (#41564) 2020-07-21 07:43:02 -07:00
onnx.rst [ONNX] Update pytorch/onnx docs for new export API args (#39802) 2020-06-19 13:38:47 -07:00
optim.rst DOC: fail to build if there are warnings (#41335) 2020-07-28 22:33:44 -07:00
quantization-support.rst DOC: split quantization.rst into smaller pieces (#41321) 2020-07-25 23:59:40 -07:00
quantization.rst DOC: split quantization.rst into smaller pieces (#41321) 2020-07-25 23:59:40 -07:00
random.rst Remove duplicated entries in random.rst (#39725) 2020-06-10 16:51:15 -07:00
rpc.rst Minor RPC doc fixes (#43337) 2020-08-20 14:17:07 -07:00
sparse.rst Update docs feature classifications (#39966) 2020-06-24 15:35:59 -07:00
storage.rst
tensor_attributes.rst Update docs feature classifications (#39966) 2020-06-24 15:35:59 -07:00
tensor_view.rst Adds movedim method, fixes movedim docs, fixes view doc links (#43122) 2020-08-17 14:24:52 -07:00
tensorboard.rst Add method add_hparams to API doc (#27344) 2019-10-03 17:07:45 -07:00
tensors.rst Implementing NumPy-like function torch.heaviside() (#42523) 2020-08-31 15:54:56 -07:00
torch.nn.intrinsic.qat.rst DOC: split quantization.rst into smaller pieces (#41321) 2020-07-25 23:59:40 -07:00
torch.nn.intrinsic.quantized.rst DOC: split quantization.rst into smaller pieces (#41321) 2020-07-25 23:59:40 -07:00
torch.nn.intrinsic.rst DOC: split quantization.rst into smaller pieces (#41321) 2020-07-25 23:59:40 -07:00
torch.nn.qat.rst DOC: split quantization.rst into smaller pieces (#41321) 2020-07-25 23:59:40 -07:00
torch.nn.quantized.dynamic.rst DOC: split quantization.rst into smaller pieces (#41321) 2020-07-25 23:59:40 -07:00
torch.nn.quantized.rst DOC: split quantization.rst into smaller pieces (#41321) 2020-07-25 23:59:40 -07:00
torch.quantization.rst DOC: split quantization.rst into smaller pieces (#41321) 2020-07-25 23:59:40 -07:00
torch.rst Update determinism documentation (#41692) 2020-08-31 21:06:24 -07:00
type_info.rst DOC: split quantization.rst into smaller pieces (#41321) 2020-07-25 23:59:40 -07:00