OSSForks/pytorch
1
0
Fork 0
mirror of https://github.com/zebrajr/pytorch.git synced 2025-12-07 12:21:27 +01:00
Code Issues Actions 38 Packages Projects Releases Wiki Activity
949ce294ff
pytorch/caffe2/python
History
Willy Blandin 949ce294ff fix race condition in text_file_reader.py 
Summary:
This fixes a race condition in text_file_reader.py.

For example in `fbcode/caffe2/caffe2/fb/text/stats.py`, in `compute_meta`, we build an execution step `read` such as:

```
.
└── step_read
    ├── net_reader
    │   ├── op_TextFileReaderRead
    │   └── op_IsEmpty
    └── net_consume:n
        └── op_Tokenize
```

Note that in `workspace.cc`, we check should_stop between each _step_ and each _net_, not between _ops_

Let's say we have 2 workers, here is a faulty interleaving of threads:
- 1 executes TextFileReaderRead
- 2 executes TextFileReaderRead
- 1 executes IsEmpty and sets should_stop to False
- 2 executes IsEmpty and sets should_stop to True
- 1 checks should_stop before running net_consume:n
- 1 stops
- 2 checks should_stop before running net_consume:n
- 2 stops

That's an issue, because 1 did read data from the file but did not run the processing step (consume:n) for this data.

Reviewed By: dzhulgakov

Differential Revision: D4203729

fbshipit-source-id: eabd94ea995527ec52fa137a8b63c277f7e4dd96
2016-11-29 15:18:36 -08:00
..
examples fbsync at f5a877 2016-11-18 15:41:06 -08:00
layers fbsync at f5a877 2016-11-18 15:41:06 -08:00
mint more build updates: 2016-08-02 23:28:23 -07:00
models fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
operator_test fbsync at f5a877 2016-11-18 15:41:06 -08:00
_import_c_extension.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
BREW more build fixing 2016-09-07 23:30:35 -07:00
caffe_translator_test.py protected legacy_pad_, replace DeleteDropout with is_test=True 2016-07-29 11:44:55 -07:00
caffe_translator.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
cnn.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
context.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
control_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
control.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
convnet_benchmarks_test.py chunky sync - build scripts to be written 2016-07-21 10:16:42 -07:00
convnet_benchmarks.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
core_gradients_test.py chunky sync 2016-09-06 15:55:19 -07:00
core_test.py sync 2016-08-10 11:02:15 -07:00
core.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
data_parallel_model_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
data_parallel_model.py fbsync at f5a877 2016-11-18 15:41:06 -08:00
dataio_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
dataio.py fbsync at f5a877 2016-11-18 15:41:06 -08:00
dataset.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
db_test.py chunky sync 2016-09-06 15:55:19 -07:00
device_checker.py chunky sync 2016-09-06 15:55:19 -07:00
dyndep.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
experiment_util.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
extension_loader.py fbsync 2016-10-07 13:08:53 -07:00
gradient_check_test.py chunky sync - build scripts to be written 2016-07-21 10:16:42 -07:00
gradient_checker.py fbsync 2016-10-07 13:08:53 -07:00
hsm_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
hsm_util.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
hypothesis_test_util.py fbsync at f5a877 2016-11-18 15:41:06 -08:00
hypothesis_test.py position weighted embedding 2016-11-29 15:18:35 -08:00
layer_model_helper.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
layer_model_instantiator.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
load_save_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
memonger_test.py add untracked files 2016-07-21 11:26:41 -07:00
memonger.py add untracked files 2016-07-21 11:26:41 -07:00
model_device_test.py sync 2016-07-28 15:06:43 -07:00
model_helper.py adding more things to the list of known operators in model_helper 2016-11-29 15:18:35 -08:00
muji_test.py chunky sync - build scripts to be written 2016-07-21 10:16:42 -07:00
muji.py fbsync 2016-10-07 13:08:53 -07:00
net_builder_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
net_builder.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
net_drawer.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
pipeline.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
pybind_state_gpu.cc fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
pybind_state_mkl.cc Expose MKLMemory to the Python Feed and Fetch interface, and misc changes 2016-11-29 15:18:36 -08:00
pybind_state.cc Expose MKLMemory to the Python Feed and Fetch interface, and misc changes 2016-11-29 15:18:36 -08:00
pybind_state.h fbsync at f5a877 2016-11-18 15:41:06 -08:00
python_op_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
queue_util.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
record_queue.py chunky sync 2016-09-06 15:55:19 -07:00
schema_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
schema.py fbsync at f5a877 2016-11-18 15:41:06 -08:00
scope_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
scope.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
session_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
session.py fbsync at f5a877 2016-11-18 15:41:06 -08:00
snapshot_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
snapshot.py fbsync at f5a877 2016-11-18 15:41:06 -08:00
sparse_to_dense_mask_test.py sync 2016-08-10 11:02:15 -07:00
task.py fbsync at f5a877 2016-11-18 15:41:06 -08:00
test_util.py chunky sync 2016-09-06 15:55:19 -07:00
text_file_reader.py fix race condition in text_file_reader.py 2016-11-29 15:18:36 -08:00
timeout_guard.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
toy_regression_test.py sync 2016-08-10 11:02:15 -07:00
tt_core_test.py sync 2016-08-10 11:02:15 -07:00
tt_core.py sync 2016-08-10 11:02:15 -07:00
utils.py fbsync 2016-10-07 13:08:53 -07:00
visualize.py chunky sync 2016-05-13 14:43:48 -07:00
workspace_test.py Expose MKLMemory to the Python Feed and Fetch interface, and misc changes 2016-11-29 15:18:36 -08:00
workspace.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00