OSSForks/pytorch
1
0
Fork 0
mirror of https://github.com/zebrajr/pytorch.git synced 2025-12-07 00:21:07 +01:00
Code Issues Actions 38 Packages Projects Releases Wiki Activity
acb2ad12e5
pytorch/caffe2/python
History
Aapo Kyrola acb2ad12e5 fix race condition at terminate 
Summary:
Looking at one segfault at exit (https://our.intern.facebook.com/intern/chronos/jobinstance/?jobinstanceid=911625597&smc=chronos_gp_admin_client&log_type=stderr&offset=0&pretty_logs=false) and it's coredump, only thing I can see that a FreeBlob() operator is called concurrently while a cudaMemcpyAsync (on thread 1) is crashing. FreeBlobOp is only called at data_workers _stop() (via utils.ResetBlobs()), and only code that could run a cudaMemcpyAsync that time is the fetcher -thread of data_workers that is enquing blobs.

Here are the stacks: P57455299

This is clearly a bug since we should only clear the scratch blobs after all threads are terminated, which happens at wait_for_finish().

I am not 100% sure this fixes all the segfaults, but at least this one was most likely caused by this.

Reviewed By: andrewwdye

Differential Revision: D5146278

fbshipit-source-id: ae00796706bfc4fee6823caf6529b62ab20c1cd3
2017-05-30 13:47:10 -07:00
..
docs fixed document generator for github 2017-05-15 11:40:46 -07:00
examples add optimizer support to data_parallel_model; Use MomentumSGDUpdate 2017-05-30 12:49:57 -07:00
helpers Rework initializers to pass a class not object 2017-05-30 09:06:56 -07:00
layers Add batch normalization layer 2017-05-26 16:46:52 -07:00
mint doxygen python block added 2017-03-29 06:46:16 -07:00
mkl MKL related files with review comments incorporated 2017-04-25 00:31:29 -07:00
modeling Rework initializers to pass a class not object 2017-05-30 09:06:56 -07:00
models new resnet building with brew 2017-05-15 22:47:24 -07:00
operator_test Mark in-place GPU dropout as broken, add test 2017-05-30 12:35:22 -07:00
predictor Fix export of SparseToDense layer. 2017-05-25 21:46:28 -07:00
rnn CuDNN comparison mode 2017-05-20 15:19:43 -07:00
_import_c_extension.py doxygen python block added 2017-03-29 06:46:16 -07:00
attention.py ModelHelper.create_param, Initializer abstraction and ParameterInfo for optimizers 2017-05-25 22:03:15 -07:00
brew_test.py arg scope in ModelHelper 2017-05-12 11:18:59 -07:00
brew.py String-related fixes for Python 3 2017-05-26 16:04:32 -07:00
caffe_translator_test.py Allow test discovery in caffe2/python/ 2017-03-14 18:16:41 -07:00
caffe_translator.py Fixed typo caffe_translator.py, fixes bug #397 2017-05-24 12:18:32 -07:00
checkpoint_test.py Adds interfaces to check the existence of a DB 2017-04-11 14:07:49 -07:00
checkpoint.py Adds interfaces to check the existence of a DB 2017-04-11 14:07:49 -07:00
CMakeLists.txt CMake completions work 2017-01-11 16:59:22 -08:00
cnn.py cnnmodelhelper deprecate warning 2017-05-18 23:35:26 -07:00
context_test.py Make ContextManager thread-safe 2017-02-13 19:45:35 -08:00
context.py doxygen python block added 2017-03-29 06:46:16 -07:00
control_test.py fbsync. TODO: check if build files need update. 2016-11-15 00:00:46 -08:00
control.py doxygen python block added 2017-03-29 06:46:16 -07:00
convnet_benchmarks_test.py
convnet_benchmarks.py doxygen python block added 2017-03-29 06:46:16 -07:00
core_gradients_test.py Fix backward pass computation when an input is used in a Fill-op input for shape 2017-04-11 19:32:22 -07:00
core_test.py String-related fixes for Python 3 2017-05-26 16:04:32 -07:00
core.py automatically infer device scope for param 2017-05-30 12:02:19 -07:00
crf.py Integrate CRF in DeepText + New caffe2 operator for viterbi decode 2017-05-01 20:39:41 -07:00
data_parallel_model_test.py add optimizer support to data_parallel_model; Use MomentumSGDUpdate 2017-05-30 12:49:57 -07:00
data_parallel_model.py add optimizer support to data_parallel_model; Use MomentumSGDUpdate 2017-05-30 12:49:57 -07:00
data_workers_test.py add test for input order 2017-05-19 23:46:38 -07:00
data_workers.py fix race condition at terminate 2017-05-30 13:47:10 -07:00
dataio_test.py Stop multi_reader if we run out of data before max_examples 2017-03-10 18:03:57 -08:00
dataio.py doxygen python block added 2017-03-29 06:46:16 -07:00
dataset.py doxygen python block added 2017-03-29 06:46:16 -07:00
db_test.py String-related fixes for Python 3 2017-05-26 16:04:32 -07:00
device_checker.py doxygen python block added 2017-03-29 06:46:16 -07:00
dyndep.py doxygen python block added 2017-03-29 06:46:16 -07:00
empty.so Adding video data layer for caffe2 2017-05-05 14:16:38 -07:00
experiment_util.py Port Xray OC workflow to elastic_data_parallel_model 2017-05-01 00:32:47 -07:00
extension_loader.py Make extension loader properly handle visibility. 2017-03-30 14:38:38 -07:00
gradient_check_test.py gradient checker for nets 2017-03-28 13:03:14 -07:00
gradient_checker.py MultiRNNCell (Caffe2) 2017-05-03 10:02:31 -07:00
hsm_util.py doxygen python block added 2017-03-29 06:46:16 -07:00
hypothesis_test_util.py Add option to control the size of lengths tensor 2017-04-20 09:53:22 -07:00
hypothesis_test.py cuda unique op 2017-05-11 21:08:10 -07:00
layer_model_helper.py fix for realtime training. 2017-05-26 23:49:40 -07:00
layer_model_instantiator.py remove non-existing blobs from output_schema in layer_model_instantiator 2017-05-25 00:36:19 -07:00
layer_test_util.py Add batch normalization layer 2017-05-26 16:46:52 -07:00
layers_test.py Add batch normalization layer 2017-05-26 16:46:52 -07:00
load_save_test.py Allow Load operator to load into overriden names 2017-04-27 01:18:12 -07:00
lstm_benchmark.py lstm_benchmark use rnn_cell.LSTM multicell + assertion 2017-05-04 17:02:32 -07:00
memonger_test.py Change hypothesis settings for slow memonger test 2017-05-30 09:16:48 -07:00
memonger.py fix another bug related to in-place ops --> treat in-place ops like any other 2017-05-24 23:32:03 -07:00
mkl_test_util.py doxygen python block added 2017-03-29 06:46:16 -07:00
model_device_test.py Comment out NHWC Alexnet test for now 2017-01-23 13:59:29 -08:00
model_helper.py ModelHelper.create_param, Initializer abstraction and ParameterInfo for optimizers 2017-05-25 22:03:15 -07:00
mpi_python.cc Fix pybind11 module name for MPI helpers 2017-05-02 23:18:50 -07:00
muji_test.py
muji.py doxygen python block added 2017-03-29 06:46:16 -07:00
net_builder_test.py Allow test discovery in caffe2/python/ 2017-03-14 18:16:41 -07:00
net_builder.py doxygen python block added 2017-03-29 06:46:16 -07:00
net_drawer.py net_drawer: --input is required 2017-05-04 11:45:57 -07:00
net_printer_test.py Debug/Analysis tools for Jobs/ExecutionSteps 2017-02-06 17:31:20 -08:00
net_printer.py doxygen python block added 2017-03-29 06:46:16 -07:00
optimizer_test_util.py automatically infer device scope for param 2017-05-30 12:02:19 -07:00
optimizer_test.py automatically infer device scope for param 2017-05-30 12:02:19 -07:00
optimizer.py add optimizer support to data_parallel_model; Use MomentumSGDUpdate 2017-05-30 12:49:57 -07:00
pipeline.py Profiles pipe stages. 2017-05-24 12:02:03 -07:00
predictor_constants.py Re-apply #266 2017-04-25 21:17:04 -07:00
pybind_state_gpu.cc Cudnn v6 2017-02-28 17:46:33 -08:00
pybind_state_mkl.cc Expose MKLMemory to the Python Feed and Fetch interface, and misc changes 2016-11-29 15:18:36 -08:00
pybind_state.cc String-related fixes for Python 3 2017-05-26 16:04:32 -07:00
pybind_state.h String-related fixes for Python 3 2017-05-26 16:04:32 -07:00
python_op_test.py Allow PythonOp to access the workspace 2016-12-05 11:53:26 -08:00
queue_util.py doxygen python block added 2017-03-29 06:46:16 -07:00
record_queue.py doxygen python block added 2017-03-29 06:46:16 -07:00
recurrent.py Restore compatibility with protobuf2 2017-05-26 08:47:24 -07:00
rnn_cell.py DropoutCell as wrapper for another RNNCell 2017-05-24 11:36:45 -07:00
schema_test.py Prevent false overwriting of a field 2017-05-02 23:18:49 -07:00
schema.py String-related fixes for Python 3 2017-05-26 16:04:32 -07:00
scope_test.py Fix corruption of NameScope when exception is thrown 2017-04-24 22:46:27 -07:00
scope.py String-related fixes for Python 3 2017-05-26 16:04:32 -07:00
session_test.py Warn on setting blob on Scalar 2017-05-01 20:18:30 -07:00
session.py doxygen python block added 2017-03-29 06:46:16 -07:00
sparse_to_dense_mask_test.py String-related fixes for Python 3 2017-05-26 16:04:32 -07:00
task.py Remove application-specific comment. 2017-05-16 12:17:03 -07:00
test_util.py doxygen python block added 2017-03-29 06:46:16 -07:00
text_file_reader.py doxygen python block added 2017-03-29 06:46:16 -07:00
timeout_guard.py doxygen python block added 2017-03-29 06:46:16 -07:00
toy_regression_test.py sync 2016-08-10 11:02:15 -07:00
tt_core_test.py sync 2016-08-10 11:02:15 -07:00
tt_core.py doxygen python block added 2017-03-29 06:46:16 -07:00
utils.py String-related fixes for Python 3 2017-05-26 16:04:32 -07:00
visualize.py doxygen python block added 2017-03-29 06:46:16 -07:00
workspace_test.py String-related fixes for Python 3 2017-05-26 16:04:32 -07:00
workspace.py String-related fixes for Python 3 2017-05-26 16:04:32 -07:00