OSSForks/pytorch
1
0
Fork 0
mirror of https://github.com/zebrajr/pytorch.git synced 2025-12-07 12:21:27 +01:00
Code Issues Actions 38 Packages Projects Releases Wiki Activity
0ccc38b773
pytorch/caffe2/python
History
Danny Huang 0ccc38b773 [caffe2] adds Cancel to SafeDequeueBlobsOp and SafeEnqueueBlobsOp (#44495) 
Summary:
## Motivation

* To be able to make C2 ops cancellable so we can safely exit.
* Some C2 operators are now blocking thus being non-cancellable. If an error
  occurs we need to be able to safely stop all net execution so we can throw
  the exception to the caller.

* When an error occurs in a net or it got cancelled, running ops will have the
 `Cancel` method called.

* This diff adds `Cancel` method to the `SafeEnqueueBlobsOp`
and `SafeDequeueBlobsOp` to have the call queue->close() to force all the
 blocking ops to return.
* Adds unit test that verified the error propagation.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/44495

Test Plan:
## Unit Test added to verify that queue ops propagate errors
```
buck test caffe2/caffe2/python:hypothesis_test
```

Reviewed By: dzhulgakov

Differential Revision: D23236088

Pulled By: dahsh

fbshipit-source-id: daa90d9ee32483fb51195e269a52cf5987bb0a5a
2020-09-16 18:17:34 -07:00
..
benchmarks [caffe2] optimize 2/4-bit row-wise quantization (#387) 2020-06-19 21:28:31 -07:00
docs remediation of S205607 2020-07-17 17:19:47 -07:00
examples Remove py2 compatible future imports (#44735) 2020-09-16 12:55:57 -07:00
fakelowp Adding Peter's Swish Op ULP analysis. (#42573) 2020-08-07 19:13:01 -07:00
helpers remediation of S205607 2020-07-17 17:19:47 -07:00
ideep [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
layers [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
mint remediation of S205607 2020-07-17 17:19:47 -07:00
mkl [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
modeling remediation of S205607 2020-07-17 17:19:47 -07:00
models [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
onnx Replace black_list with block_list (#42088) 2020-08-20 14:34:02 -07:00
operator_test Enable fp16 for UniformFill (#44540) 2020-09-15 15:09:18 -07:00
predictor [DPER3] AOT integration 2020-09-04 18:37:22 -07:00
rnn Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
serialized_test [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
test add fake fp16 fusions to net transforms (#42927) 2020-08-14 13:30:27 -07:00
trt remediation of S205607 2020-07-17 17:19:47 -07:00
__init__.py Fixes caffe2 loading issues on Windows (#39513) 2020-06-23 20:11:24 -07:00
_import_c_extension.py [AMD] Remove num_gpu check for remote execution (#34318) 2020-03-06 09:53:57 -08:00
allcompare_test.py [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
attention.py
benchmark_generator.py
binarysize.py
brew_test.py
brew.py Testing for folded conv_bn_relu (#19298) 2019-04-16 19:04:06 -07:00
build.py
cached_reader.py Pass loop_over optional parameter for cached reader properly. (#21929) 2019-06-19 18:15:32 -07:00
caffe_translator_test.py Fix several ResourceWarning: unclosed file (#15746) 2019-01-09 15:36:53 -08:00
caffe_translator.py Fix several ResourceWarning: unclosed file (#15746) 2019-01-09 15:36:53 -08:00
checkpoint_test.py
checkpoint.py Fix typos, via a Levenshtein-type corrector (#31523) 2020-01-17 16:03:19 -08:00
CMakeLists.txt Fix CMakeLists.txt for Int8 python bindings (#15047) 2018-12-11 10:48:47 -08:00
cnn.py Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
compatibility.py
context_test.py
context.py
control_ops_grad_test.py Fix the weird bug in control_flow_op_test.py (#26931) 2019-09-26 20:44:03 -07:00
control_ops_grad.py DeviceScope support for CUDA and testing (#15357) 2019-01-30 18:42:12 -08:00
control_ops_util.py
control_test.py
control.py
convert_test.py
convert.py
convnet_benchmarks_test.py Skip convnets benchmark in rocm CI (#17331) 2019-02-20 21:12:24 -08:00
convnet_benchmarks.py
core_gradients_test.py [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
core_test.py Skip TestExtractPredictorNet if compiled without OpenCV (#42168) 2020-07-28 11:26:55 -07:00
core.py [HT] Clear the device placement tag for the auto gen sum so that we could break the component for FC sharing the same input (#42219) 2020-07-29 15:21:27 -07:00
crf_predict.py
crf_viterbi_test.py [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
crf.py Fix typos, via a Levenshtein-type corrector (#31523) 2020-01-17 16:03:19 -08:00
data_parallel_model_test.py [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
data_parallel_model.py [Caffe2] raise exceptions instead of str (#37744) 2020-05-05 13:34:33 -07:00
data_workers_test.py Disables test_atomic_ops and testInputOrder (#29145) 2019-11-05 16:53:53 -08:00
data_workers.py
dataio_test.py Skip DataIO tests relying on LevelDB if compiled without it (#42169) 2020-07-28 10:18:26 -07:00
dataio.py Rearrange stopping condition in CompositeReader (#20062) 2019-05-06 15:06:32 -07:00
dataset.py
db_file_reader.py Adding support for manifold files in DBReader (#37727) 2020-05-15 07:18:30 -07:00
db_test.py
device_checker.py
dlpack.h [TVM] Fix build and sync with caffe2/caffe2/python/dlpack.h (#40888) 2020-07-02 15:37:45 -07:00
dyndep.py Create lazy_dyndeps to avoid caffe2 import costs. (#41343) 2020-07-16 15:17:41 -07:00
embedding_generation_benchmark.py
experiment_util.py
extension_loader.py always restore dlopen flag in dyndep (#22958) 2019-07-17 10:26:25 -07:00
fakefp16_transform_lib.py add fake fp16 fusions to net transforms (#42927) 2020-08-14 13:30:27 -07:00
filler_test.py caffe2 - Expose tensor filler util to Python (#18886) 2019-04-08 11:54:10 -07:00
functional_test.py
functional.py Fix typos, via a Levenshtein-type corrector (#31523) 2020-01-17 16:03:19 -08:00
fused_8bit_rowwise_conversion_ops_test.py [caffe2] make fused rowwise quant/dequant op work for N-dim tensors (#33426) 2020-02-19 23:29:42 -08:00
gradient_check_test.py Unify gpu_support variable in python tests (#16748) 2019-02-07 00:29:51 -08:00
gradient_checker.py [caffe2] fix type and shape inference for common gradient ops (#35857) 2020-04-02 11:17:04 -07:00
gru_cell.py
hip_test_util.py Make CUDNN an alias of MIOPEN for HIP ops (#12278) 2018-10-24 17:07:31 -07:00
hsm_util.py
hypothesis_test_util.py [Caffe2] gcs_cuda_only is trivial if CUDA not available (#44578) 2020-09-16 12:24:08 -07:00
hypothesis_test.py [caffe2] adds Cancel to SafeDequeueBlobsOp and SafeEnqueueBlobsOp (#44495) 2020-09-16 18:17:34 -07:00
ideep_test_util.py
layer_model_helper.py [2/n][Compute Meta] support analysis for null flag features 2020-07-20 13:13:45 -07:00
layer_model_instantiator.py
layer_parameter_sharing_test.py Add validator for optimizers when parameters are shared 2019-04-17 21:10:38 -07:00
layer_test_util.py
layers_test.py [hypothesis] Deadline followup (#42842) 2020-08-11 15:33:23 -07:00
lazy_dyndep_test.py [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
lazy_dyndep.py Modify lazy_dyndep loading to trigger inside workspace. (#41687) 2020-07-22 15:36:43 -07:00
lazy.py Modify lazy_dyndep loading to trigger inside workspace. (#41687) 2020-07-22 15:36:43 -07:00
lengths_reducer_fused_8bit_rowwise_ops_test.py [caffe2] fix how np.clip is used in lengths_reducer_fused_{4,8}_rowwise_ops_test (#32086) 2020-01-14 22:53:28 -08:00
lengths_reducer_rowwise_8bit_ops_test.py
lstm_benchmark.py Fix typos (#30606) 2019-12-02 20:17:42 -08:00
memonger_test.py [hypothesis] Deadline followup (#42842) 2020-08-11 15:33:23 -07:00
memonger.py [pyfi] override TP2 networkx -> PyFI networkx (#37764) 2020-05-11 13:20:00 -07:00
mkl_test_util.py
model_device_test.py Unify gpu_support variable in python tests (#16748) 2019-02-07 00:29:51 -08:00
model_helper_test.py
model_helper.py Fix TensorProtosDBInput AttributeError (#32274) 2020-01-29 12:05:43 -08:00
modifier_context.py Fix typos (#30606) 2019-12-02 20:17:42 -08:00
mpi_python.cc Replace c10::guts::stuff with std::stuff (#30915) 2019-12-16 13:57:19 -08:00
muji_test.py Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
muji.py Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
net_builder_test.py
net_builder.py
net_drawer.py Fix typos, via a Levenshtein-type corrector (#31523) 2020-01-17 16:03:19 -08:00
net_printer_test.py
net_printer.py Fix spelling errors (#21665) 2019-06-13 15:21:55 -07:00
nomnigraph_test.py nomnigraph - support subgraph visualization (#13795) 2018-11-16 08:19:20 -08:00
nomnigraph_transformations_test.py Add transpose network pass (#13437) 2018-11-01 14:27:07 -07:00
nomnigraph_transformations.py Add transpose network pass (#13437) 2018-11-01 14:27:07 -07:00
nomnigraph.py [caffe2/nomnigraph] handle when PATH env is not defined (#39373) 2020-06-10 17:09:59 -07:00
normalizer_context.py Fix typos (#30606) 2019-12-02 20:17:42 -08:00
normalizer_test.py
normalizer.py Scale init for batch-norm and layer-norm (#31983) 2020-01-10 11:55:56 -08:00
numa_benchmark.py Revert D13205604: Move numa.{h, cc} to c10/util 2018-12-07 10:01:25 -08:00
numa_test.py Move numa.{h, cc} to c10/util (#15024) 2018-12-12 12:21:10 -08:00
observer_test.py [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
operator_fp_exceptions_test.py Caffe2 - Add flag to fails if float point exceptions is detected in operator runs (#18040) 2019-03-16 12:28:05 -07:00
optimizer_context.py Fix typos (#30606) 2019-12-02 20:17:42 -08:00
optimizer_test_util.py Fix typos (#30606) 2019-12-02 20:17:42 -08:00
optimizer_test.py [pruning] add rowwise counter to sparse adagrad 2020-06-30 14:40:02 -07:00
optimizer.py Expose the interface of nesterov of SGD Optimizer from caffe2 to dper 2020-09-09 19:37:00 -07:00
parallel_workers_test.py ParallelWorkersTest.testParallelWorkersInitFun is flaky (#29045) 2019-11-01 13:59:02 -07:00
parallel_workers.py get rid of deprecated thread.isAlive() to use py2.6 modern form is_alive() 2019-10-22 15:37:31 -07:00
parallelize_bmuf_distributed_test.py [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
pipeline_test.py
pipeline.py Fix typos, via a Levenshtein-type corrector (#31523) 2020-01-17 16:03:19 -08:00
predictor_constants.py
pybind_state_dlpack.cc Upgrade DLPack 2018-11-12 15:59:46 -08:00
pybind_state_dlpack.h Remove PythonOp non-CPU path and PytorchOp (#15417) 2019-01-02 16:36:37 -08:00
pybind_state_gpu.cc add simple memory analyzer and log warning if GPU underutilized (#21024) 2019-05-28 19:58:54 -07:00
pybind_state_hip.cc Make caffe2/fb folder compatible with AMD (#29131) 2019-11-04 16:40:29 -08:00
pybind_state_ideep.cc Upgrade MKL-DNN to DNNL v1.2 (#32422) 2020-03-26 22:07:59 -07:00
pybind_state_int8.cc Renaming meta() to dtype() - 2/2 (#13334) 2018-10-30 18:24:30 -07:00
pybind_state_nomni.cc nomnigraph - support subgraph visualization (#13795) 2018-11-16 08:19:20 -08:00
pybind_state_registry.cc
pybind_state_registry.h
pybind_state.cc [caffe2] exposes Net cancellation through pybind state (#44043) 2020-09-09 18:13:13 -07:00
pybind_state.h Add API for onnxifi with AOT Glow ONNX (#44021) 2020-09-03 22:46:20 -07:00
python_op_test.py [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
queue_util.py
record_queue.py
recurrent.py
regularizer_context.py Fix typos (#30606) 2019-12-02 20:17:42 -08:00
regularizer_test.py [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
regularizer.py Adding LpNorm regularization for sparse features in DPER3 (#38582) 2020-06-09 12:34:50 -07:00
rnn_cell.py [caffe2] Remove python2 from operator_test (#33977) 2020-03-02 08:55:53 -08:00
schema_test.py Pass LRU hash output evicted_values to SparseLookup (#21389) 2019-07-02 11:27:37 -07:00
schema.py Fix typos, via a Levenshtein-type corrector (#31523) 2020-01-17 16:03:19 -08:00
scope_test.py Add EmptyNameScope to allow you jump out from current scope. (#14631) 2018-12-12 01:39:50 -08:00
scope.py Add EmptyNameScope to allow you jump out from current scope. (#14631) 2018-12-12 01:39:50 -08:00
session_test.py
session.py Fix typos (#30606) 2019-12-02 20:17:42 -08:00
sparse_to_dense_mask_test.py Increase static tolerance for negative feature ids 2019-05-20 19:09:22 -07:00
sparse_to_dense_test.py Shape inference for SparseToDense in ExpertCombiner 2020-07-15 08:04:48 -07:00
task_test.py caffe2/python/task: added __repr__ methods to all task definitions (#15250) 2018-12-17 16:02:16 -08:00
task.py Fix typos (#30606) 2019-12-02 20:17:42 -08:00
test_util.py caffe2 - support flaky operator tests for caffe2 build (#18155) 2019-03-25 16:58:34 -07:00
text_file_reader.py Create Node2Vec ModuleKeeper 2019-04-01 10:36:23 -07:00
timeout_guard.py
toy_regression_test.py Enable junk fill for the default CPU allocator (#13377) 2018-11-08 00:02:37 -08:00
transformations_test.py Remove sinkMaxPool transformation (#17694) 2019-03-12 20:10:46 -07:00
transformations.py support pre-convert filter format for mkldnn training mode and change 'OptimizeForIdeep' to 'OptimizeForMkldnn' (#15171) 2019-03-29 19:00:48 -07:00
tt_core_test.py
tt_core.py
utils_test.py [C2] Introduce extra_info force CPU tags for auto-generated iteration counter blobs (#32607) 2020-02-05 23:49:27 -08:00
utils.py [2/n][Compute Meta] support analysis for null flag features 2020-07-20 13:13:45 -07:00
visualize.py Fix typos, via a Levenshtein-type corrector (#31523) 2020-01-17 16:03:19 -08:00
workspace_test.py [PyFI] Update hypothesis and switch from tp2 (#41645) 2020-08-08 12:13:04 -07:00
workspace.py Add python inferface to create OfflineTensor (#42516) 2020-08-04 01:31:34 -07:00