mirror of
https://github.com/zebrajr/pytorch.git
synced 2025-12-06 12:20:52 +01:00
d6d726f781
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/55462 handles and symbolicate exception callstack thrown from backend. Objective of this diff is to achieve improve error reporting when exceptions are raised from lowered backend. We would effectively like to get the same model level stack trace that you would get without having lowered some module to backend. For example: ``` class AA(nn.Module): def forward(self, x, y): return x + y class A(nn.Module): def __init__(...): self.AA0 = AA() def forward(self, x, y): return self.AA0.forward(x, y) + 3 class B(nn.Module): def forward(self, x): return x + 2 class C(nn.Module): def __init__(...): self.A0 = A() self.B0 = B() def forward(self, x, y): return self.A0.forward(x, y) + self.B0.forward(x) ``` If the we then do C().forward(torch.rand((2,3)), torch.rand(14,2))) we will likely see error stack like: ``` C++ exception with description "The following operation failed in the TorchScript interpreter. Traceback of TorchScript (most recent call last): File "<string>", line 3, in forward def forward(self, x, y): return self.A0.forward(x, y) + self.B0.forward(x) ~~~~~~~~~~~~~~~ <--- HERE File "<string>", line 3, in forward def forward(self, x, y): return self.AA0.forward(x, y) + 3 ~~~~~~~~~~~~~~~~ <--- HERE File "<string>", line 3, in forward def forward(self, x, y): return x + y ~~~~~ <--- HERE ``` We would like to see the same error stack if we lowered C.A0 to some backend. With this diff we get something like: ``` Module hierarchy:top(C).A0(backend_with_compiler_demoLoweredModule).AA0(AA) Traceback of TorchScript (most recent call last): File "<string>", line 3, in FunctionName_UNKNOWN def forward(self, x, y): return self.A0.forward(x, y) + self.B0.forward(x) ~~~~~~~~~~~~~~~ <--- HERE File "<string>", line 5, in FunctionName_UNKNOWN typed_inputs: List[Any] = [x, y, ] if self.__backend.is_available() : _0, = self.__backend.execute(self.__handles["forward"], typed_inputs) ~~~~~~~~~~~~~~~~~~~~~~ <--- HERE assert isinstance(_0, Tensor) return _0 File "<string>", line 3, in FunctionName_UNKNOWN def forward(self, x, y): return self.AA0.forward(x, y) + 3 ~~~~~~~~~~~~~~~~ <--- HERE File "<string>", line 3, in FunctionName_UNKNOWN def forward(self, x, y): return x + y ~~~~~ <--- HERE ``` This is achieved in 3 parts: Part 1: A. BackendDebugInfoRecorder: During backend lowering, in `to_backend`, before calling the preprocess function corresponding to the backend. This will facilitate recording of debug info (such as source range + inlined callstack) for the lowered module. B. Instantiate WithBackendDebugInfoRecorder with BackendDebugInfoRecorder. This initializes thread local pointer to BackendDebugInfoRecorder. C. generate_debug_handles: In preprocess function, the backend will call generate_debug_handles for each method being lowered separately. generate_debug_handles takes `Graph` of the method being lowered and returns a map of Node*-to-debug_handles. Backend is responsible for storing debug handles appropriately so as to raise exception (and later profiling) using debug handles when the exception being raised corresponds to particular Node that was lowered. Inside generate_debug_handles, we will query the current BackendDebugHandleInfoRecorder, that is issuing debug handles. This debug handle manager will issue debug handles as well as record debug_handles-to-<source range, inlined callstack> map. D. Back in `to_backend`, once the preprocess function is has finished lowering the module, we will call `stopRecord` on BackendDebugInfoRecorder. This will return the debug info map. This debug info is then stored inside the lowered module. Part 2: Serialization: During serialization for bytecode (lite interpreter), we will do two things: 1. Extract all the source ranges that are contained inside debug_handles-to-<source range, inlined callstack> map for lowered module. This will be source range corresponding to debug handles, including what is there is inlined callstack. Since we replaced original module with lowered module, we wont be serializing code for the original module and thus no source range. That is why the source range will have to be stored separately. We will lump all the source ranges for all the lowered modules in one single debug_pkl file. 2. Then we will serialize debug_handles-to-<source range, inlined callstack> map. Now during deserialization we will be able to reconstruct debug_handles-to-<source range, inlined callstack> map. Given all debug_handles are unique we would not need any module information. Test Plan: Tests are added in test_backend.cpp Tests are added in test_backend.cpp Imported from OSS Differential Revision: D27621330 D27621330 Reviewed By: raziel Pulled By: kimishpatel fbshipit-source-id: 0650ec68cda0df0a945864658cab226a97ba1890
101 lines
3.2 KiB
C++
101 lines
3.2 KiB
C++
#pragma once
|
|
|
|
#include <torch/csrc/jit/ir/irparser.h>
|
|
#include <torch/csrc/jit/runtime/autodiff.h>
|
|
#include <torch/csrc/jit/runtime/interpreter.h>
|
|
#include <torch/csrc/jit/testing/file_check.h>
|
|
|
|
namespace {
|
|
static inline void trim(std::string& s) {
|
|
s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](unsigned char ch) {
|
|
return !std::isspace(ch);
|
|
}));
|
|
s.erase(
|
|
std::find_if(
|
|
s.rbegin(),
|
|
s.rend(),
|
|
[](unsigned char ch) { return !std::isspace(ch); })
|
|
.base(),
|
|
s.end());
|
|
for (int64_t i = 0; i < s.size(); ++i) {
|
|
if (s[i] == '\n') {
|
|
s.erase(i, 1);
|
|
i--;
|
|
}
|
|
}
|
|
for (int64_t i = 0; i < s.size(); ++i) {
|
|
if (s[i] == ' ') {
|
|
for (int64_t j = i + 1; j < s.size(); j++) {
|
|
if (s[j] == ' ') {
|
|
s.erase(j, 1);
|
|
j--;
|
|
} else {
|
|
break;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
} // namespace
|
|
|
|
#define ASSERT_THROWS_WITH_MESSAGE(statement, substring) \
|
|
try { \
|
|
(void)statement; \
|
|
FAIL(); \
|
|
} catch (const std::exception& e) { \
|
|
std::string substring_s(substring); \
|
|
trim(substring_s); \
|
|
auto exception_string = std::string(e.what()); \
|
|
trim(exception_string); \
|
|
ASSERT_NE(exception_string.find(substring_s), std::string::npos); \
|
|
}
|
|
|
|
namespace torch {
|
|
namespace jit {
|
|
|
|
using tensor_list = std::vector<at::Tensor>;
|
|
using namespace torch::autograd;
|
|
|
|
// work around the fact that variable_tensor_list doesn't duplicate all
|
|
// of std::vector's constructors.
|
|
// most constructors are never used in the implementation, just in our tests.
|
|
Stack createStack(std::vector<at::Tensor>&& list);
|
|
|
|
void assertAllClose(const tensor_list& a, const tensor_list& b);
|
|
|
|
std::vector<at::Tensor> run(
|
|
InterpreterState& interp,
|
|
const std::vector<at::Tensor>& inputs);
|
|
|
|
std::pair<tensor_list, tensor_list> runGradient(
|
|
Gradient& grad_spec,
|
|
tensor_list& tensors_in,
|
|
tensor_list& tensor_grads_in);
|
|
|
|
std::shared_ptr<Graph> build_lstm();
|
|
std::shared_ptr<Graph> build_mobile_export_analysis_graph();
|
|
std::shared_ptr<Graph> build_mobile_export_analysis_graph_with_vararg();
|
|
std::shared_ptr<Graph> build_mobile_export_analysis_graph_nested();
|
|
std::shared_ptr<Graph> build_mobile_export_analysis_graph_non_const();
|
|
|
|
at::Tensor t_use(at::Tensor x);
|
|
at::Tensor t_def(at::Tensor x);
|
|
|
|
// given the difference of output vs expected tensor, check whether the
|
|
// difference is within a relative tolerance range. This is a standard way of
|
|
// matching tensor values up to certain precision
|
|
bool checkRtol(const at::Tensor& diff, const std::vector<at::Tensor> inputs);
|
|
bool almostEqual(const at::Tensor& a, const at::Tensor& b);
|
|
|
|
bool exactlyEqual(const at::Tensor& a, const at::Tensor& b);
|
|
|
|
std::pair<at::Tensor, at::Tensor> lstm(
|
|
at::Tensor input,
|
|
at::Tensor hx,
|
|
at::Tensor cx,
|
|
at::Tensor w_ih,
|
|
at::Tensor w_hh);
|
|
|
|
} // namespace jit
|
|
} // namespace torch
|