Commit Graph

2 Commits

Author SHA1 Message Date
Kevin Wilfong
5dba88b40b Caffe2 [easy]: Better exception logging in parallel_workers/data_workers
Summary: Instead of printing the exception using print() use traceback.print_exc()  This way you get a stack trace

Reviewed By: jay-mahadeokar

Differential Revision: D5604642

fbshipit-source-id: f8cb67e554305cd2fbed384a4a2040fa2b16e7c0
2017-08-10 15:27:19 -07:00
Kevin Wilfong
d072701547 Caffe2: Refactor the core logic from data_workers.py into parallel_workers.py
Summary:
data_workers.py provides a really nice, easy way to run background threads for data input.  Unfortunately, it's restrictive, the output of the fetcher function has to be a numpy array.

I pulled out that core nice thread management into parallel_workers, and updated the classes data_workers to extend those classes.  The main change was refactoring out most of the queue handling logic into QueueManager.

This way parallel_workers can be used to manage background threads without having to use the queue for output.

Reviewed By: akyrola

Differential Revision: D5538626

fbshipit-source-id: f382cc43f800ff90840582a378dc9b86ac05b613
2017-08-07 10:14:08 -07:00