mirror of
https://github.com/zebrajr/pytorch.git
synced 2025-12-07 00:21:07 +01:00
ab39a55331
Summary: This diff is to support python user defined function over rpc for https://github.com/pytorch/pytorch/issues/23110, work flow is like this: 1. pickle python udf 2. pass pickle to C++ 3. C++ pass over rpc from client to server 4. server call runPythonUDF() python function to unpickle and run python udf and pickle the udf result using python embedder 6. pass back serialized result from server to client 7. client call loadPythonUDFResult() python function to unpickle result 7. return it to python right now, put rpc_sync_builtin() and rpc_async_builtin() as temporary interfaces for builtin operator remote calls, they accept qualified name string, this interface can execute builtin operators in C++ land. rpc_sync() and rpc_async() accept python callables only right now, it could be user define python functions or builtin operator python functions, the python functions will be executed in python land. once we can resolve builtin operator python callables to qualified name string, we can merge rpc_sync_builtin() into rpc_sync() then Pull Request resolved: https://github.com/pytorch/pytorch/pull/23569 Test Plan: unit tests Differential Revision: D16390764 Pulled By: zhaojuanmao fbshipit-source-id: 2cf2c22a979646830b5581bd75eabf8b3cca564c |
||
|---|---|---|
| .. | ||
| amd_build | ||
| autograd | ||
| cwrap | ||
| docker | ||
| jit | ||
| nnwrap | ||
| pyi | ||
| setup_helpers | ||
| shared | ||
| __init__.py | ||
| aten_mirror.sh | ||
| build_libtorch.py | ||
| build_pytorch_libs.py | ||
| build_variables.py | ||
| clang_format.py | ||
| clang_tidy.py | ||
| download_mnist.py | ||
| flake8_hook.py | ||
| generated_dirs.txt | ||
| git_add_generated_dirs.sh | ||
| git_reset_generated_dirs.sh | ||
| git-pre-commit | ||
| pytorch.version | ||
| README.md | ||
| run-clang-tidy-in-ci.sh | ||
This folder contains a number of scripts which are used as
part of the PyTorch build process. This directory also doubles
as a Python module hierarchy (thus the __init__.py).
Overview
Modern infrastructure:
- autograd - Code generation for autograd. This includes definitions of all our derivatives.
- jit - Code generation for JIT
- shared - Generic infrastructure that scripts in
tools may find useful.
- module_loader.py - Makes it easier to import arbitrary Python files in a script, without having to add them to the PYTHONPATH first.
Legacy infrastructure (we should kill this):
- nnwrap - Generates the THNN/THCUNN wrappers which make legacy functionality available. (TODO: What exactly does this implement?)
- cwrap - Implementation of legacy code generation for THNN/THCUNN. This is used by nnwrap.
Build system pieces:
- setup_helpers - Helper code for searching for third-party dependencies on the user system.
- build_pytorch_libs.sh - Script that builds all of the constituent libraries of PyTorch, but not the PyTorch Python extension itself. We are working on eliminating this script in favor of a unified cmake build.
- build_pytorch_libs.bat - Same as above, but for Windows.
- build_libtorch.py - Script for building libtorch, a standalone C++ library without Python support. This build script is tested in CI.
Developer tools which you might find useful:
- clang_tidy.py - Script for running clang-tidy on lines of your script which you changed.
- git_add_generated_dirs.sh and git_reset_generated_dirs.sh - Use this to force add generated files to your Git index, so that you can conveniently run diffs on them when working on code-generation. (See also generated_dirs.txt which specifies the list of directories with generated files.)
Important if you want to run on AMD GPU:
- amd_build - HIPify scripts, for transpiling CUDA
into AMD HIP. Right now, PyTorch and Caffe2 share logic for how to
do this transpilation, but have separate entry-points for transpiling
either PyTorch or Caffe2 code.
- build_amd.py - Top-level entry point for HIPifying our codebase.
Tools which are only situationally useful:
- aten_mirror.sh - Mirroring script responsible for keeping https://github.com/zdevito/ATen up-to-date.
- docker - Dockerfile for running (but not developing) PyTorch, using the official conda binary distribution. Context: https://github.com/pytorch/pytorch/issues/1619
- download_mnist.py - Download the MNIST dataset; this is necessary if you want to run the C++ API tests.
- run-clang-tidy-in-ci.sh - Responsible for checking that C++ code is clang-tidy clean in CI on Travis