pytorch/scripts
Ivan Kobzarev 92b9de1428 Test application for profiling, CMake params for debug symbols (#28406)
Summary:
Reason:
To have one-step build for test android application based on the current code state that is ready for profiling with simpleperf, systrace etc. to profile performance inside the application.

## Parameters to control debug symbols stripping
Introducing  /CMakeLists parameter `ANDROID_DEBUG_SYMBOLS` to be able not to strip symbols for pytorch (not add linker flag `-s`)
which is checked in `scripts/build_android.sh`

On gradle side stripping happens by default, and to prevent it we have to specify
```
android {
  packagingOptions {
       doNotStrip "**/*.so"
  }
}
```
which is now controlled by new gradle property `nativeLibsDoNotStrip `

## Test_App
`android/test_app` - android app with one MainActivity that does inference in cycle

`android/build_test_app.sh` - script to build libtorch with debug symbols for specified android abis and adds `NDK_DEBUG=1` and `-PnativeLibsDoNotStrip=true` to keep all debug symbols for profiling.
Script assembles all debug flavors:
```
└─ $ find . -type f -name *apk
./test_app/app/build/outputs/apk/mobilenetQuant/debug/test_app-mobilenetQuant-debug.apk
./test_app/app/build/outputs/apk/resnet/debug/test_app-resnet-debug.apk
```

## Different build configurations

Module for inference can be set in `android/test_app/app/build.gradle` as a BuildConfig parameters:
```
    productFlavors {
        mobilenetQuant {
            dimension "model"
            applicationIdSuffix ".mobilenetQuant"
            buildConfigField ("String", "MODULE_ASSET_NAME", buildConfigProps('MODULE_ASSET_NAME_MOBILENET_QUANT'))
            addManifestPlaceholders([APP_NAME: "PyMobileNetQuant"])
            buildConfigField ("String", "LOGCAT_TAG", "\"pytorch-mobilenet\"")
        }
        resnet {
            dimension "model"
            applicationIdSuffix ".resnet"
            buildConfigField ("String", "MODULE_ASSET_NAME", buildConfigProps('MODULE_ASSET_NAME_RESNET18'))
            addManifestPlaceholders([APP_NAME: "PyResnet"])
            buildConfigField ("String", "LOGCAT_TAG", "\"pytorch-resnet\"")
        }
```

In that case we can setup several apps on the same device for comparison, to separate packages `applicationIdSuffix`: 'org.pytorch.testapp.mobilenetQuant' and different application names and logcat tags as `manifestPlaceholder` and another BuildConfig parameter:
```
─ $ adb shell pm list packages | grep pytorch
package:org.pytorch.testapp.mobilenetQuant
package:org.pytorch.testapp.resnet
```

In future we can add another BuildConfig params e.g. single/multi threads and other configuration for profiling.

At the moment 2 flavors - for resnet18 and for mobilenetQuantized
which can be installed on connected device:

```
cd android
```
```
gradle test_app:installMobilenetQuantDebug
```
```
gradle test_app:installResnetDebug
```

## Testing:
```
cd android
sh build_test_app.sh
adb install -r test_app/app/build/outputs/apk/mobilenetQuant/debug/test_app-mobilenetQuant-debug.apk
```

```
cd $ANDROID_NDK
python simpleperf/run_simpleperf_on_device.py record --app org.pytorch.testapp.mobilenetQuant -g --duration 10 -o /data/local/tmp/perf.data
adb pull /data/local/tmp/perf.data
python simpleperf/report_html.py
```

Simpleperf report has all symbols:
![Screenshot 2019-10-22 11 06 21](https://user-images.githubusercontent.com/6638825/67315740-0bc50100-f4bc-11e9-8f9e-2499be13d63e.png)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/28406

Differential Revision: D18386622

Pulled By: IvanKobzarev

fbshipit-source-id: 3a751192bbc4bc3c6d7f126b0b55086b4d586e7a
2019-11-08 14:19:04 -08:00
..
appveyor move flags to c10 (#12144) 2018-10-04 02:09:56 -07:00
fbcode-dev-setup remove "build_deps" arg from setup.py command in (#26113) 2019-09-12 15:34:21 -07:00
model_zoo Bump up opset version to 7 in Caffe2 ONNX exporter (#8854) 2018-06-28 07:39:02 -07:00
onnx fix the CI job for onnx (#22946) 2019-07-17 09:50:06 -07:00
add_apache_header.sh Re-license to Apache 2017-09-28 16:22:00 -07:00
apache_header.txt Re-license to Apache 2017-09-28 16:22:00 -07:00
apache_python.txt Add script for removing Apache header. 2018-03-27 13:10:18 -07:00
build_android.sh Test application for profiling, CMake params for debug symbols (#28406) 2019-11-08 14:19:04 -08:00
build_host_protoc.sh Port all PyTorch and Caffe2 jobs to CircleCI (#11264) 2018-09-05 16:28:11 -07:00
build_ios.sh Only set CCACHE_WRAPPER_PATH in the build scripts if it is not already passed in. 2019-11-01 18:39:12 -07:00
build_local.sh Only set CCACHE_WRAPPER_PATH in the build scripts if it is not already passed in. 2019-11-01 18:39:12 -07:00
build_mobile.sh add script to build mobile library with host toolchain (#26440) 2019-09-18 19:34:09 -07:00
build_pytorch_android.sh Script for full android build to aars; script to run android tests (#26833) 2019-09-30 14:25:00 -07:00
build_raspbian.sh Fix target name. 2019-02-22 17:27:16 -08:00
build_tegra_x1.sh Allow specifying cmake build directory in the build scripts 2017-11-20 20:32:30 -08:00
build_tizen.sh Allow specifying cmake build directory in the build scripts 2017-11-20 20:32:30 -08:00
build_windows.bat Turn off warnings on Windows CI. (#24331) 2019-08-30 07:11:07 -07:00
diagnose_protobuf.py Fix several DeprecationWarning: invalid escape sequence (#15733) 2019-01-05 08:53:35 -08:00
get_python_cmake_flags.py [caffe2][build] Update python cmake flag print script (#7306) 2018-05-08 00:34:42 -07:00
proto.ps1 Sync fbcode/caffe2 and xplat/caffe2 (1) (#19218) 2019-04-13 21:45:52 -07:00
read_conda_versions.sh Adding gcc4 conda builds (#2283) 2018-03-17 17:26:37 -07:00
README.md Enhance build_ios.sh to be consistent with build_android.sh (#18564) 2019-03-28 21:37:55 -07:00
remove_apache_header.sh Add script for removing Apache header. 2018-03-27 13:10:18 -07:00
run_mobilelab.py Sync fbcode/caffe2 and xplat/caffe2 (1) (#19218) 2019-04-13 21:45:52 -07:00
temp.sh Add script for removing Apache header. 2018-03-27 13:10:18 -07:00
xcode_build.rb add benchmark code to iOS TestApp (#28405) 2019-10-22 12:52:30 -07:00

This directory contains the useful tools.

build_android.sh

This script is to build PyTorch/Caffe2 library for Android. Take the following steps to start the build:

  • set ANDROID_NDK to the location of ndk
export ANDROID_NDK=YOUR_NDK_PATH
  • run build_android.sh
#in your PyTorch root directory
bash scripts/build_android.sh

If succeeded, the libraries and headers would be generated to build_android/install directory. You can then copy these files from build_android/install to your Android project for further usage.

You can also override the cmake flags via command line, e.g., following command will also compile the executable binary files:

bash scripts/build_android.sh -DBUILD_BINARY=ON

build_ios.sh

This script is to build PyTorch/Caffe2 library for iOS, and can only be performed on macOS. Take the following steps to start the build:

  • Install Xcode from App Store, and configure "Command Line Tools" properly on Xcode.
  • Install the dependencies:
brew install cmake automake libtool
  • run build_ios.sh
#in your PyTorch root directory
bash scripts/build_ios.sh

If succeeded, the libraries and headers would be generated to build_ios/install directory. You can then copy these files to your Xcode project for further usage.