This updated script will count the number of model that contains a certain op, and calculate the coverage based on that.
Example output:
Generated 693 ops
Covered 344/393 (87.53%) production ops
Covered 6053/7146 (84.7%) occurrence
pytorch ver 1.12.0.dev20220401
Also updated some test generation scripts
Pull Request resolved: https://github.com/pytorch/pytorch/pull/75155
Approved by: https://github.com/kit1980
This pr
moved some model generation scripts to a central place (mobile net v2, android test model)
updated scripts so these models can run on Android (Java doesn't support certain scalar types as return value)
updated model generation script to take arguments (generate models for android/ios, checked-in model or on-the-fly model)
add Android instrumentation tests for these new models
After this change, the Android instrumentation test will run 35 models which covered 91% of production ops. The coverage information can be found in this file: https://github.com/pytorch/pytorch/blob/master/test/mobile/model_test/coverage.yaml
Note that these models are checked-in for back compatibility check (to ensure they can run with newer pytorch versions).
The script generates models for mobile test. For each model we have a "checked-in" version
and an "on-the-fly" version. The "on-the-fly" version will be generated during test, and
should not be committed to the repo. The "checked-in" version is used for back compatibility check.
Note that Android only support checked-in model right now. iOS can test both (in another pr).
use 'gen_test_model.py android-test' to generate on the fly models for android
use 'gen_test_model.py ios-test' to generate on the fly models for ios
use 'python gen_test_model.py android' to generate checked-in models for android
use 'python gen_test_model.py ios' to generate on-the-fly models for ios
use 'gen_test_model.py <model_name_no_suffix>' to update the given checked-in model
Pull Request resolved: https://github.com/pytorch/pytorch/pull/74793
Approved by: https://github.com/kit1980
- update model generation script to cover more ops (including quant ops)
- add more test cases for iOS simulator test
- Need to update GA config to run the new model test on iOS
```
Generated 703 ops and covered 356/393 (90.59%) production ops.
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/74599
Approved by: https://github.com/kit1980
