mirror of
https://github.com/zebrajr/pytorch.git
synced 2025-12-07 12:21:27 +01:00
e941a462a3
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/43883 Check the result of GCC coverage in OSS is reasonable and ready to ship. The amount of executable lines are not the same between `gcc` and `clang` because of the following reasons: * Lines following are counted in `clang` but not in `gcc`: 1. empty line or line with only “{” or “}” 3. some comments are counted in clang but not in gcc 5. `#define ...` -- not supported by gcc according to official documentation * Besides, a statement that explains to more than one line will be counted as only one executable line in gcc, but several lines in clang ## Advantage of `gcc` coverage 1. Much faster - code coverage tool runtime is onle **4 min** (*ammazzzing!!*) by `gcc`, compared to **3 hours!!** by `clang`, to analyze all the tests' artifacts 2. Use less disk - `Clang`'s artifacts will take as large as 170G, but `GCC` is 980M Besides, also update `README.md`. Test Plan: Compare the result in OSS `clang` and OSS `gcc` with the same command: ``` python oss_coverage.py --run-only atest test_nn.py --interested-folder=aten ``` ---- ## GCC **Summary** > time: 0:15:45 summary percentage: 44.85% **Report and Log** [File Coverage Report](P140825162) [Line Coverage Report](P140825196) [Log](P140825385) ------ ## CLANG **Summary** > time: 0:21:35 summary percentage: 44.08% **Report and Log** [File Coverage Report](P140825845) [Line Coverage Report](P140825923) [Log](P140825950) ---------- # Run all tests ``` # run all tests and get coverage over Pytorch python oss_coverage.py ``` **Summary** > time: 1:27:20. ( time to run tests: 1:23:33) summary percentage: 56.62% **Report and Log** [File Coverage Report](P140837175) [Log](P140837121) Reviewed By: malfet Differential Revision: D23416772 fbshipit-source-id: a6810fa4d8199690f10bd0a4f58a42ab2a22182b
68 lines
2.8 KiB
Markdown
68 lines
2.8 KiB
Markdown
# Code Coverage Tool for Pytorch
|
||
|
||
## Overview
|
||
|
||
This tool is designed for calculating code coverage for Pytorch project.
|
||
It’s an integrated tool. You can use this tool to run and generate both file-level and line-level report for C++ and Python tests. It will also be the tool we use in *CircleCI* to generate report for each master commit.
|
||
|
||
### Simple
|
||
* *Simple command to run:*
|
||
* `python oss_coverage.py `
|
||
* *Argument `--clean` will do all the messy clean up things for you*
|
||
|
||
### But Powerful
|
||
|
||
* *Choose your own interested folder*:
|
||
* Default folder will be good enough in most times
|
||
* Flexible: you can specify one or more folder(s) that you are interested in
|
||
* *Run only the test you want:*
|
||
* By default it will run all the c++ and python tests
|
||
* Flexible: you can specify one or more test(s) that you want to run
|
||
* *Final report:*
|
||
* File-Level: The coverage percentage for each file you are interested in
|
||
* Line-Level: The coverage details for each line in each file you are interested in
|
||
* *More complex but flexible options:*
|
||
* Use different stages like *--run, --export, --summary* to achieve more flexible functionality
|
||
|
||
## How to use
|
||
|
||
This part will introduce about the arguments you can use when run this tool. The arguments are powerful, giving you full flexibility to do different work.
|
||
We have two different compilers, `gcc` and `clang`, and this tool supports both. But it is recommended to use `gcc` because it's much faster and use less disk place. The examples will also be divided to two parts, for `gcc` and `clang`.
|
||
|
||
|
||
## Examples
|
||
First step is to set some experimental value if needed:
|
||
```bash
|
||
# pytorch folder, by default all the c++ binaries are in build/bin/
|
||
export PYTORCH_FOLDER=...
|
||
# set compiler type
|
||
export COMPILER_TYPE="GCC" or export COMPILER_TYPE="CLANG"
|
||
# make sure llvm-cov is available, by default it is /usr/local/opt/llvm/bin
|
||
export LLVM_TOOL_PATH=...
|
||
```
|
||
|
||
then command will run all the tests in `build/bin/` and `test/` folder
|
||
```bash
|
||
python oss_coverage.py
|
||
```
|
||
Most times you don't want collect coverage for the entire Pytorch folder, use --interested-folder to report coverage only over the folder you want:
|
||
```bash
|
||
python oss_coverage.py --interested-folder=aten
|
||
```
|
||
Then, still in most cases, if you only run one or several test(s):
|
||
```bash
|
||
python oss_coverage.py --run-only=atest
|
||
python oss_coverage.py --run-only atest basic test_nn.py
|
||
```
|
||
|
||
### For more complex arguments and functionality
|
||
*To Be Done*
|
||
|
||
## Reference
|
||
|
||
For `gcc`
|
||
* See about how to invoke `gcov`, read [Invoking gcov](https://gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html#Invoking-Gcov) will be helpful
|
||
|
||
For `clang`
|
||
* If you are not familiar with the procedure of generating code coverage report by using `clang`, read [Source-based Code Coverage](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html) will be helpful.
|