README Instructions of the coverage tool should be fixed.
1. Some CMAKE options are not consistent with 'pytorch/CmakeLists.txt'.
- 'CODE_COVERAGE' should be 'USE_CPP_CODE_COVERAGE'.
- 'CMAKE_BUILD_CONFIG' should be 'CMAKE_BUILD_TYPE'.
2. Some arguments of 'oss_coverage.py' are incorrect.
- Both '--interested-only' and '--interested-folder' doesn't work. I guess both of them were meant to be '--interest-only'
Pull Request resolved: https://github.com/pytorch/pytorch/pull/75091
Approved by: https://github.com/ezyang
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/44568
By `lcov`, we can generate beautiful html. It's better than current file report and line report. Therefore in oss gcc, remove `export` code and `file/line level report` code, only use the html report.
But in clang, since such tool is not available, we will still use file-report and line-report generated by ourself.
Test Plan:
Test in docker ubuntu machine.
## Mesurement
1. After running `atest`, it takes about 15 mins to collect code coverage and genrate the report.
```
# gcc code coverage
python oss_coverage.py --run-only=atest
```
## Presentation
**The html result looks like:**
*Top Level:*
{F328330856}
*File Level:*
{F328336709}
Reviewed By: malfet
Differential Revision: D23550784
fbshipit-source-id: 1fff050e7f7d1cc8e86a6a200fd8db04b47f5f3e
Summary: Some tests like `test_dataloader.py` are not able to run under `clang` in oss, because it generates too large intermediate files (~40G) that can't be merged by `llvm`. Skip them when user doesn't specify the `--run-only` option
Test Plan: Test locally. But still, not recomend user to run `clang` coverage in default mode, because it takes too much space.
Reviewed By: malfet
Differential Revision: D23549829
fbshipit-source-id: 0737e6e9dcbe3f38de00580ee6007906e743e52f
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/43890
1. auto-detect `CXX` default compiler type in oss, and `clang` as default compiler type in fbcode (because auto-detecting will say `gcc` is the default compiler on devserver).
2. change `compiler type` from str `"CLANG" "GCC"` to enum type
3. rename function `get_cov_type` to `detect_compiler_type`
4. auto-set the default pytorch folder for users in oss
Test Plan:
on devserver:
```
buck run :coverage //caffe2/c10:
```
on oss:
```
python oss_coverage.py --run-only=atest
```
Reviewed By: malfet
Differential Revision: D23420034
fbshipit-source-id: c0ea88188578bb1343a286f2090eb8a74cdf3982
Summary: Update `README.md` for oss to explain the usage of `--run` `--export` `--summary`
Test Plan: Test locally.
Reviewed By: malfet
Differential Revision: D23431508
fbshipit-source-id: 368b8dd8cd5099f39c7f5bc985203c417bf7af39
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/43893
Update `README.md` in oss, provide more examples, start from the most common use to specified use. Make `README.md` be more friendly and more specific.
Test Plan: `README.md` doesn't need test.
Reviewed By: malfet, seemethere
Differential Revision: D23420203
fbshipit-source-id: 1a4c146393fbcaf2893321e7892740edf5d0c248
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/43883
Check the result of GCC coverage in OSS is reasonable and ready to ship.
The amount of executable lines are not the same between `gcc` and `clang` because of the following reasons:
* Lines following are counted in `clang` but not in `gcc`:
1. empty line or line with only “{” or “}”
3. some comments are counted in clang but not in gcc
5. `#define ...` -- not supported by gcc according to official documentation
* Besides, a statement that explains to more than one line will be counted as only one executable line in gcc, but several lines in clang
## Advantage of `gcc` coverage
1. Much faster
- code coverage tool runtime is onle **4 min** (*ammazzzing!!*) by `gcc`, compared to **3 hours!!** by `clang`, to analyze all the tests' artifacts
2. Use less disk
- `Clang`'s artifacts will take as large as 170G, but `GCC` is 980M
Besides, also update `README.md`.
Test Plan:
Compare the result in OSS `clang` and OSS `gcc` with the same command:
```
python oss_coverage.py --run-only atest test_nn.py --interested-folder=aten
```
----
## GCC
**Summary**
> time: 0:15:45
summary percentage: 44.85%
**Report and Log**
[File Coverage Report](P140825162)
[Line Coverage Report](P140825196)
[Log](P140825385)
------
## CLANG
**Summary**
> time: 0:21:35
summary percentage: 44.08%
**Report and Log**
[File Coverage Report](P140825845)
[Line Coverage Report](P140825923)
[Log](P140825950)
----------
# Run all tests
```
# run all tests and get coverage over Pytorch
python oss_coverage.py
```
**Summary**
> time: 1:27:20. ( time to run tests: 1:23:33)
summary percentage: 56.62%
**Report and Log**
[File Coverage Report](P140837175)
[Log](P140837121)
Reviewed By: malfet
Differential Revision: D23416772
fbshipit-source-id: a6810fa4d8199690f10bd0a4f58a42ab2a22182b
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/43800
1. Move fbcode related coverage code to fb/ folder and add TARGETS so that we can use buck run to run the tool and solved the import probelm.
2. Write `README.md` to give users guidance about the tool
Test Plan:
On devserver:
```
buck run //caffe2/fb/code_coverage/tool:coverage -- //caffe2/c10:
```
More examples in README.md
Reviewed By: malfet
Differential Revision: D23404988
fbshipit-source-id: 4942cd0e0fb7bd28a5e884d9835b93f00adb7b92
