Summary:
We want to store the file names that triggers each test suite so that we can use this data for categorizing those test files.
~~After considering several solutions, this one is the most backwards compatible, and the current test cases in test_testing.py for print test stats don't break.~~
The previous plan did not work, as there are multiple Python test jobs that spawn the same suites. Instead, the new S3 format will store test files (e.g., `test_nn` and `distributed/test_distributed_fork`) which will contain the suites they spawn, which will contain the test cases run within the suite. (Currently, there is no top layer of test files.)
Because of this major structural change, a lot of changes have now been made (thank you samestep!) to test_history.py and print_test_stats.py to make this new format backwards compatible.
Old test plan:
Make sure that the data is as expected in S3 after https://github.com/pytorch/pytorch/pull/52873 finishes.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/52869
Test Plan: Added tests to test_testing.py which pass, and CI.
Reviewed By: samestep
Differential Revision: D26672561
Pulled By: janeyx99
fbshipit-source-id: f46b91e16c1d9de5e0cb9bfa648b6448d979257e
Summary:
This is less surprising than the current default, `--delta=12`.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/52877
Test Plan: Run the example commands from `tools/test_history --help` and check that their output matches that shown.
Reviewed By: pritamdamania87
Differential Revision: D26674258
Pulled By: samestep
fbshipit-source-id: 1413e11519854b0a47e14af2f1d20c57f145dacd
Summary:
This PR makes several UX improvements to `tools/test_history.py`:
- warn if `--all` is unset and no jobs are passed
- print output even in `multiline` mode if no reports are found for a commit
- this makes it easier to tell whether the script is just hanging
- if there are multiple reports for a commit/job pair, say so
- distinguish between not finding any reports and just not finding the desired test in the reports found
- don't require the suite name as a CLI arg, just use the test name
Pull Request resolved: https://github.com/pytorch/pytorch/pull/52640
Test Plan: Example shell session: https://pastebin.com/SSemHqP8
Reviewed By: walterddr
Differential Revision: D26594350
Pulled By: samestep
fbshipit-source-id: 9ce2245f91eef289817aafe955a4343d4a068eda
