Adds a ruff lint rule to ban raising raw exceptions. Most of these should at the very least be runtime exception, value errors, type errors or some other errors. There are hundreds of instance of these bad exception types already in the codebase, so I have noqa'd most of them. Hopefully this error code will get commiters to rethink what exception type they should raise when they submit a PR.
I also encourage people to gradually go and fix all the existing noqas that have been added so they can be removed overtime and our exception typing can be improved.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/124570
Approved by: https://github.com/ezyang
Fixes https://github.com/pytorch/pytorch/issues/118129
Suppressions automatically added with
```
import re
with open("error_file.txt", "r") as f:
errors = f.readlines()
error_lines = {}
for error in errors:
match = re.match(r"(.*):(\d+):\d+: error:.*\[(.*)\]", error)
if match:
file_path, line_number, error_type = match.groups()
if file_path not in error_lines:
error_lines[file_path] = {}
error_lines[file_path][int(line_number)] = error_type
for file_path, lines in error_lines.items():
with open(file_path, "r") as f:
code = f.readlines()
for line_number, error_type in sorted(lines.items(), key=lambda x: x[0], reverse=True):
code[line_number - 1] = code[line_number - 1].rstrip() + f" # type: ignore[{error_type}]\n"
with open(file_path, "w") as f:
f.writelines(code)
```
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Co-authored-by: Catherine Lee <csl@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/118533
Approved by: https://github.com/Skylion007, https://github.com/zou3519
Fixes https://github.com/pytorch/pytorch/issues/118129
Suppressions automatically added with
```
import re
with open("error_file.txt", "r") as f:
errors = f.readlines()
error_lines = {}
for error in errors:
match = re.match(r"(.*):(\d+):\d+: error:.*\[(.*)\]", error)
if match:
file_path, line_number, error_type = match.groups()
if file_path not in error_lines:
error_lines[file_path] = {}
error_lines[file_path][int(line_number)] = error_type
for file_path, lines in error_lines.items():
with open(file_path, "r") as f:
code = f.readlines()
for line_number, error_type in sorted(lines.items(), key=lambda x: x[0], reverse=True):
code[line_number - 1] = code[line_number - 1].rstrip() + f" # type: ignore[{error_type}]\n"
with open(file_path, "w") as f:
f.writelines(code)
```
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/118533
Approved by: https://github.com/Skylion007, https://github.com/zou3519
This PR increases the read size for the `hub.download_url_to_file` function from 8,192 bytes to 131,072 bytes (128 * 1,024), as reading in larger chunks should be more efficient. The size could probably be larger still, at the expense of the progress bar not getting updated as often.
It re-introduces use of the `READ_DATA_CHUNK` constant that was originally used for this purpose in 4a3baec961 and since forgotten.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/116536
Approved by: https://github.com/NicolasHug
### Description
The `download_url_to_file` function in torch.hub uses a temporary file to prevent overriding a local working checkpoint with a broken download.This temporary file is created using `NamedTemporaryFile`. However, since `NamedTemporaryFile` creates files with overly restrictive permissions (0600), the resulting download will not have default permissions and will not respect umask on Linux (since moving the file will retain the restrictive permissions of the temporary file). This is especially problematic when trying to share model checkpoints between multiple users as other users will not even have read access to the file.
The change in this PR fixes the issue by using custom code to create the temporary file without changing the permissions to 0600 (unfortunately there is no way to override the permissions behaviour of existing Python standard library code). This ensures that the downloaded checkpoint file correctly have the default permissions applied. If a user wants to apply more restrictive permissions, they can do so via usual means (i.e. by setting umask).
See these similar issues in other projects for even more context:
* https://github.com/borgbackup/borg/issues/6400
* https://github.com/borgbackup/borg/issues/6933
* https://github.com/zarr-developers/zarr-python/issues/325
### Issue
https://github.com/pytorch/pytorch/issues/81297
### Testing
Extended the unit test `test_download_url_to_file` to also check permissions.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/82869
Approved by: https://github.com/vmoens
When minifying extremely large repros, the minifier can run out of memory. This is because, for delta debugging, the minifier keeps a copy of every intermediate output in the network. This can easily put you over the memory limit for your GPU. To make matters worse, we cannot easily delta debug in such a situation, as delta debugging involves replacing intermediates with inputs, but doing so can cause an intermediate to become live longer than its actual extent in the original model (since inputs all have to be allocated up front).
The strategy in this PR is to use `load_tensor` from the previous PR to offer a low memory mode for delta debugging. Instead of putting intermediates as inputs, we instead load them in the middle of the graph in question. If, through DCE, the load_tensor ends up floating to the top of the graph, we can input-ify it. We now no longer save all intermediates in memory, but instead save them to disk. I used this to successfully minify the repro that helped us solve https://github.com/pytorch/pytorch/pull/100332
The testing is not very good. I can try to add more robust testing but it will involve a more involved refactor to FX minifier. Let me know if that's what you want.
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/100546
Approved by: https://github.com/anijain2305, https://github.com/voznesenskym
We are using the idiom
```py
sys.path.insert(0, path)
# do something
sys.path.remove(path)
```
three times in `torch.hub`. This is a textbook case for using a context manager. In addition, by using `try` / `finally` we can enforce the Python path is back in its original state even if the actual action raises an exception:
```py
import sys
path = "/tmp"
# PR
try:
sys.path.insert(0, path)
try:
# Any exception raised while performing the actual functionality
raise Exception
finally:
sys.path.remove(path)
except Exception:
assert path not in sys.path
# main
try:
sys.path.insert(0, path)
# Any exception raised while performing the actual functionality
raise Exception
sys.path.remove(path)
except Exception:
assert path in sys.path
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/75786
Approved by: https://github.com/NicolasHug
This is a new version of #15648 based on the latest master branch.
Unlike the previous PR where I fixed a lot of the doctests in addition to integrating xdoctest, I'm going to reduce the scope here. I'm simply going to integrate xdoctest, and then I'm going to mark all of the failing tests as "SKIP". This will let xdoctest run on the dashboards, provide some value, and still let the dashboards pass. I'll leave fixing the doctests themselves to another PR.
In my initial commit, I do the bare minimum to get something running with failing dashboards. The few tests that I marked as skip are causing segfaults. Running xdoctest results in 293 failed, 201 passed tests. The next commits will be to disable those tests. (unfortunately I don't have a tool that will insert the `#xdoctest: +SKIP` directive over every failing test, so I'm going to do this mostly manually.)
Fixes https://github.com/pytorch/pytorch/issues/71105
@ezyang
Pull Request resolved: https://github.com/pytorch/pytorch/pull/82797
Approved by: https://github.com/ezyang
As pointed by #71205, `torch.hub.load` assumes that the user trusts the repo from where the code is gathered and exececuted. We propose a solution to make sure that the user is aware of the security threat that this can represent.
**Solution**: Adds a `trust_repo` parameter to the `load`, `list` and `help` functions in torch.hub.
For now, the default `trust_repo=None` warns that, in the future, the user will need to authorize explicitly every repo before downloading it.
Once the repo has been trusted (via `trust_repo=True` or via a command prompt input) it will be added to the list of trusted repositories.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/72060
Approved by: https://github.com/NicolasHug
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/67990
Duplicate of the following PR which was merged by mistake without ghimport
https://github.com/pytorch/pytorch/pull/67914
cc albanD NicolasHug
Test Plan: Imported from OSS
Reviewed By: H-Huang
Differential Revision: D32247560
Pulled By: jdsgomes
fbshipit-source-id: 8ba5ba7d17fc3d0d2c377da467ea805822e21ec1
Summary:
Closes https://github.com/pytorch/pytorch/issues/63753
This PR changes the assumption regarding the default branch of a repo to the following:
> If main exist then use main,otherwise use master
This will make torchhub more robust w.r.t. to the ongoing changes where repo use `main` instead of `master` as the development / default branch.
cc nairbv NicolasHug
Pull Request resolved: https://github.com/pytorch/pytorch/pull/64364
Reviewed By: saketh-are
Differential Revision: D30731551
Pulled By: NicolasHug
fbshipit-source-id: 7232a30e956dcccca21933a29de5eddd711aa99b
Summary:
This PR adds more detailed error messages to torchhub if the commit hash validation goes wrong, providing suggestions to the users on how to resolve the issue.
It also documents why such validation is important.
EDIT: it also avoids validatating some stuff when we know "stuff" isn't a commit since there's no risk in this case
CC malfet mthrok
cc nairbv NicolasHug
Pull Request resolved: https://github.com/pytorch/pytorch/pull/64362
Reviewed By: gchanan, malfet
Differential Revision: D30731191
Pulled By: NicolasHug
fbshipit-source-id: d1ee7c2ef2591dd7a5291977af1635ada2552d1b
Summary:
This PR:
- adds a few details regarding the newly added `skip_validation` parameter https://github.com/pytorch/pytorch/pull/62139
- uses double-backticks instead of single-backticks since this is rst, not mardown.
- adds a few minor doc nits here and there
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63783
Reviewed By: zou3519
Differential Revision: D30696658
Pulled By: NicolasHug
fbshipit-source-id: 6f01c7eb3cfcd7e17e4c33c09d193054fa18ad36
Summary:
This PR Fixes the help() and list() torchhub functions which were probably failing for Windows since the `/` OS separator was hardcoded.
Before merging this I need to double check whether the CI actually runs the corresponding tests on Windows or not
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63773
Reviewed By: zou3519
Differential Revision: D30695664
Pulled By: NicolasHug
fbshipit-source-id: fac328163fd05db804a8186ae28f22b3cc3a6404
Summary:
This PR removes an outdated comment about Python2 that was orginally introduced in https://github.com/pytorch/pytorch/pull/25083/files. The code has changed since then, but the comment wasn't removed.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63757
Reviewed By: zou3519
Differential Revision: D30695656
Pulled By: NicolasHug
fbshipit-source-id: 431cf414588b9e5a1ad6acdae724ff5af1b16971
Summary:
This PR addresses an old comment about Python2 EOL, directly putting some parameters in the function signature instead of in a `**kargs` dict.
I believe the changes are fully backward compatible.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63755
Reviewed By: zou3519
Differential Revision: D30695634
Pulled By: NicolasHug
fbshipit-source-id: 398f347c5a04bfb58e77e46773a869cb9d0eb225
