## `pip install -r requirements.txt` in build-from-source documentation
This line
81b5eff3c3/README.md (L182-L188)
Is outdated. Let's default to `requirements.txt`
### My problem
Without touching this codebase for years I'm trying to build repo for local development and run unit tests. I go to `build from source => Contributing.md`. I immediately run into various problems.
* [Contributing.md](https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#developing-pytorch) suggests one way of setting up environment different from [README.md#from-source](https://github.com/pytorch/pytorch/blob/master/README.md#from-source) that does not work for me.
* [README.md#from-source](https://github.com/pytorch/pytorch/blob/master/README.md#from-source) suggests a different set of dependencies than [`requirements.txt`](https://github.com/pytorch/pytorch/blob/master/requirements.txt), many of which are unnecessary, and there's still missing ones to run unit tests.
* Dependencies in `requirements.txt` are needed to run unit tests
So there's competing, inlined and outdated equally confident recommendations on how to set up. https://github.com/pytorch/pytorch/pull/91850 tries to remove one recommendation, this PR tries to make the default one simpler.
### Goals
* Improve society somewhat 😁
* Remove a dead end roundtrip in the developer onboarding funnel
* Update a duplicated & outdated line of documentation
* Two broken things => one broken thing
* Improve doc maintainability and nudge us to a productive discussion of what `requirements.txt` is there for.
### Non-goals
* Give a definite recommendation how to set up your machine for local development. I read the instructions in readme at this moment as an outline on how to do it.
* Say that `requirements.txt` is a definite guide to dependencies, I know it's not (but probably should be)
### Background
* Dependency handling/reproducibility in this repo is tricky! See geist of [this](fdbbd20f32/.github/requirements/README.md). There's many different sets of dependencies with different setups for different environments.
* There's been great attempts of _"one requirements.txt to rule them all"_ which got halted https://github.com/pytorch/pytorch/pull/60697/ see https://github.com/pytorch/pytorch/issues/61375
* The unofficial `requirements.txt` file seem to be .circleci/docker/requirements-ci.txt https://github.com/pytorch/pytorch/issues/72556
* Unofficial _"how to build from source"_ docs seem to be here https://github.com/pytorch/pytorch/tree/master/.circleci#how-to-build-a-binary-locally
### Considered alternatives
* a) Point only to python dependencies in `requirements.txt` **(Chosen option)**
```
conda install cmake ninja
pip install -r requirements.txt
```
This guarantees `python setup.py` to run (on my machine) and gets me one step closer to be able to `python test/run_test.py`
* b) Only add whats needed to `python setup.py install`. Point to `Contributing.md` for explanations on how to run tests (which doesn't exactly mention how yet).
```
conda create -n pytorch-source python cmake ninja pyyaml typing_extensions
conda activate pytorch-source
python setup.py develop
```
* c) Add dependencies needed to run (most) unit tests
I assume _"Install from source"_ describes how to "install so I can do development.". This is why we recommend `python setup.py develop`. Doing development implies running unit tests.
```
conda create -n pytorch-source python cmake ninja pytest click
conda activate pytorch-source
pip install -r requirements.txt xdoctest
python setup.py develop
python test/run_test.py --keep-going
```
This still eclectically goes outside the simple principle _"Use dependencies in requirements.txt"_ without solving the whole problem. Instructions to get tests to run is not the goal of this PR.
* d) Point to ex [`.circleci/docker/requirements-ci.txt`](https://github.com/pytorch/pytorch/blob/master/.circleci/docker/requirements-ci.txt) or any of the system-specific sets of pinned requirements like [`requirements-{conda-env-macOS-ARM64}.txt`](https://github.com/pytorch/pytorch/blob/master/.github/requirements/conda-env-macOS-ARM64)
I don't want to jump into this rabbit hole.
<details>
<summary>My system according to setup.py when verifying it runs</summary>
```
Target system: Darwin-21.6.0
Target processor: arm64
Host system: Darwin-21.6.0
Host processor: arm64
Detected C compiler: AppleClang @ /Library/Developer/CommandLineTools/usr/bin/cc
CMake: 3.22.1
Make program: /opt/homebrew/Caskroom/miniconda/base/envs/pytorch-source/bin/ninja
Python version : 3.10.8
Python executable : /opt/homebrew/Caskroom/miniconda/base/envs/pytorch-source/bin/python
Pythonlibs version : 3.10.8
Python library : /opt/homebrew/Caskroom/miniconda/base/envs/pytorch-source/lib/libpython3.10.a
Python includes : /opt/homebrew/Caskroom/miniconda/base/envs/pytorch-source/include/python3.10
Python site-packages: lib/python3.10/site-packages
```
</details>
See details in comments below.
[skip ci]
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91861
Approved by: https://github.com/malfet, https://github.com/ZainRizvi
The main changes are:
1. Remove outdated checks for old compiler versions because they can't support C++17.
2. Remove outdated CMake checks because it now requires 3.18.
3. Remove outdated CUDA checks because we are moving to CUDA 11.
Almost all changes are in CMake files for easy audition.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/90599
Approved by: https://github.com/soumith
If someone is building the project from source they're likely a contributor for which develop will be much more useful. For people that want to try the latest and greatest they can leverage the nightlies
Fixes #ISSUE_NUMBER
Pull Request resolved: https://github.com/pytorch/pytorch/pull/88507
Approved by: https://github.com/malfet
### Description
Simple README fix to add all supported Jetson platforms (Xavier NX, new Jetson Orin AGX)
### Issue
No issue was reported
### Testing
No tests required for given change
Pull Request resolved: https://github.com/pytorch/pytorch/pull/82404
Approved by: https://github.com/ezyang
Fixes various friction points with the documentation for onboarding new users and remove instructions that were no longer valid
Changes include:
- Listing prerequisites earlier, so that devs can ensure they're met before encountering error messages
- Removing linter invocations that are no longer valid
- Modifying instructions to install mkl packages to only apply to x86 based CPUs
[skip ci]
Pull Request resolved: https://github.com/pytorch/pytorch/pull/78682
Approved by: https://github.com/seemethere, https://github.com/janeyx99, https://github.com/malfet
Summary:
This changes the link for installing binaries to the page on pytorch.org that is entirely the download command selector (which isn't visible on a normal aspect ratio screen when the main website page first loads anymore).
This also includes some other random fixes:
* Update HUD link
* Clean ups
Fixes #{issue number}
Pull Request resolved: https://github.com/pytorch/pytorch/pull/66828
Reviewed By: malfet
Differential Revision: D31750654
Pulled By: driazati
fbshipit-source-id: aef9ceba71418f6f7648eab9a8c8a78d6c60518b
Summary:
1. In fact, Visual Studio isn't supported as CMAKE generator
2. I was asked many times why there's error as 'Could NOT find OpenMP'
3. Add Newly added Best Practices link in contributing.md
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63843
Reviewed By: seemethere, heitorschueroff
Differential Revision: D30514095
Pulled By: janeyx99
fbshipit-source-id: 76715a1d8c049122546e5a7778cafe54e4dfd5d6
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/55172
Description:
This is part 1 of series of PRs for supporting torch.jit.ignore as context manager. Following features are implemented in this PR:
- Unique name for the registered function under torch.jit.frontend module. The unique name is generated based on the file name and line number of context manager
- Forcing user to explicitly annotate the input and outputs.
- No side effects are considered.
Test Plan: Imported from OSS
Reviewed By: gmagogsfm
Differential Revision: D27895283
Pulled By: tugsbayasgalan
fbshipit-source-id: 5d36d9aa5d457055a6bb1676f264647a745ec36a
Summary:
Why:
To keep VS version always updated in README
1. update VS version link in CI. It's more convenient for my PR robot to update the version in README once the VS in CI is updated. and permlink isn't stable.
2. Move `building on legacy code` to development tips. The table is big and it looks the REAMD not updated at the first sight.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56686
Reviewed By: janeyx99
Differential Revision: D28272060
Pulled By: samestep
fbshipit-source-id: 4bb879ea2914cc8bcd68343a9ed230418e1f9268
Summary:
* Visual studio versions: clarify and shorten.
* Remove obsolete note about a bug that has been fixed.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56193
Reviewed By: albanD
Differential Revision: D27939766
Pulled By: ezyang
fbshipit-source-id: e142ec04ba98d5468f28ddf2e8bba5d99d3cfc26
Summary:
There might be regressions in newest VS.
Remind users to choose the stable VC version as our CI's
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54933
Reviewed By: walterddr
Differential Revision: D27466645
Pulled By: malfet
fbshipit-source-id: a6a1ebea4cc1b22e13c7342ee4c061afcef7e2b5
Summary:
Instructions for compiling PyTorch from source for ROCm were missing now that PyTorch 1.8 announced beta support for ROCm.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/53845
Reviewed By: heitorschueroff
Differential Revision: D27237916
Pulled By: malfet
fbshipit-source-id: c8be92fd76ea8df7e9f6944c0036568189f58808
Summary:
I found a number of spelling & grammatical mistakes in the repository. Previously I had these fixes submitted individually, but I saw that a single word change was apparently too small for a PR to be merged. Hopefully this new PR has a sufficient number of changes.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/48592
Reviewed By: ejguan
Differential Revision: D25224216
Pulled By: mrshenli
fbshipit-source-id: 2af3db2aee486563efd0dffc4e8f777306a73e44
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/42629
How to approach reviewing this diff:
- The new codegen itself lives in `tools/codegen`. Start with `gen.py`, then read `model.py` and them the `api/` folder. The comments at the top of the files describe what is going on. The CLI interface of the new codegen is similar to the old one, but (1) it is no longer necessary to explicitly specify cwrap inputs (and now we will error if you do so) and (2) the default settings for source and install dir are much better; to the extent that if you run the codegen from the root source directory as just `python -m tools.codegen.gen`, something reasonable will happen.
- The old codegen is (nearly) entirely deleted; every Python file in `aten/src/ATen` was deleted except for `common_with_cwrap.py`, which now permanently finds its home in `tools/shared/cwrap_common.py` (previously cmake copied the file there), and `code_template.py`, which now lives in `tools/codegen/code_template.py`. We remove the copying logic for `common_with_cwrap.py`.
- All of the inputs to the old codegen are deleted.
- Build rules now have to be adjusted to not refer to files that no longer exist, and to abide by the (slightly modified) CLI.
- LegacyTHFunctions files have been generated and checked in. We expect these to be deleted as these final functions get ported to ATen. The deletion process is straightforward; just delete the functions of the ones you are porting. There are 39 more functions left to port.
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Test Plan: Imported from OSS
Reviewed By: bhosmer
Differential Revision: D23183978
Pulled By: ezyang
fbshipit-source-id: 6073ba432ad182c7284a97147b05f0574a02f763
Summary:
Fixed grammatical errors and punctuation so that it be can more understandable.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/43779
Reviewed By: ZolotukhinM
Differential Revision: D23407849
Pulled By: malfet
fbshipit-source-id: 09c064ce68d0f37f8023c2ecae8775fc00541a2c
Summary:
The changes are minor.
1. Add back the external links so that readers can find out more about external tools on how to accelerate PyTorch.
2. Fix typo
Pull Request resolved: https://github.com/pytorch/pytorch/pull/43100
Reviewed By: colesbury
Differential Revision: D23192251
Pulled By: mrshenli
fbshipit-source-id: dde54b7942ebff5bbe3d58ad95744c6d95fe60fe
Summary:
Apply syntax highlighting to the command in `README.md`. This makes `README.md` easier to read.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/42065
Reviewed By: pbelevich
Differential Revision: D22753418
Pulled By: mrshenli
fbshipit-source-id: ebfa90fdf60478c34bc8a7284d163e0254cfbe3b
Summary:
Added a link to `CONTRIBUTION.md` in `README.md` for easy reference.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/40353
Differential Revision: D22167138
Pulled By: ezyang
fbshipit-source-id: fe7b7f190c8135fdd2e71696c1cf8d84bcd40fc6
Summary:
This corrects the build info for ppc64le in the main README.
I am opening this PR before renaming the build job. (So, the "live" master README has the correct "live" link and the PR does not.)
Immediately after submitting the PR, I will correct the name of the build job. This will make the new PR link correct, and the current "master" link will briefly appear broken until this PR gets merged.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/39475
Differential Revision: D21883184
Pulled By: malfet
fbshipit-source-id: 148353b632448c98e5aff560d31642328afe7963
Summary:
I think, it's help faster compile pytorch from source without errors about incompatible compiler(such as: unsupported GNU version! gcc versions later than 8 are not supported!)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/37301
Differential Revision: D21396682
Pulled By: ngimel
fbshipit-source-id: 5e21c36ee550424e820f3aa6e6131ca858994ae4
Summary:
Some more cleanup now that we no longer support python2 or 3.5 on master and eventually PyTorch 1.6 release.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/35677
Differential Revision: D20838097
Pulled By: orionr
fbshipit-source-id: 95d553a1e8769f3baa395e0bc6d4ce7cd93236e9
Summary:
## Commit Message:
Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile.
Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images.
Adds the new Dockerfile process to our CircleCI workflows
## How to build:
Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving.
### Development images
For `runtime` images:
```
make -f docker.Makefile runtime-image
```
For `devel` images:
```
make -f docker.Makefile devel-image
```
Builds are tagged as follows:
```bash
docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type}
```
Example:
```
docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime
```
### Official images
Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch)
To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries:
Example:
```
make -f docker.Makefile BUILD_TYPE=official runtime-image
```
## How to push:
Pushing is also super simple (And will automatically tag the right thing based off of the git tag):
```
make -f docker.Makefile runtime-push
make -f docker.Makefile devel-push
```
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515
Differential Revision: D19558619
Pulled By: seemethere
fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
Summary:
For ppc64le, we no longer plan to run regular builds on Python 2.7, and we wish to stop
publicizing the build status for those two builds (ppc64le/CPU and ppc64le/GPU each on py27).
This pull request simply removes the build status links for these two builds, replacing them
with a generic dash character (consistent with other un-publicized builds within the table).
Pull Request resolved: https://github.com/pytorch/pytorch/pull/32315
Differential Revision: D19435939
Pulled By: soumith
fbshipit-source-id: c9f31e7acba83e42f6a758ac011bbef36fd8aaa0
Summary:
Was just trying to build pytorch from source and had a small hiccup because the instructions say to `conda install typing`. Because `typing` is a built-in module in recent Python 3 versions, conda interpreted that to mean that I want Python 2. So I added a note to the docs.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/29103
Differential Revision: D18294139
Pulled By: soumith
fbshipit-source-id: 621a2f62ebe870520197baec8f8bcdc1a0c57de9
Summary:
I know this is really a minor one and the list of people to mention will be significantly larger in the future. Nevertheless I would love to see my name written in correct international spelling (the strange German o-umlaut in my name becomes oe).
Pull Request resolved: https://github.com/pytorch/pytorch/pull/28281
Differential Revision: D18007518
Pulled By: ezyang
fbshipit-source-id: 1d03065636d7f65ac6b376690256c0d021482958
Summary:
Technically, we don't need a C++14 compiler yet, but we will soon stop support for GCC 4. Requiring a "C++14" compiler excludes GCC 4, so it is a defensive statement. Some time later, we will actually require a C++14 compiler.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/26670
Differential Revision: D17907257
Pulled By: smessmer
fbshipit-source-id: 5363d714f8d93597db008135f681b2e14d052fa0
Summary:
Grammar edits to the Readme file to make it read better in English
Pull Request resolved: https://github.com/pytorch/pytorch/pull/27808
Differential Revision: D17901414
Pulled By: soumith
fbshipit-source-id: 02e67289dafaf9280cb1c3bb2f37087cd134cc23
Summary:
There is a magma package for the newest CUDA verson (10.1), mention it here lest someone try to mistakenly use the version for CUDA 10.0.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/27325
Differential Revision: D17749535
Pulled By: soumith
fbshipit-source-id: 2d34a7af1218e6157935bfd5e03f4d2c0f00f200
Summary:
There is an issue with the torchvision version not matching the pytorch version if one builds the docker from a tag, see issue https://github.com/pytorch/pytorch/issues/25917. The current solution requires one to re-init the submodules or manually change the version of torchvision. This PR allows one to build the docker image without torchvision, which not only fixes the above mentioned bug but also frees non-image pytorch users from the tyranny of torchvision 😆.
In all seriousness, for NLP researchers especially torchvision isn't a necessity for pytorch and all non-essential items shouldn't be in the docker. This option removes one extra thing that can go wrong.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/26168
Differential Revision: D17550001
Pulled By: soumith
fbshipit-source-id: 48b8b9e22b75eef3afb392c618742215d3920e9d
Summary:
local build is slow... test in CI...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/26083
Differential Revision: D17346949
Pulled By: ailzhang
fbshipit-source-id: f552d1a4be55ad4e2bd915af7c5a2c1b6667c446
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/25596
Giving up on trying to limit this to just a py2 dependency
Test Plan: Imported from OSS
Differential Revision: D17171063
Pulled By: jamesr66a
fbshipit-source-id: 5df35fd128f3051dd9c6709f7d45323fedc12e65
Summary:
Currently specifying different build options in respect to the "USE_"
series is in quite a disarray. There are a lot of build options that
accept three variants: USE_OPTION, WITH_OPTION, and NO_OPTION. Some
build options only accept USE_ and NO_ variant. Some accept only USE_.
This inconsistency is quite confusing and hard to maintain.
To resolve this inconsistency, we can either let all these build options
support all three variants, or we only support the USE_ variant.
This commit makes a step to the latter choice, i.e., deprecates and sets
a date for removing the NO_ and WITH_ variants and keeps only the
USE_ variant. This is likely better than the former solution because:
- NO_ and WITH_ variants are not documented.
- CMakeLists.txt only has the USE_ variants for relevant build options
defined. It would be a surprise that when user pass these variables to
CMake during rebuild and find them ineffective.
- Multiple variants are difficult to maintain.
- The behavior is confusing if more than one variant is passed. For
example, what to be expected if one sets "NO_CUDA=1 USE_CUDA=1"?
The downside is that this will break backward compatibility for existing
build scripts in the future (if they used the undocumented build
options).
Pull Request resolved: https://github.com/pytorch/pytorch/pull/22474
Differential Revision: D16149396
Pulled By: ezyang
fbshipit-source-id: 7145b88ad195db2051772b9665dd708dfcf50b7d
Summary:
Fixes#21026.
1. Improve build docs for Windows
2. Change `BUILD_SHARED_LIBS=ON` for Caffe2 local builds
3. Change to out-source builds for LibTorch and Caffe2 (transferred to #21452)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/21190
Differential Revision: D15695223
Pulled By: ezyang
fbshipit-source-id: 0ad69d7553a40fe627582c8e0dcf655f6f63bfdf
Summary:
Add an option to setup.py to stop the build process once cmake terminates. This leaves users a chance to fine adjust build options. Also update README accordingly.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/21034
Differential Revision: D15530096
Pulled By: soumith
fbshipit-source-id: 71ac6ff8483c3ee77c38d88f0d059db53a7d3901
Summary:
I didn't update the Windows references because I wasn't sure if they apply to CUDA 9. peterjc123 what should the Windows section say?
Pull Request resolved: https://github.com/pytorch/pytorch/pull/20718
Differential Revision: D15459276
Pulled By: colesbury
fbshipit-source-id: 917e22f8ac75378d88c962c226b5a42b6799c79a
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/20726
Edward says it doesn't actually provide compilers,
but it does provide dependencies, so let's mention that instead.
Reviewed By: ezyang
Differential Revision: D15423316
fbshipit-source-id: 9b384f88e5bf7a3d2c132508620c276b49e1569f
Summary:
Sometimes people need to checkout an older version and build PyTorch. In that case, they need to do `git submodule sync` and maybe `git submodule update --init` as mentioned [here](https://github.com/pytorch/pytorch/issues/20074).
Pull Request resolved: https://github.com/pytorch/pytorch/pull/20088
Differential Revision: D15195729
Pulled By: soumith
fbshipit-source-id: 73232b801e5524cdba462dd504fb973d95d0498c
Summary:
Thanks to dusty-nv , we now have Stable and Weekly wheels provided for the NVIDIA Jetson Platform. They require JetPack 4.2.
He's also maintaining source build instructions.
This PR adds links to the binaries and source build instructions to the README.
The links are dynamic, so when new stable / weekly wheels are available, Dustin will update the same URL to point to the new files
Pull Request resolved: https://github.com/pytorch/pytorch/pull/18990
Differential Revision: D14820158
Pulled By: soumith
fbshipit-source-id: 761a56557decb72ad9c1b9f8a2745667f558eec3
Summary:
Trainer has been removed long time ago
Pull Request resolved: https://github.com/pytorch/pytorch/pull/18980
Differential Revision: D14819855
Pulled By: ezyang
fbshipit-source-id: f62020e688ebf6663416aec7435bf1f531607941
