Setting "build" options in the RC file prevents applying the flags to the query command. "common" works for both build and query commands.
Flags like `--experimental_cc_shared_library` changes the starlark semantics which forces re-fetching all repo rules when switching between commands.
Ideally, more flags should be common instead of build.
PiperOrigin-RevId: 728658294
Imported from GitHub PR https://github.com/openxla/xla/pull/21901
Add rocm 6.1.0 dependency for ubuntu 20.04
Copybara import of the project:
--
0acf028eeca5923c7f2aa5762297686836eda310 by Alexandros Theodoridis <atheodor@amd.com>:
Add rocm6.1 deps for ubuntu 20.04
--
fc88c83061d6efff2482599489d622ab3114b9a7 by Alexandros Theodoridis <atheodor@amd.com>:
Fix hermetic build for 6.0
--
73ace5591f4731e1b95b6d3e6a349b528977c580 by Alexandros Theodoridis <atheodor@amd.com>:
Add ci config for hermetic build
--
bbc048bcffd9d35bfad76ff816ed22f3e3f761f8 by Alexandros Theodoridis <atheodor@amd.com>:
Introduce rocm 6.1.0 dependency for 22.04
--
9776f398c2711ba37333d29b934d6ba67c55dbef by Alexandros Theodoridis <atheodor@amd.com>:
Add missing 24.04 redist
--
acf275d57cc185b9c2122d5930d8cf54e473ad95 by Alexandros Theodoridis <atheodor@amd.com>:
Fix test
--
3e49285b0f55597ab5f44c1d0a422bf931d72cda by Alexandros Theodoridis <atheodor@amd.com>:
Add comment explaining the reason for a new target
--
35838bf8d6e678717e9b1c551f840918b00a91f8 by Alexandros Theodoridis <atheodor@amd.com>:
Rever force verbose in the compiler wrapper
--
2952e115b044e1a8ac8aadc7eac7802e8d79cf91 by Alexandros Theodoridis <atheodor@amd.com>:
Add explanation comment for the new target
Merging this change closes#21901
PiperOrigin-RevId: 721043735
Imported from GitHub PR https://github.com/openxla/xla/pull/21396
`NCCL_MAX_NCHANNELS=1` is necessary for collective ops tests to pass in CI.
As for XNNPACK problem, similar fix has already been done for single-gpu tests -> https://github.com/openxla/xla/pull/20975
Copybara import of the project:
--
631fa6b7fc859c083e0735d2ce47167cbf57c174 by Milica Makevic <Milica.Makevic@amd.com>:
Fix build break due to XNNPACK update
--
d226a07701ddd88d45e7b27406d2915d032832a4 by Milica Makevic <Milica.Makevic@amd.com>:
Add NCCL_MAX_NCHANNELS env variable to multi gpu tests
--
b826eee7ab0e2a1f4d062871465ab4dac48ead37 by Milica Makevic <Milica.Makevic@amd.com>:
Split bazel command arguments in multiple lines
Merging this change closes#21396
PiperOrigin-RevId: 715779335
Also sets `test --test_env="GTEST_INSTALL_FAILURE_SIGNAL_HANDLER=1"` in the bazelrc to retain stacktrace behavior when tests are killed
PiperOrigin-RevId: 715072546
`set -u` (does not allow unbound variables) has been removed from all scripts.
This is due to Docker on Windows treating variables in an env file,
set to an empty value (`MY_VAR=`), as unbound variables. Consequently,
these variables, even though they are "set", do not make it into the Docker
container at all, and various checks for those variables fail outright.
PiperOrigin-RevId: 713717958
These were created to be able to set a different path to the toolchain configs when building XLA. Instead of creating duplicated configs, we will use copybara to transform paths in the .bazelrc between TF and XLA.
PiperOrigin-RevId: 707109121
Imported from GitHub PR https://github.com/openxla/xla/pull/19660
This PR switches the default rocm build to clang as the gcc config is broken at the moment.
Copybara import of the project:
--
ea48f7c480d110eab3f133ed6ea8989da0e1e724 by Alexandros Theodoridis <atheodor@amd.com>:
[ROCm] switch rocm build to clang
--
2743fabafd6a358c05e858781064e7fa2e389c78 by Alexandros Theodoridis <atheodor@amd.com>:
Remove explicit clang path from the bazelrc rocm config
--
202dea0a80602cafdbee6067d8f20dc3055c6bbb by Alexandros Theodoridis <atheodor@amd.com>:
Address review comments
Merging this change closes#19660
PiperOrigin-RevId: 699222609
Rename `nvcc_clang` to `cuda_nvcc` according to the changes in JAX
Fix "Line too long" error in xla/.../configure.py issued by PyLint
PiperOrigin-RevId: 681940132
Imported from GitHub PR https://github.com/openxla/xla/pull/16882Fixes#16877
Copybara import of the project:
--
1ff356ac0870002b369c3ec09547aae2a62c70e2 by tchatow <tchatow@users.noreply.github.com>:
Symlink hermetic cuda headers to permit clang cuda version detection
Fixes#16877
Merging this change closes#16882
PiperOrigin-RevId: 679764212
This test verifies whether the API v2 packages can be imported from the
current build. It utilizes the `_api/v2/api_packages.txt` list of packages from
the local wheel file specified in the `requirements_lock_<python_version>.txt`.
The test should be executed after the TF wheel was built and put into `dist` dir inside Tensorflow repository.
PiperOrigin-RevId: 676893008
This updates CUDA for both TF and XLA. It also enables the CUDA driver forward
compatibility mode for XLA since XLA's CUDA graph integration needs a newer
driver version.
PiperOrigin-RevId: 673974335
These configs are no longer needed as the old GCC toolchain is no longer supported.
If you still need these configs, add them to a user-specific bazelrc file for your project. Refer to the Bazel documentation https://bazel.build/run/bazelrc for more guidance.
PiperOrigin-RevId: 673047905
Remote configurations of python repositories are removed because hermetic Python repository rules install and configure python modules in Bazel cache on the host machine. The cache is shared across host and remote machines.
PiperOrigin-RevId: 671512134
The CUDA and NCCL repositories are created on a host machine now and shared via Bazel cache between host and remote machines.
PiperOrigin-RevId: 671089856
Add `--@local_config_cuda//cuda:override_include_cuda_libs` to override settings for TF wheel.
Forbid building TF wheel with `--@local_config_cuda//cuda:include_cuda_libs=true`
PiperOrigin-RevId: 666848518
1) Hermetic CUDA rules allow building wheels with GPU support on a machine without GPUs, as well as running Bazel GPU tests on a machine with only GPUs and NVIDIA driver installed. When `--config=cuda` is provided in Bazel options, Bazel will download CUDA, CUDNN and NCCL redistributions in the cache, and use them during build and test phases.
[Default location of CUNN redistributions](https://developer.download.nvidia.com/compute/cudnn/redist/)
[Default location of CUDA redistributions](https://developer.download.nvidia.com/compute/cuda/redist/)
[Default location of NCCL redistributions](https://pypi.org/project/nvidia-nccl-cu12/#history)
2) To include hermetic CUDA rules in your project, add the following in the WORKSPACE of the downstream project dependent on XLA.
Note: use `@local_tsl` instead of `@tsl` in Tensorflow project.
```
load(
"@tsl//third_party/gpus/cuda/hermetic:cuda_json_init_repository.bzl",
"cuda_json_init_repository",
)
cuda_json_init_repository()
load(
"@cuda_redist_json//:distributions.bzl",
"CUDA_REDISTRIBUTIONS",
"CUDNN_REDISTRIBUTIONS",
)
load(
"@tsl//third_party/gpus/cuda/hermetic:cuda_redist_init_repositories.bzl",
"cuda_redist_init_repositories",
"cudnn_redist_init_repository",
)
cuda_redist_init_repositories(
cuda_redistributions = CUDA_REDISTRIBUTIONS,
)
cudnn_redist_init_repository(
cudnn_redistributions = CUDNN_REDISTRIBUTIONS,
)
load(
"@tsl//third_party/gpus/cuda/hermetic:cuda_configure.bzl",
"cuda_configure",
)
cuda_configure(name = "local_config_cuda")
load(
"@tsl//third_party/nccl/hermetic:nccl_redist_init_repository.bzl",
"nccl_redist_init_repository",
)
nccl_redist_init_repository()
load(
"@tsl//third_party/nccl/hermetic:nccl_configure.bzl",
"nccl_configure",
)
nccl_configure(name = "local_config_nccl")
```
PiperOrigin-RevId: 662981325
This is needed for hermetic CUDA integration in Google ML projects since tensorRT is not distributed in the same free way as other CUDA/CUDNN distributives.
PiperOrigin-RevId: 662601190
Imported from GitHub PR https://github.com/openxla/xla/pull/11299
This PR aims to enable the XLA test cases on the Windows Platform. The changes made:
1. Changed the .bazelrc file to use the correct toolchain and platform
This change will allow the user to successfully run XLA tests on the Windows platform using the Clang compiler using '--config=win_clang' in the bazel command
2. Added conditions to a few test cases to successfully run on the Windows platform
These test cases check the exit/termination status of a process
WIFEXITED is typically supported in POSIX-compliant operating systems like Unix and Linux to check if a process has terminated normally. WEXITSTATUS allows examining the termination status of child processes. However, these macros are not Windows compliant, hence the additional condition block was added to check the exit/termination status of process or child process for the Windows platform
Copybara import of the project:
--
ece9eefa224a6d051bcac089fe2a9a393af16a2b by Raunak <mayank.kumar.raunak@intel.com>:
Enable XLA Windows
--
347c0326af8f608047f06345cad4dfbb53a52150 by mraunak <83710963+mraunak@users.noreply.github.com>:
Update interactive_graphviz_bin_test.cc
--
2d4a3c2bb2ea23f12029583c53087d8739da0319 by mraunak <83710963+mraunak@users.noreply.github.com>:
Update xla/tools/interactive_graphviz_bin_test.cc
Co-authored-by: Penporn Koanantakool <38085909+penpornk@users.noreply.github.com>
--
90ad8b2730900d7f82b0fb1a83a73ffa2e452e0e by mraunak <83710963+mraunak@users.noreply.github.com>:
Update run_hlo_module_bin_test.cc
--
7f31412f6b57e53bd56ba92149b015fcba92b07c by mraunak <83710963+mraunak@users.noreply.github.com>:
Update xla/tools/run_hlo_module_bin_test.cc
Co-authored-by: Penporn Koanantakool <38085909+penpornk@users.noreply.github.com>
--
4d39e35461f6977f36404a435c68b4809fb51a44 by mraunak <83710963+mraunak@users.noreply.github.com>:
Update hlo_expand_test.cc
--
816b9ae0498831b55139c72d627d07f05e51213b by mraunak <83710963+mraunak@users.noreply.github.com>:
Update hlo_expand_test.cc
--
a728fff1aca4258602c3d7c78afcc7d38b545b7a by mraunak <83710963+mraunak@users.noreply.github.com>:
Update hlo_expand_test.cc
--
ffcb6861becf8de7a5c4e64ef0f19d977475d281 by mraunak <83710963+mraunak@users.noreply.github.com>:
Update interactive_graphviz_bin_test.cc
--
f181497793c2c48079d72b545e6efb837f490504 by mraunak <83710963+mraunak@users.noreply.github.com>:
Update interactive_graphviz_bin_test.cc
--
f9af75677e4663b3348e9e89376262eaff389ea9 by mraunak <83710963+mraunak@users.noreply.github.com>:
Update run_hlo_module_bin_test.cc
Merging this change closes#11299
PiperOrigin-RevId: 627216606
We expect a noticable performance improvement on NVIDIA GPUs that support
sm_89 (NVIDIA L4 and L40) over just providing sm_80 (which is compatible).
To avoid an increase in package size we remove support for the oldest generation of GPUs (sm_50). The last GPU that does not support
sm_60 was released in early 2016 (8 years ago) and it's a laptop GPU.
The last desktop GPU without sm_60 support was released in in early 2015 (9 years ago).
PiperOrigin-RevId: 610648659
`fully_async` lets Bazel upload the build event in background and does not prematurely mark the build as failed in case it is taking longer than the value in `--bes_timeout`. However, BES uploads are sometimes taking a very long time (>5hrs) to finish. Having test results show up in result store would be nice but since it is not a priority for these builds right now, we can disable BES upload when cross-compiling for macOS.
PiperOrigin-RevId: 608994569
The experimental cross-compile build fail sometimes because build event protocol upload for these builds is flaky and sometimes timeout. This is causing the job (even if `bazel build` ran successfully) to reported as failed. By changing it to full async, Bazel uploads BES asynchronously so it can exit and the upload still continues in the background.
PiperOrigin-RevId: 607416907
`rbe_cross_compile_macos_x86` expands `rbe_cross_compile_base` which in turn expands `rbe_base` which has `--jobs=800`. We avoid the jobs setting being overridden by setting it after `--config rbe_cross_compile_base`.
PiperOrigin-RevId: 603804012
This is an experimental job that only builds the tests and does not run them. The ML DevInfra team is using this to collect data on how these builds perform in production when compared to the regular continuous job (build + test) and to see if their runtime is fast enough to be enabled as presubmit (~20-25 mins)
"pycpp.sh" is modified such that it can be used either in build or test mode. Set `TFCI_PYCPP_SWAP_TO_BUILD_ENABLE` to 1 to enable build mode (default is test mode). Since test configs inherit from build, `linux_arm64_pycpp_test` and `linux_arm64_pycpp_test_filters` are changed to be prefixed with "build:" so that we can run both `bazel build` and `bazel test` commands with the same config.
PiperOrigin-RevId: 603543484
Even though we cross-compile with RBE, the tests are still run locally on the host Mac which require these increase timeout thresholds.
PiperOrigin-RevId: 599673837
We need to install Bazelisk and Pyenv manually as these are not present on the x86 Mac VMs. Note that the uploads from these new jobs are disabled as they are not yet ready. However, the old Mac x86 nightly builds will still be running and upload to tf-nightly so there won't be any missing nightly packages while we are doing this migration.
PiperOrigin-RevId: 587930871
This adds support for cross-compiling TensorFlow targets for Linux Aarch64 on a Linux x86 machine. We use Clang as the cross-compiler, `ld.lld` as the linker and build in a special cross-compile supported [Docker image](http://gcr.io/tensorflow-testing/ml-devinfra-linux-aarch64-cross-compile@sha256:de26c1dbddcb42b48e665972f62f128f2c69e0f1aa6f0ba6c7411dd23d4de785) that contains all the necessary build tools and the sysroots for both Linux Aarch64 and Linux x86. We need a Linux x86 toolchain because Bazel needs it to be able to build the tools used during the build—such as Protoc, llvm-tablegen, flatc—correctly for our execution platform (Linux x86).
In addition, this adds support for cross-compiling using RBE. We do this by invoking a Bazel remote build from a Linux Aarch64 host which would then send build requests to remote Linux x86 VMs. The Linux x86 VMs build inside the cross-compile Docker image using the cross-compile toolchain configs to build the targets for Aarch64. The targets, once built, are automatically transferred to the host seamlessly. RBE cross-compiling is necessary for us to be able to run `bazel test` commands. Note that lot of the "host_" flags such as "host_cpu" and "host_crosstool_top" flags seem to be actually used to specify the execution platform details. It seems it is this way because these flags are old and predate the distinction between host and execution platform.
The toolchain configs can be found in "tensorflow/tools/toolchains/cross_compile/cc/BUILD" and the RBE platform configs can be found in "tensorflow/tools/toolchains/cross_compile/config/BUILD".
If trying to cross-compile without RBE, run your build from a Linux x86 host in the [Docker image](http://gcr.io/tensorflow-testing/ml-devinfra-linux-aarch64-cross-compile@sha256:de26c1dbddcb42b48e665972f62f128f2c69e0f1aa6f0ba6c7411dd23d4de785)
and use `--config=cross_compile_linux_arm64`.
If you are trying to cross-compile with RBE, run your build from a Linux Aarch64 host and use `--conig=rbe_cross_compile_linux_arm64`. Since RBE uses GCP VM instances and requires authentication, it is only available to Googlers and TF CI builds. Tests can only be run with RBE.
PiperOrigin-RevId: 583062010
With the recent CUDA update we gained support for SM 9.0 compute capabilities (Hopper-based GPUs),
but so far we didn't enabled this by default in the nightly builds.
To keep file sizes in check we remove the build for SM 7.5. User's of the
Turing architecture (SM 7.5) will fall back to Volta (SM 7.0) which is compatible.
The main difference between Turing and Volta are TensorCores which TF doesn't target from custom CUDA kernels, hence we don't expect any peformance disadvantages
for Turing users.
It also brings us in line with what JAX is doing.
PiperOrigin-RevId: 576835080
