This change makes most Bazel options apply to all commands (e.g., `build`, `test`, `query`, `mod`) rather than just the `build` command, which prevents external repo being refetched accidentally when switching between commands (e.g `build` vs `query`).
PiperOrigin-RevId: 812842964
Enable CUDA forward-compatibility mode in all RBE jobs by default.
Forward compatibility mode in hermetic CUDA allows the linker to use the user-mode driver from Bazel cache, so there is no need to install UMD in the RBE Docker image.
UMD on RBE machines is rarely updated, thus RBE jobs need forward compatibility mode to enable the most recent CUDA features usage in the tests.
The non-RBE job runners are updated more often, hence we can update the drivers on those machines and not rely on forward compatibility mode.
PiperOrigin-RevId: 810595379
`com_google_protobuf` and `rules_cc` initialization was moved to `python_init_rules` macros.
The JAX repository update will be done [in the next step](https://github.com/jax-ml/jax/pull/31709)
PiperOrigin-RevId: 808689003
If the repository env var `USE_MIRRORED_TAR_ARCHIVE_FILES` is set to `1`, the `.tar` files are downloaded from the mirror instead of `.tar.xz`.
`USE_MIRRORED_TAR_ARCHIVE_FILES` is not set by default.
PiperOrigin-RevId: 794631542
Upgrading manylinux compliancy tag in [JAX PR](https://github.com/jax-ml/jax/pull/29672) enabled building targets with linked `nvshmem` libraries.
PiperOrigin-RevId: 786533277
Hermetic toolchains give us builds that are isolated from the host system, cutting down on unexpected dependencies and side effects.
With these changes, TensorFlow will build for Linux x86_64 architectures (both CPU and CUDA-enabled GPU) using self-contained C++ toolchains. If you need to use a non-hermetic toolchain, you can do so by adding the flag --config=clang_local. For remote builds with a non-hermetic toolchain, simply append _clang_local to your existing RBE flag. For example, if your hermetic RBE build uses --config=rbe_linux_cpu, the non-hermetic version would be --config=rbe_linux_cpu_clang_local.
Example: Run CPU tests for Linux x86_64
For hermetic tests, run following command (no env variables like CC, CXX, BAZEL_COMPILER, CLANG_COMPILER_PATH):
bazel test \
--config=avx_linux \
--config=release_linux_base \
--config=linux_cpu_pycpp_test_filters \
--repo_env=HERMETIC_PYTHON_VERSION=3.11 \
//tensorflow/... -- -//tensorflow/compiler/tf2tensorrt/... -//tensorflow/core/tpu/... -//tensorflow/lite/... -//tensorflow/tools/toolchains/...
For Linux x86_64 non-hermetic tests use commands with the flag "--config=clang_local" and env variables CC, CXX, BAZEL_COMPILER, CLANG_COMPILER_PATH, etc.:
bazel test \
--config=clang_local \
--config=avx_linux \
--config=release_linux_base \
--config=linux_cpu_pycpp_test_filters \
--repo_env=HERMETIC_PYTHON_VERSION=3.11 \
--action_env=CLANG_COMPILER_PATH=/usr/lib/llvm-18/bin/clang \
--host_action_env=CLANG_COMPILER_PATH=/usr/lib/llvm-18/bin/clang \
--repo_env=CC=/usr/lib/llvm-18/bin/clang \
--repo_env=CXX=/usr/lib/llvm-18/bin/clang++ \
--repo_env=BAZEL_COMPILER=/usr/lib/llvm-18/bin/clang \
//tensorflow/... -- -//tensorflow/compiler/tf2tensorrt/... -//tensorflow/core/tpu/... -//tensorflow/lite/... -//tensorflow/tools/toolchains/...
PiperOrigin-RevId: 783911228
Bumps the aarch64-compatible oneDNN version to 3.7 and the ACL version
to 24.12. This brings better performance, improved memory management,
and numerous bug fixes over the previous, long out-of-date versions.
The previous version of this patch caused failures in a couple of unit
tests which have now been fixed.
Co-authored-by: David Svantesson-Yeung <david.svantesson-yeung@arm.com>
Signed-off-by: Siddhartha Menon <siddhartha.menon@arm.com>
`libtensorflow_framework.so.2` and `libtensorflow_cc.so.2` volumes are increased by 50 and 150MB with hermetic C++ toolchains.
PiperOrigin-RevId: 776186539
Hermetic C++ toolchains and CUDA are enabled for Linux x86_64 platform by default. List of covered OSs will be extended in a few closest months. Developers still could use non hermetic toolchains with help of --config=clang_local flag.
std::reduce replace with a traditional for loop. This is necessary because GCC 8 offers only partial support for C++17, and using std::reduce in this environment leads to "Undefined method" error.
PiperOrigin-RevId: 775771057
-Protobuf 5.28.3
-Grpc 1.68.2
-Abseil: LTS 20240116.3
-Plus some other transitive/related deps, riegeli and pybind11 in particular.
-rules_python & rules_cc will be updated in a subsequent CL as they are their own can of worms, plus there are a few pending changes in rules_python which were not pushed yet.
This also switches default protobuf implementation we rely on for bazel builds from cpp to upb, meaning all projects dependin on htis one must be built with build --action_env=PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb.
PiperOrigin-RevId: 773586210
TF wheel build rule implementation is also updated to exclude accidental dependencies on NVSHMEM libraries in the wheel content. If the wheel needs to be built with these dependencies, provide `--@local_config_nvshmem//:override_include_nvshmem_libs=True` in Bazel options.
`NVSHMEM` binaries are included in the dependencies if `CUDA` binary dependencies are added as well, e.g. `--@local_config_cuda//:enable_cuda`.
`NVSHMEM` libraries are included in the dependencies if `--@local_config_nvshmem//:include_nvshmem_libs=True` (the default flag value is `False`). Please note that this is a temporary solution, and it should be removed after GLIBC is updated on RBE runners. At the moment `libnvshmem.so` files can't be linked to the targets because they are built with GLIBC version higher then on RBE runners. In the future `--@local_config_cuda//cuda:include_cuda_libs=True` should be used.
The next change will contain adding `NVSHMEM` deps to individual Bazel targets via `select`.
PiperOrigin-RevId: 769344482
cl/755989315 should have only added the single flag to affect Python compiling,
but somehow it ended up reverting the edits from cl/755781948
Manually restore the edits made by cl/755781948
PiperOrigin-RevId: 761660001
When switching some remaining `native.py_` usages to the one
loaded from rules_python, some action conflicts occur. As
a short term solution, disable precompiling to avoid those
action conflicts.
PiperOrigin-RevId: 755989315
The extraction of `.tar` files is 10 times faster than the extraction of `.tar.xz` files. By enabling `.tar` files usage in RBE jobs we are going to save at least one min of execution time in all Bazel RBE GPU jobs.
PiperOrigin-RevId: 753278068
Bumps the aarch64-compatible oneDNN version to 3.7 and the ACL version
to 24.12. This brings better performance, improved memory management,
and numerous bug fixes over the previous, long out-of-date versions.
Signed-off-by: Siddhartha Menon <siddhartha.menon@arm.com>
This change is made to prevent hermetic CUDA repositories cache invalidation between the builds with `--config=cuda` and without it.
It should speed up Github presubmit jobs. Currently CPU and GPU jobs use the machines in the same pool, and they share the RBE cache.
Previously the cache was invalidated every time when `TF_NEED_CUDA` value changed between CPU and GPU builds, hence loading CUDA redistributions for GPU jobs took several minutes (see [this job](https://github.com/openxla/xla/actions/runs/14114621736/job/39541688832) for example: all the test results are cached, but CUDA redistributions were still downloaded).
With adding `--repo_env USE_CUDA_REDISTRIBUTIONS=1` to RBE CPU linux job configurations, we load some CUDA redistributions once in RBE cache, and then reuse it between the jobs.
PiperOrigin-RevId: 743700713
To use new rules the project needs to be built with the following parameters (flags are already into TF's .bazelrc)
--repo_env=USE_PYWRAP_RULES=True
--@com_google_protobuf//:use_dlls=True
--@com_google_absl//absl:use_dlls
Note: if after this CL, in one of your changes you see linker errors complaining about symbols missing on windows, you most likely need to modify _pywrap_tensorflow.def file:
- remove symbols which got deleted in your CL from _pywrap_tensorflow.def if they are mentioned there
- add new symbols into _pywrap_tensorflow.def if they start getting used in one of pybind_extension modules
To make sense out of mangled windows symbol names you may use https://demangler.com/ or `undname` (on windows) or any other simillar tool.
Automatic generation of _pywrap_tensorflow.def is possible but may be potentially slow, if there is evidence that _pywrap_tensorflow.def needs to get updated often, we'll add the automation.
PiperOrigin-RevId: 738203468
Mostly updating `WORKSPACE` and related files from `@tsl//third_party` -> `@xla//third_party`. Please tag me if this change breaks you.
PiperOrigin-RevId: 733489153
This change introduces a uniform way of building the TF wheel and controlling the filename version suffixes.
A new repository rule `python_wheel_version_suffix_repository` provides information about project and wheel version suffixes. The final value depends on environment variables passed to Bazel command: `_ML_WHEEL_WHEEL_TYPE, _ML_WHEEL_BUILD_DATE, _ML_WHEEL_GIT_HASH, _ML_WHEEL_VERSION_SUFFIX`
`tf_version.bzl` defines the TF project version and loads the version suffix information calculated by `python_wheel_version_suffix_repository`.
The targets `//tensorflow/core/public:release_version, //tensorflow:tensorflow_bzl //tensorflow/tools/pip_package:setup_py` use the version chunks defined above.
The version of the wheel in the build rule output depends on the environment variables.
Environment variables combinations for creating wheels with different versions:
* snapshot (default build rule behavior): `--repo_env=ML_WHEEL_TYPE=snapshot`
* release: `--repo_env=ML_WHEEL_TYPE=release`
* release candidate: `--repo_env=ML_WHEEL_TYPE=release --repo_env=ML_WHEEL_VERSION_SUFFIX=-rc1`
* nightly build with date as version suffix: `--repo_env=ML_WHEEL_TYPE=nightly --repo_env=ML_WHEEL_BUILD_DATE=<YYYYmmdd>`
* build with git data as version suffix: `--repo_env=ML_WHEEL_TYPE=custom --repo_env=ML_WHEEL_BUILD_DATE=$(git show -s --format=%as HEAD) --repo_env=ML_WHEEL_GIT_HASH=$(git rev-parse HEAD)`
PiperOrigin-RevId: 733444080
Due to 24ba4fa60c, --nobuild_runfile_links is no longer implied by --remote_download_minimal, which caused performance regression on Windows after upgrading Bazel to 7.4.1
PiperOrigin-RevId: 731694745
- Disabled Bzlmod for now before we start the migration
- Disabled cc toolchain resolution for now
- Fixed transition on `--modify_execution_info` due to https://github.com/bazelbuild/bazel/pull/16262
- Explicitly added `-Wl,-undefined,dynamic_lookup` as linkopt on macOS after https://github.com/bazelbuild/bazel/pull/16414
- Set `-force_no_whole_archive` for host features
- Addressed Windows linking issue by adding `@com_googlesource_code_re2//:__subpackages__` in `exports_filter` of `//tensorflow/python:pywrap_tensorflow_internal`
- Removed `license` attribute on cc_shared_library
- Fixed license checks
PiperOrigin-RevId: 731344894
