- Added MODULE.bazel and module extensions to introduce external dependencies with Bzlmod.
- Added a CI config for Linux CPU build with Bzlmod (enabled by `--config=bzlmod`)
TODOs:
- Support overriding Python runtime
- Support build with more configs and platforms
PiperOrigin-RevId: 817711851
The `@pypi_<name>` references are deprecated and their repo names are
an implementation detail of how an underlying library is downloaded.
The modern, supported, way is to go through the hub (`@pypi`). This
also makes the code compatible with both workspace and bzlmod.
PiperOrigin-RevId: 812915387
A custom runtime can be specified by setting the `HERMETIC_PYTHON_URL`, which
may point to a mostly arbitrary URL, i.e. one not following the same format
as python-build-standalone URLs, such e.g. `file:///__w/jax/jax/python-tsan.tgz`.
Internally, some rules_python code assumes the python-build-standalone URL
format and eventually errors.
To fix, the python version is used to determine the URL build suffix.
This also patches the rule_python versions URL generation logic to add musl
to the list of platforms it generates freethreaded runtime urls for.
Work around for https://github.com/bazel-contrib/rules_python/issues/3285, https://github.com/bazel-contrib/rules_python/issues/3286
PiperOrigin-RevId: 810132685
Some wheels like `jax-cuda12-plugin` have `-` symbol in the name, but `_` symbol in the filename (e.g. `jax_cuda12_plugin-0.8.0.dev0+selfbuilt-cp313-cp313t-manylinux_2_27_x86_64.whl`). Older `rules_python` implementation didn't make a difference whether the lock file had `jax-cuda12-plugin` or `jax_cuda12_plugin` records. `rules_python` v1.6 treats them differently: if there are records like below, then the local wheel doesn't override the one that should be downloaded from pypi.
```
jax-cuda12-plugin==0.7.2 ; sys_platform == "linux" and python_version < "3.14" \
--hash=sha256:05b6942985f015be82becd2cec363f0aceb25311981821d7613a51f630490e8c \
--hash=sha256:1d00f9f5c5f68ae0f41cb7b589005ed5cb556517d65bbab5a891be46ed7a781c \
--hash=sha256:23b8f1050c48b4020610fb818930d3cbe0304c6681b069687e5416ee349bd734 \
--hash=sha256:2a727a89ae69ac21c1f5093d8d5aef89a0e692e66b034fc934c8accc72e40290 \
--hash=sha256:45d5a1cbf0b9d05318722382fc71c4cede0c028bad6aa8e53f7a7032392f719c \
--hash=sha256:5e3e2aa4d721fb02dd1028262aaeaec2958e45bca5c4d3512b29151b570cb425 \
--hash=sha256:7212c12d75b7dc51275f271827df4a6d378430c06f650e6c31c162fe9579ff12 \
--hash=sha256:7ad3afc51bcbc4e8117845d359e5d02cbc5ca2b152efdebd3c55fb9e4c2f848e \
--hash=sha256:8284e7cf7f544906604f111702a6f0011a96df7f0113878b381bec0905172536 \
--hash=sha256:98a975655382858d874d6471ce97194310609d0a2a7c4283c6e07e37933b7768 \
--hash=sha256:adc924ebc7a45c8d3400ea0118dc70a7082b2a86e35711738d403dd3815d09bf \
--hash=sha256:e881b56fe27e6870db2f2e9c574b81965fe1102b1532eae60e240a40c065daf5
# via -r build/requirements.in
jax_cuda12_plugin @ file:///jax_cuda12_plugin-0.8.0.dev0+selfbuilt-cp311-cp311-manylinux_2_27_x86_64.whl
```
To fix this, `python_repo()` implementation was changed to add the local wheels with exactly the same module names as in the original lock file.
PiperOrigin-RevId: 809147267
The problem was that `@python_<>_host` repository wasn't aware of the python type (controlled by the build flag `@rules_python//python/config_settings:py_freethreaded`) and selected the first candidate, e.g. GIL version of the python.
PiperOrigin-RevId: 808825370
`com_google_protobuf` and `rules_cc` initialization was moved to `python_init_rules` macros.
The JAX repository update will be done [in the next step](https://github.com/jax-ml/jax/pull/31709)
PiperOrigin-RevId: 808689003
This change is made to have consistency in parsing NVIDIA dependencies with [JAX repository](https://github.com/jax-ml/jax/pull/30706)
`nvidia-requirements.txt` is used in Bazel hermetic python lock files and in the `tools/pip_package/setup.py` with the package requirements.
The file content is saved in the `nvidia_wheel_versions repository` and passed as an argument to `modify_setup_py.py` that populates `setup_py.tpl` script.
PiperOrigin-RevId: 802755816
This allows downstream projects to apply project-specific patches without
having to redefine the entire rules_python repo and the custom patches.
PiperOrigin-RevId: 792220068
- grpc to 1.72.2
- pybind11 to 2.13.6
- pybind11_abseil to commit 13d4f99d5309df3d5afa80fe2ae332d7a2a64c6b
- pybind11_bazel to 2.13.6
- bazel-skylib to 1.7.1
PiperOrigin-RevId: 786616601
Instead, the macros and underlying rule copies all the files belonging to `:pkg` target into the output dir.
Action wall time from JSON profiler tracing results:
Action | Old implementation | New implementation
:------: | :----------------: | :----------------:
jaxlib/tools/jax_cuda_pjrt_py_import_unpacked_wheel | 73,309.311 ms | 6,249.759 ms
jax_py_import_unpacked_wheel | 631.268 ms | 1,442.022 ms
jaxlib/tools/jaxlib_py_import_unpacked_wheel | 8,807.849 ms | 3,942.539 ms
jaxlib/tools/jax_cuda_plugin_py_import_unpacked_wheel | 7,873.122 ms | 7,968.307 ms
PiperOrigin-RevId: 777741244
Forked from https://github.com/openxla/xla/pull/25183
Without this change rules_python doesn't correctly install free-threaded wheels for a free-threaded Python build.
PiperOrigin-RevId: 749128015
Imported from GitHub PR https://github.com/openxla/xla/pull/24255
Returning from finally clause raises SyntaxWarning in python 3.14
cc @hawkinsp
Copybara import of the project:
--
2f7a929473cc67b3498ed88df4ab8e080e7def0f by vfdev-5 <vfdev.5@gmail.com>:
Fixed rules_python patch for python 3.14
Returning from finally clause raises SyntaxWarning in python 3.14
Merging this change closes#24255
PiperOrigin-RevId: 741203333
Mostly updating `WORKSPACE` and related files from `@tsl//third_party` -> `@xla//third_party`. Please tag me if this change breaks you.
PiperOrigin-RevId: 733489153
The include paths for headers within the ml_dtypes package have changed.
We therefore need to adjust the TF/XLA build rules and paths to account
for this. Also updated the pip ml_dtypes version to match.
The main ml_dtypes repo name needed to be changed to avoid
conflicts with the ml_dtypes subfolder. The subfolder contains the main
python package that needs to be added to the PYTHON_PATH.
PiperOrigin-RevId: 723654395
Also add mxfloat as a dependency to TensorFlow and TSL. This is needed to merge https://github.com/openxla/xla/pull/19096. Previously this was done in the merge commit for that PR, but the PR was rolled back since the new types caused an internal TF Android build to fail. Now it's being done in this separate, smaller change so its easier to rollback if issues occur.
PiperOrigin-RevId: 714285735
Example of usage:
```
load(
"@tsl//:third_party/py/py_manylinux_compliance_test.bzl",
"verify_manylinux_compliance_test",
)
verify_manylinux_compliance_test(
name = "manylinux_compliance_test",
aarch64_compliance_tag = "manylinux_2_17_aarch64",
test_tags = [
"mac_excluded",
"windows_excluded",
],
wheel = ":wheel",
x86_64_compliance_tag = "manylinux_2_17_x86_64",
)
```
The test target is executed only when specified in Bazel command line. The test passes if `auditwheel show` results have the compliance tag value (depends on the machine type). The test fails otherwise and prints the `auditwheel show` results.
PiperOrigin-RevId: 708024471
Imported from GitHub PR https://github.com/openxla/xla/pull/19346
cc @hawkinsp
Copybara import of the project:
--
292e7ebb7ee57e5af5977c08f0aaf28fc1f852e2 by vfdev-5 <vfdev.5@gmail.com>:
Bumped rules_python version to 0.39.0
Merging this change closes#19346
PiperOrigin-RevId: 699100796
The compliance check is disabled by default, but can be turned on by `--@local_tsl//third_party/py:wheel_compliance=true` option.
PiperOrigin-RevId: 695870736
The gist of this change (the new rules implementation) is contained within `rules_pywrap` folder. The rules are generic and not tensorflow-specific
1) (internal-specific)
2) (internal-specific)
3) It provides same linking strategy of final artifacts on all 3 supported platforms (no major differences between Linux, Mac and Windows).
4) It makes it possible to abandon usage of header-only targets to prevent ODR violations. Simply speaking you can now depend on generated protobuf message classes normally, without need to worry how that is linked afterwards.
5) The current version is backward-compatible and unless explicitly enabled is a no-op. To enable the new rules pass `--repo_env=USE_PYWRAP_RULES=True` flag to build/test command.
6) The `if_pywrap` construct is temporary and will be removed once full migration is completed. Currently if_pywrap is mainly used to pass normal dependencies (instead of header-only). The header-only stuff is kept for backward compatibility and smoother migration but will be eventually removed.
7) This CL migrates TF and the most problematic among all google ML repositories. Once TF is sabilized the other repositories, such as JAX and XLA will be migrated too (which should be way easier than migrating TF anyways)
PiperOrigin-RevId: 684324990
This test verifies whether the API v2 packages can be imported from the
current build. It utilizes the `_api/v2/api_packages.txt` list of packages from
the local wheel file specified in the `requirements_lock_<python_version>.txt`.
The test should be executed after the TF wheel was built and put into `dist` dir inside Tensorflow repository.
PiperOrigin-RevId: 676893008
ml_dtypes Updates:
Add float8_e4m3 and float8_e3m4 types support
Fix float divmod with zero denominator
Add int2 and uint2 types
ml_dtypes/commits
Related PRs
ml_dtypes PR Add float8_e4m3 jax-ml/ml_dtypes#161 Add float8_e4m3 (Merged)
XLA PR Add support for float8_e4m3 #16585 (In Review)
This closes https://github.com/openxla/xla/pull/17075
PiperOrigin-RevId: 675687080
TODO: once rules_python has 3.13 out of the box, the patch can be safely removed.
rules_python does not depend on rc versions of Python (seems to be against its philosophy), while JAX wants to support it, that is why adding RC support in a form of a patch instead of adding it directly to rules_python.
PiperOrigin-RevId: 675261914
