This change is made to have consistency in parsing NVIDIA dependencies with [JAX repository](https://github.com/jax-ml/jax/pull/30706)
`nvidia-requirements.txt` is used in Bazel hermetic python lock files and in the `tools/pip_package/setup.py` with the package requirements.
The file content is saved in the `nvidia_wheel_versions repository` and passed as an argument to `modify_setup_py.py` that populates `setup_py.tpl` script.
PiperOrigin-RevId: 802755816
Upgrading manylinux compliancy tag in [JAX PR](https://github.com/jax-ml/jax/pull/29672) enabled building targets with linked `nvshmem` libraries.
PiperOrigin-RevId: 786533277
-Protobuf 5.28.3
-Grpc 1.68.2
-Abseil: LTS 20240116.3
-Plus some other transitive/related deps, riegeli and pybind11 in particular.
-rules_python & rules_cc will be updated in a subsequent CL as they are their own can of worms, plus there are a few pending changes in rules_python which were not pushed yet.
This also switches default protobuf implementation we rely on for bazel builds from cpp to upb, meaning all projects dependin on htis one must be built with build --action_env=PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb.
PiperOrigin-RevId: 773586210
This only allows for zstd decompression of data produced by other tools, and
does so directly inline. It would be nice to add full zstd support for reading
and writing streams of data like
https://github.com/tensorflow/tensorflow/pull/58871 did, but DecodeCompressedOp
only needs a straightforward decompress implementation (even without streaming!)
since the end result is going to be an in-memory tensor anyway.
PiperOrigin-RevId: 731749205
The include paths for headers within the ml_dtypes package have changed.
We therefore need to adjust the TF/XLA build rules and paths to account
for this. Also updated the pip ml_dtypes version to match.
The main ml_dtypes repo name needed to be changed to avoid
conflicts with the ml_dtypes subfolder. The subfolder contains the main
python package that needs to be added to the PYTHON_PATH.
PiperOrigin-RevId: 723654395
Usage example: provide NVIDIA wheel dependencies for ML wheels that have rpaths pointing to NVIDIA folders. When a user executes `pip install tensorflow[and_cuda]`, NVIDIA wheels are installed together with Tensorflow wheel. To reproduce this behavior in hermetic Python approach, we need to define `py_import` as follows (provided NVIDIA dependencies are defined in `requirements.in` and requirements lock files):
py_import(
name = "tf_py_import",
wheel = ":wheel",
deps = [
"@pypi_absl_py//:pkg",
"@pypi_astunparse//:pkg",
"@pypi_flatbuffers//:pkg",
"@pypi_gast//:pkg",
"@pypi_ml_dtypes//:pkg",
"@pypi_numpy//:pkg",
"@pypi_opt_einsum//:pkg",
"@pypi_packaging//:pkg",
"@pypi_protobuf//:pkg",
"@pypi_requests//:pkg",
"@pypi_termcolor//:pkg",
"@pypi_typing_extensions//:pkg",
"@pypi_wrapt//:pkg",
],
wheel_deps = [
"@pypi_nvidia_cublas_cu12//:whl",
"@pypi_nvidia_cuda_cupti_cu12//:whl",
"@pypi_nvidia_cuda_nvcc_cu12//:whl",
"@pypi_nvidia_cuda_nvrtc_cu12//:whl",
"@pypi_nvidia_cuda_runtime_cu12//:whl",
"@pypi_nvidia_cudnn_cu12//:whl",
"@pypi_nvidia_cufft_cu12//:whl",
"@pypi_nvidia_curand_cu12//:whl",
"@pypi_nvidia_cusolver_cu12//:whl",
"@pypi_nvidia_cusparse_cu12//:whl",
"@pypi_nvidia_nccl_cu12//:whl",
"@pypi_nvidia_nvjitlink_cu12//:whl",
],
)
PiperOrigin-RevId: 705666137
The compliance check is disabled by default, but can be turned on by `--@local_tsl//third_party/py:wheel_compliance=true` option.
PiperOrigin-RevId: 695870736
This test verifies whether the API v2 packages can be imported from the
current build. It utilizes the `_api/v2/api_packages.txt` list of packages from
the local wheel file specified in the `requirements_lock_<python_version>.txt`.
The test should be executed after the TF wheel was built and put into `dist` dir inside Tensorflow repository.
PiperOrigin-RevId: 676893008
To ensure compatibility with numpy>=2.0.0, the following dependencies have been upgraded as well: ml-dtype>=0.4.0, h5py>=3.11.0, and scipy>=1.13.0.
Maintain lock files and an env file for follow up numpy1 jobs as planned.
PiperOrigin-RevId: 676114036
