Commit Graph

234 Commits

Author SHA1 Message Date
Amit Patankar
1b00e4d951 Migrate the remote config toolchains from tensorflow to the toolchains repo.
PiperOrigin-RevId: 357813386
Change-Id: Idac837134f2cf8fb7390b96c2dc3aa93c6beabfb
2021-02-16 15:13:51 -08:00
A. Unique TensorFlower
897e0d868f Switch clang RBE build to latest image.
PiperOrigin-RevId: 351994732
Change-Id: I387cb5a583690748460461d633e0916a6e316066
2021-01-15 06:22:00 -08:00
A. Unique TensorFlower
556ad7c8a6 Use image with latest clang for RBE.
PiperOrigin-RevId: 351990199
Change-Id: Icf07511f5176a89f8be2747f67b0445df25569d8
2021-01-15 05:39:59 -08:00
Adrian Kuegel
6a848f27ab Create a new ROCM RBE Docker image with ROCM 4.0.
Update the hash accordingly.

PiperOrigin-RevId: 350711537
Change-Id: I8ef9a84d59831c377e867bb6c88dd4307953b365
2021-01-08 00:09:30 -08:00
Haiming Bao
04458594a8 Migrate the remote config toolchains from tensorflow to the toolchains repo.
PiperOrigin-RevId: 345158948
Change-Id: Id09fabbee85baaf2cfec6124457ddc1a13259f2e
2020-12-01 21:12:23 -08:00
Amit Patankar
0d4b7e2b08 Migrate the remote config toolchains from tensorflow to the toolchains repo.
PiperOrigin-RevId: 345065642
Change-Id: Id8ce10895bda0b2b76517f1873d69bbc56fe365b
2020-12-01 11:48:35 -08:00
Amit Patankar
4858553802 Migrate the remote config toolchains from tensorflow to the toolchains repo.
PiperOrigin-RevId: 342367364
Change-Id: I2c077b3c5d6b1d7ffd3dcd1eaa625af05094824a
2020-11-13 17:21:08 -08:00
Amit Patankar
8ef785452a Migrate the remote config toolchains from tensorflow to the toolchains repo.
PiperOrigin-RevId: 342283052
Change-Id: I75b84860268a7b941d304a5c97197638843be343
2020-11-13 10:14:48 -08:00
Adrian Kuegel
7ff73bb4b2 Create a new ROCM RBE Docker image with ROCM 3.9.
Update the hash accordingly.

PiperOrigin-RevId: 341636759
Change-Id: Id49d6889c2686fe038bcd67faacabb8e5a478928
2020-11-10 09:44:45 -08:00
Sanjoy Das
d4461a52e4 Update the SHA1 for RBE images
Without it, we're still using cuDNN 8.0.0

PiperOrigin-RevId: 336762900
Change-Id: I8b0aae871be1e2b9b8021704dc8e965e470c4ec5
2020-10-12 16:06:02 -07:00
Chris Kennelly
184da74b4f Remove ARCH_K8 from x86_64 toolchain configurations.
Nothing in Tensorflow depends on it being defined.

PiperOrigin-RevId: 335817300
Change-Id: I43e641fc1da087f9652ba6c59f9595a5b007e2a4
2020-10-07 01:48:24 -07:00
TensorFlower Gardener
0764c90855 Merge pull request #42689 from ROCmSoftwarePlatform:google_upstream_switch_to_rocm37
PiperOrigin-RevId: 333874384
Change-Id: Ic5e8b2394120d907e049f71cf5c00029ec75ad9f
2020-09-25 23:37:01 -07:00
Pankaj Kanwar
7fd83a73f0 update cuda paths for cuda11
PiperOrigin-RevId: 326022147
Change-Id: I7e0dff924c3d4ad5e3b50cdcd6551572ca93de93
2020-08-11 08:09:56 -07:00
Pankaj Kanwar
0e368fa399 Test build for testing cuda11 changes.
PiperOrigin-RevId: 325810725
Change-Id: Id018e4512c074fba2af19b8d8919c3dbb6c7768b
2020-08-10 08:46:02 -07:00
Christian Sigg
22def20bae New ROCm 3.5 RBE docker based on Ubuntu 18.04, re-enable RBE.
Fix list of cxx_builtin_include_directories. Only a few are needed, but those are more complicated (mix of symlinked and real paths).

Properly return error from crosstool wrapper.

PiperOrigin-RevId: 318788040
Change-Id: Ia66898e98a9a4d8fb479c7e75317f4114f6081e5
2020-06-29 05:23:57 -07:00
Christian Sigg
6a6071412a Remove @io_bazel_rules_docker dependencies, we no longer use them.
PiperOrigin-RevId: 318212634
Change-Id: Iada36cd6cf549b4c615be29b76803637de36d5de
2020-06-24 23:03:35 -07:00
Christian Sigg
18569dde74 Set up RBE builds with CUDA11, cuDNN8, TensorRT7.
PiperOrigin-RevId: 317692556
Change-Id: Ifd1e78cec6a6083e507e98baf6702401f18f67aa
2020-06-22 12:20:18 -07:00
Gunhan Gulsoy
f32ea19517 Change windows RBE to use a new toolchain image and its related configs.
PiperOrigin-RevId: 315781185
Change-Id: I11110fa5eb254640aa43118a7927134fb92feb25
2020-06-10 15:36:50 -07:00
Christian Sigg
0266394d50 Compress CUDA kernel binaries (CUBINs).
Impact:
 -32MB wheel size
-600MB _pywrap_tensorflow_internal.so
 -70MB memory during startup
+120ms startup time
PiperOrigin-RevId: 315419813
Change-Id: I2c39a88d95a4aa3a692560c8e3d78b125e8445c9
2020-06-08 22:20:31 -07:00
Gunhan Gulsoy
926cd57d1f Check in the bazel toolchain for the new windows RBE image
PiperOrigin-RevId: 315395076
Change-Id: I47ff27beaf67cb71c93f7238b8a0aee40bef3838
2020-06-08 18:31:54 -07:00
Christian Sigg
10c005554d Change RBE's default compute capability to compute_35,compute_60 (from compute_30,compute_60), because compute_30 is deprecated and will be removed in CUDA 11. We don't build releases for compute_30 and XLA requires compute_35.
PiperOrigin-RevId: 314495728
Change-Id: Idf5902adb8e12073bb372592a8da6d9b2289f950
2020-06-03 02:50:41 -07:00
Christian Sigg
83eb4048ba Update gcc7_manylinux2010-nvcc-cuda10 config, primarily for TF nightly and releases, but also Windows presubmits.
I think the preconfig hasn't been updated since cl/308598480. Most users have moved off of preconfigs, but some remain.

Manually, because the update.sh script is broken for me.

PiperOrigin-RevId: 314310108
Change-Id: I746ac0ac62f90bef6624eedf01c7be7a346a2dfd
2020-06-02 05:04:28 -07:00
Ran Chen
00868972a8 Try again add dill and tblib to RBE image
PiperOrigin-RevId: 314222042
Change-Id: Ib14707381951e857e09d469345d3d244d86ddfd8
2020-06-01 16:13:23 -07:00
Christian Sigg
edbe5e189c Support Ubuntu 18.04 as GPU RBE platform.
PiperOrigin-RevId: 314007642
Change-Id: Ic5a432718cf3419589b658fb7d7326ba1bbd71f0
2020-05-31 03:19:14 -07:00
Ran Chen
59cebfb711 Add dill and tblib to RBE image
This is needed for tf.distribute multi worker tests. Those tests use multi processes to simulate multiple workers, and need to serialize test arguments and results. pickle is too limited.

PiperOrigin-RevId: 313463711
Change-Id: Id01c997bf1b2f2cc98b90357759ecf1c2c0bd7c4
2020-05-27 14:33:02 -07:00
Terry Heo
ff99778343 Build pip_package with Bazel
Added new script build_pip_package_with_bazel.sh.
This scirpt can be used with ci_build.sh of TF for cross building.

ex) Build armhf Python3 based PIP
$ CI_DOCKER_EXTRA_PARAMS="-e CI_BUILD_PYTHON=python3 -e CROSSTOOL_PYTHON_INCLUDE_PATH=/usr/include/python3.5" \
  tensorflow/tools/ci_build/ci_build.sh PI-PYTHON3 \
  tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh armhf

PiperOrigin-RevId: 312420603
Change-Id: I5504fb22248e6a9d29560fa9216113a8705b7399
2020-05-19 22:19:00 -07:00
Scott Zhu
3200e57b9c Update the OSS image to pickup latest tf estimator PIP package.
PiperOrigin-RevId: 311220029
Change-Id: I1bbcdf92f410a074d6918cf61cfb380daa74339e
2020-05-12 16:03:19 -07:00
A. Unique TensorFlower
b730a73909 Update manylinux docker image to latest hash.
PiperOrigin-RevId: 311170200
Change-Id: Icebc7ba48fc5de8b9a638d39ecc87e6a47140e08
2020-05-12 11:53:46 -07:00
A. Unique TensorFlower
b908cb8a4e Refactor rbe_config.bzl to avoid calling platform / python config multiple times.
PiperOrigin-RevId: 309397530
Change-Id: I9bc3fa3cf79408693e2fc651b7300569b0881160
2020-05-01 06:29:04 -07:00
A. Unique TensorFlower
4f4503b7d9 Migrate our clang configuration to the new multi-python setup.
Instead of using --config=rbe_linux_cuda_clang --config=rbe_linux_py3, users
can now use --config=rbe_linux_cuda_clang_py36 (or any other supported version)
to build on RBE with the respective python version.

PiperOrigin-RevId: 309043827
Change-Id: Id0299a6ceca8e050eec63907f317c7a2c3058031
2020-04-29 10:07:10 -07:00
A. Unique TensorFlower
bb68770fe5 Add RBE configuration that allows building against multiple python versions.
This allows release builds to use RBE.

PiperOrigin-RevId: 309014910
Change-Id: Iec428478a61ec444647b9c4bac8bd8372fb305fb
2020-04-29 07:03:46 -07:00
A. Unique TensorFlower
9ca9a4f8e9 Roll forward with fix for Windows & PPC.
PiperOrigin-RevId: 308975869
Change-Id: I22abc77396e8d2a9c8f6a0907bf9d1d22169cc93
2020-04-29 00:43:38 -07:00
Terry Heo
b1c4cf2183 Fix RPI CI build failure
Update missing toolchain systempath to include python files correctly.
This regression happens by PR #38399.

Also skip OpenBLAS build for RPI 2/3 to improve build speed.

PiperOrigin-RevId: 308721738
Change-Id: I42a1ff1b104190ba3de7644f1af8d12997b7dd1e
2020-04-27 16:32:30 -07:00
Mihai Maruseac
40d3f089be Roll forward with fix for tf-serving.
PiperOrigin-RevId: 308282593
Change-Id: I947d78592a9ca5c481f98a95946bb4acbbb0437f
2020-04-24 10:48:45 -07:00
A. Unique TensorFlower
45e0468e02 Roll forward with fix for tf-serving.
PiperOrigin-RevId: 308011159
Change-Id: I91cffa1f10e8fcb621d18578225be7cc870b6839
2020-04-23 03:00:38 -07:00
Christina Sorokin
a659b93433 rollback of change
Add two repository rules:
- @local_execution_config_platform: local platform to allow selecting locally
  executed tools on
- @local_execution_config_python: python configured for execution...

PiperOrigin-RevId: 307862682
Change-Id: Ie0320f2f137a40b418632989981c9dc072ef80e6
2020-04-22 11:47:06 -07:00
A. Unique TensorFlower
f5bb643360 Add two repository rules:
- @local_execution_config_platform: local platform to allow selecting locally
  executed tools on
- @local_execution_config_python: python configured for execution on the local
  machine during otherwise remote builds

Mark rules that are required to run locally to require our local platform.

This allows pyth...

PiperOrigin-RevId: 307771596
Change-Id: If1f0013ec88a35d507b2b622894208aab2416fe5
2020-04-22 01:47:55 -07:00
Taehee Jeong
0f631f8fb1 Add two repository rules:
- @local_execution_config_platform: local platform to allow selecting locally
  executed tools on
- @local_execution_config_python: python configured for execution on the local
  machine during otherwise remote builds

Mark rules that are required to run locally to require our local platform.

This allows python paths to differ between the remote docker image and local
machine.

For example, the local machine might have python 3.7 installed in
/usr/bin/python3, while the remote docker should use a python installed
in /usr/local/bin/python3.8.

PiperOrigin-RevId: 307585019
Change-Id: I29313121beb967b77ae123e7d1b614c688cb40ca
2020-04-21 05:20:54 -07:00
A. Unique TensorFlower
d68284a16f Add two repository rules:
- @local_execution_config_platform: local platform to allow selecting locally
  executed tools on
- @local_execution_config_python: python configured for execution on the local
  machine during otherwise remote builds

Mark rules that are required to run locally to require our local platform.

This allows python paths to differ between the remote docker image and local
machine.

For example, the local machine might have python 3.7 installed in
/usr/bin/python3, while the remote docker should use a python installed
in /usr/local/bin/python3.8.

PiperOrigin-RevId: 307558811
Change-Id: I0dc2d877a7c26b294bf2b569b4f121cf6506e7fc
2020-04-21 01:10:05 -07:00
Terry Heo
4961f18733 Add embedded toolchains for cross-compiling
Added Bazel toolchain configuration for ARM's GNU toolchain.
- https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain
- GCC 8.3-2019.03

This configuration handles following CPU and build config_settings.
- armhf: linux_armhf
- aarch64: linux_aarch64

PiperOrigin-RevId: 306351085
Change-Id: I479d52e2243be9f75888ea81c7250e9072825da5
2020-04-13 18:27:07 -07:00
Lukas Geiger
b437c2bdfa Add aarch64 cross compile toolchain 2020-04-09 17:55:14 +01:00
A. Unique TensorFlower
f05de3fdee Use latest RBE docker image and latest clang release.
PiperOrigin-RevId: 302400255
Change-Id: I68afba780aeab479980b71cfa900afaa06ac98e6
2020-03-23 03:48:58 -07:00
A. Unique TensorFlower
765ddddb22 Add clang RBE configuration.
PiperOrigin-RevId: 301975133
Change-Id: I6bb5929d5def7669a81a360a34edda10c068aaae
2020-03-20 00:25:09 -07:00
A. Unique TensorFlower
91e9cf2156 Update RBE docker image to include a newer version of clang.
PiperOrigin-RevId: 301974757
Change-Id: Ic1dfb379fc66104f9efe9faf3c4022d40504ef1c
2020-03-20 00:20:59 -07:00
A. Unique TensorFlower
f920c18c0e Remove support for -Xgcc-only= options.
PiperOrigin-RevId: 300384182
Change-Id: I078a3fd571f0dfc994cdfdc6a5e940a837f2d625
2020-03-11 12:40:03 -07:00
Jakob Buchgraber
1d8b9fcff2 Delete unused preconfig for CUDA 10.1 and TensorRT 6.1
PiperOrigin-RevId: 298327898
Change-Id: I7aa4d4fc915e73ff0c573aed5989344cc2ba0765
2020-03-02 05:10:57 -08:00
Gunhan Gulsoy
e5eebbb5de Move windows 2019 RBE builds back to the default pool, and remove now obsolete
windows 1803 toolchains.

PiperOrigin-RevId: 297696691
Change-Id: I0b7ffc74afc415fd9c3b996cd29dcba5e7ca68d1
2020-02-27 15:06:51 -08:00
Gunhan Gulsoy
dde9bae64f Create a new windows 2019 toolchain, and on windows RBE use that.
Upgrade windows RBE settings to update worker base images.

PiperOrigin-RevId: 297622110
Change-Id: I34f54797d47ae8fd41bfcb8071108ab38cc9e8bb
2020-02-27 10:00:33 -08:00
Jakob Buchgraber
24d5cb4edc Fix bazelrc configuration that still uses preconfig instead of remote config.
PiperOrigin-RevId: 297390543
Change-Id: I8ac2456d6e41bbdc056d9ef8a5377f1e527cc74f
2020-02-26 10:22:50 -08:00
Jakob Buchgraber
e9c7aa7f1a Roll out remote config for Python 3.7 on Windows RBE
PiperOrigin-RevId: 296949118
Change-Id: I9881ad0c76cc1095955e643096143ecc206fccf6
2020-02-24 12:34:54 -08:00