1. Rewrote hipcc_cc_toolchain_config.bzl.tpl.oss based on third_party/bazel/tools/cpp/unix_cc_toolchain_config.bzl
2. Cleaned up non-Linux stuff in toolchain configuration
3. Added support for parameter file in the compiler wrapper script
4. Re-generated preconfigured toolchain by third_party/tensorflow/third_party/toolchains/preconfig/generate/update.sh ubuntu16.04-py3_opt-gcc5-rocm
5. Bumped min Bazel version to 0.27.1 because toolchain configure requires newer Bazel
6. Removed --noincompatible_do_not_split_linking_cmdline
PiperOrigin-RevId: 278844463
Change-Id: I477ec5b44e6c634db7c6d65d02b3e307f0be338b
This may be valuable to enable globally (and disable via a user flag), but for now I've enabled it for one specific call which was causing an error in #32776.
Resolves#32776.
PiperOrigin-RevId: 276094797
Change-Id: I6f007a50311c00885cf0db2d291f4d00294d53d7
Besides upgrading the Bazel version, we also refactored all build scripts to use rbe options in .bazelrc file.
In order to migrate for https://github.com/bazelbuild/bazel/issues/7480, we have to specify the complete strategies list in .bazelrc file.
PiperOrigin-RevId: 275459466
Change-Id: Iaec997da7862245955a36ebb1018d901f61c591d
We always use RTLD_LOCAL for dynamic linking TensorRT in open source so the
symbol conflicts should not affect OSS builds.
PiperOrigin-RevId: 268983417
The blacklist was working around an ODR violation problem that is triggered by
linking in TensorRT and cuDNN into the same binary.
In open source TensorRT is off by default and this CL adds an admonition about
the possible problem.
PiperOrigin-RevId: 268942282
This check was added because libopenblas on Power had threading
issues if OMP_NUM_THREADS was set higher than 1. This has been
fixed in all the latest versions of libopenblas and this check
is no longer needed.
HDFS is now supported through bazel build config and not through a flag in this file (`configure.py`).
However, the docstrings still refer to it. Replaced it with `TF_NEED_CUDA`, which is still a flag set in this script
Use a single python script (third_party/gpus/find_cuda_config.py) from configure.py and the different *_configure.bzl scripts to find the different CUDA library and header paths based on a set of environment variables.
PiperOrigin-RevId: 243669844
Allow setting the level via $ANDROID_NDK_API_LEVEL, rather than
hardcoding to the current NDK version. Also increase the default
NDK API level to 18.
PiperOrigin-RevId: 241981236
Right now, "0.24" is treated as lower than "*.*.*" because of the odd comparison method that adds digits to each existing section. This change converts "0.24" to "0.24.0" to fix that.
This will probably need to be pulled into r2.0.
PiperOrigin-RevId: 241950768
Imported from GitHub PR #26752
This makes builds not fail with --incompatible_remove_old_python_version_api enabled, which will become the default in Bazel 0.25. Fixes#26616.
Copybara import of the project:
- a3b9e5cde69beb5b1ceebe9a4a20c352bf320290 Bump abseil-py dep and min Bazel version by Jon Brandvein <brandjon@google.com>
- d189be43b2bf4bcd229d4fe23bc95870a196ff79 Merge a3b9e5cde69beb5b1ceebe9a4a20c352bf320290 into 9ab5b... by Jon Brandvein <brandjon@google.com>
PiperOrigin-RevId: 240409003
ERROR: cc_toolchain_suite '@local_config_cuda//crosstool:toolchain' does not contain a toolchain for CPU 'k8', you may want to add an entry for 'local|compiler' into toolchains and toolchain_identifier 'local_linux' into the corresponding cc_toolchain rule (see --incompatible_disable_cc_toolchain_label_from_crosstool_proto).
PiperOrigin-RevId: 225430355
Bazel-0.18.0 adds a try-import option that will non-fatally try and
import a file. Use this for the configure options so that .bazelrc does
not need to change. ./configure rewriting .bazelrc makes using the git
repo annoying because the file is changed.
The allowed bazel range is now 0.18.0-0.20.0 inclusive. The env var
TF_IGNORE_MAX_BAZEL_VERSION can be set to skip the max bazel version
check.
Also optionally import a /.bazelrc.user file that is gitignored so
user-specific options can go in there.
Fixes: https://github.com/tensorflow/tensorflow/issues/22762
Fixes: https://github.com/tensorflow/tensorflow/pull/22906
Signed-off-by: Jason Zaman <jason@perfinion.com>
After a few fixes, TensorFlow now builds with Bazel 0.20.0. This allows us to keep testing TF on Bazel CI, where bazel is already upgraded to 0.20.0
PiperOrigin-RevId: 224006905
When the user choosen python_lib_path was retreived from the
PYTHONPATH environment variable, need to set PYTHONPATH in the
bazelrc file so bazel includes it in all the build operations.
Fixes tensorflow#23695
Note to users manually patching ptxas from a later toolkit version:
Building NCCL requires the same version of ptxas and nvlink.
PiperOrigin-RevId: 215911973
This allows repositories that depend on TensorFlow to execute
'bazel run @org_tensorflow//:configure -- --workspace $(pwd)'
to configure TensorFlow.
END_PUBLIC
Before this change, the .tf_configure.bazelrc ended up in the bazel exec root, and 'bazel clean' would undo the configuration.
PiperOrigin-RevId: 215209207
Building binaries with XLA support does not enable it by default, it
simply makes it accessible via default binary builds.
PiperOrigin-RevId: 214942824
/.bazelrc is not gitignored anymore so this should help in case the
import line is accidentally committed. Bazel 0.18.0 will support a new
'try-import' statement that should be used once 0.18.0 has been out long
enough.
Signed-off-by: Jason Zaman <jason@perfinion.com>
This change reduce the size of //tensorflow/tools/pip_package:simple_console_windows's zip file from 1000027677 bytes to 47690474 bytes for a CPU build. For GPU build, it will avoid going over 4GB when multiple CUDA compatibility are specified.
To fix#22390
PiperOrigin-RevId: 214764423
Allow TF_SYSTEM_LIBS env var to split on spaces or commas.
Write PREFIX and related variables to bazelrc as well.
Signed-off-by: Jason Zaman <jason@perfinion.com>
The commit contains following components to support TensorFlow on ROCm platform
- bazel build system
- continuous integration logic
Authors:
- Jack Chung: jack.chung@amd.com
- Jeffrey Poznanovic: Jeffrey.Poznanovic@amd.com
- Peng Sun: Peng.Sun@amd.com
This improves build times when using the downloaded clang toolchain.
Additionally, remove '-B/usr/bin' flags from the cuda CROSSTOOL when using
the downloaded toolchain.
It was forcing 'clang' to first search for the linker in '/usr/bin',
preventing downloaded LLD from being selected.
PiperOrigin-RevId: 211430374
Please approve this CL. It will be submitted automatically, and its GitHub pull request will be marked as merged.
Imported from GitHub PR #21373
A new version of nsync is released, this updates the workspace.bzl and adds nsync to the libs that can be unbundled. I've packaged nsync on Gentoo Linux and tested the full unbundled build.
The CMake build is also updated to the latest version, the patch is no longer required and is removed. I have tested the cmake build on my machine but the CI needs to test windows since I can't.
The last commits update ./configure to handle adding TF_SYSTEM_LIBS and an option to disable stripping.
@m3bm3b, can you review too?
Copybara import of the project:
- 3d224b29fcf699b84cf538e2d16e3dd4e78e6a91 workspace: update nsync dep to 1.20.1 by Jason Zaman <jason@perfinion.com>
- 34536b88bedc5a83ad939438a49a3cc53aea210c CMake: nsync: update to 1.20.1 by Jason Zaman <jason@perfinion.com>
- f2444ba218ae280e03710f936b249b6b650ea3dd third_party: unbundle nsync by Jason Zaman <jason@perfinion.com>
- 4386d8c4b01020ba1c979258e5102ac1947d2574 configure: Add TF_SYSTEM_LIBS handling by Jason Zaman <jason@perfinion.com>
- f3fd1f110781cfb8cc941241a69eb41114787bfd Merge 4386d8c4b01020ba1c979258e5102ac1947d2574 into a05c7... by Jason Zaman <jasonzaman@gmail.com>
COPYBARA_INTEGRATE_REVIEW=https://github.com/tensorflow/tensorflow/pull/21373 from perfinion:nsync 4386d8c4b01020ba1c979258e5102ac1947d2574
PiperOrigin-RevId: 209201545
* Added nGraph bridge as a third_party to be built with TensorFlow based on user selection.
* Added a limited set of C++ unit tests to verify the correctness of the computation
I was using bazel 0.12.0 and was getting the error:
file '@bazel_tools//tools/cpp:windows_cc_configure.bzl' does not contain symbol 'setup_vc_env_vars'
PiperOrigin-RevId: 205316270
This simplifies ./configure process on Windows, we can remove them when we actually support the corresponding feature on Windows.
PiperOrigin-RevId: 204804112
After this change, to build TensorFlow with GPU support on Windows, you just need to follow the same procedure as Linux.
Also re-enable remote cache since the bug in b/111106393 has been fixed by the new wrapper script.
The clean-up of the wrapper scripts also made the build around 5 mins faster than using Bazel's msvc wrapper script.
PiperOrigin-RevId: 204105368
* Add KinesisDataset support for tensorflow Dataset
This fix is an attempt to add Kinesis support
for tensorflow's Dataset. Kinesis is provided by
AWS as a managed data streaming service. It is
similiar to Apache Kafka, often used in places
where maintaining a independent Kafka cluster on AWS
is not desirable or possible.
This fix adds the Kinesis support for tensorflow Dataset.
Similiar to the Kafka integration in tensorflow,
KinesisDataset outputs tf.string for records.
Test cases have also been added, which could be invoked manually.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Expose KinesisDataset in dataset_ops.cc
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Expose KinesisDataset in python wrapper
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Add test cases for KinesisDataset
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Update AWS library include files
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Add Bazel BUILD files
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Rename s3_crypto to aws_crypto
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Rename with_s3_support to with_aws_support
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Selectively add kinesis to tensorflow/contrib/BUILD
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Set different partition key and pylint fix.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Add missing modules in cmake's python_modules.txt
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Address review feedback
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
These rules currently get written by configure.py script to WORKSPACE
file which is not ideal since (1) WORKSPACE file is tracked by git and
(2) we require users to manually delete the rules in order to
update/regenerate them.
Moving these rules into an external repo that is generated based on
several ENV variables set by the configure.py script. Modifying any
of these ENV variables will cause the rules to be updated.
PiperOrigin-RevId: 199388460
Revert #18413. Too many internal test failures due to the name scope change caused by this change.
Revert #18192. Cannot use re2::StringPiece internally. Need alternative for set call. Will pull and clean this up in a separate change.
PiperOrigin-RevId: 197991247
Remove all whitespace characters from the user specified tf_cuda_compute_capabilities string as this can results in errors during the split operation, and is easy for users to do as it is natural to insert a space after a comma
Since py_runtime was introduced, Bazel ignores options such as
--force_python2 and --python2_path. Deleting to clean stuff up and
make sure people are not misled.
PiperOrigin-RevId: 193127681
The configure.py script will error out on Windows GPU builds due
to NCCL attempted to be configured (and is currently Linux only).
PiperOrigin-RevId: 192461362
The nccl_configure.bzl generates two different BUILD files based on the chose NCCL version. For NCCL 1, it aliases to the existing 'nccl_archive' http_repo on GitHub. For NCCL 2, it creates a target containing the NCCL 2 library and headers from the chosen install directory.
PiperOrigin-RevId: 191718007
Previously we only allowed to download clang when doing GPU builds.
The added skylark files use bazel's autoconf scripts, which were only added in
0.10.0. To provide nice error message for older versions of bazel
(i.e. 'version is less than 0.10' vs 'can't load @bazel_tools/cpp/...'), we
move the bazel version check into WORKSPACE file from workspace.bzl.
PiperOrigin-RevId: 190050798
