HDFS is now supported through bazel build config and not through a flag in this file (`configure.py`).
However, the docstrings still refer to it. Replaced it with `TF_NEED_CUDA`, which is still a flag set in this script
Use a single python script (third_party/gpus/find_cuda_config.py) from configure.py and the different *_configure.bzl scripts to find the different CUDA library and header paths based on a set of environment variables.
PiperOrigin-RevId: 243669844
Allow setting the level via $ANDROID_NDK_API_LEVEL, rather than
hardcoding to the current NDK version. Also increase the default
NDK API level to 18.
PiperOrigin-RevId: 241981236
Right now, "0.24" is treated as lower than "*.*.*" because of the odd comparison method that adds digits to each existing section. This change converts "0.24" to "0.24.0" to fix that.
This will probably need to be pulled into r2.0.
PiperOrigin-RevId: 241950768
Imported from GitHub PR #26752
This makes builds not fail with --incompatible_remove_old_python_version_api enabled, which will become the default in Bazel 0.25. Fixes#26616.
Copybara import of the project:
- a3b9e5cde69beb5b1ceebe9a4a20c352bf320290 Bump abseil-py dep and min Bazel version by Jon Brandvein <brandjon@google.com>
- d189be43b2bf4bcd229d4fe23bc95870a196ff79 Merge a3b9e5cde69beb5b1ceebe9a4a20c352bf320290 into 9ab5b... by Jon Brandvein <brandjon@google.com>
PiperOrigin-RevId: 240409003
ERROR: cc_toolchain_suite '@local_config_cuda//crosstool:toolchain' does not contain a toolchain for CPU 'k8', you may want to add an entry for 'local|compiler' into toolchains and toolchain_identifier 'local_linux' into the corresponding cc_toolchain rule (see --incompatible_disable_cc_toolchain_label_from_crosstool_proto).
PiperOrigin-RevId: 225430355
Bazel-0.18.0 adds a try-import option that will non-fatally try and
import a file. Use this for the configure options so that .bazelrc does
not need to change. ./configure rewriting .bazelrc makes using the git
repo annoying because the file is changed.
The allowed bazel range is now 0.18.0-0.20.0 inclusive. The env var
TF_IGNORE_MAX_BAZEL_VERSION can be set to skip the max bazel version
check.
Also optionally import a /.bazelrc.user file that is gitignored so
user-specific options can go in there.
Fixes: https://github.com/tensorflow/tensorflow/issues/22762
Fixes: https://github.com/tensorflow/tensorflow/pull/22906
Signed-off-by: Jason Zaman <jason@perfinion.com>
After a few fixes, TensorFlow now builds with Bazel 0.20.0. This allows us to keep testing TF on Bazel CI, where bazel is already upgraded to 0.20.0
PiperOrigin-RevId: 224006905
When the user choosen python_lib_path was retreived from the
PYTHONPATH environment variable, need to set PYTHONPATH in the
bazelrc file so bazel includes it in all the build operations.
Fixes tensorflow#23695
Note to users manually patching ptxas from a later toolkit version:
Building NCCL requires the same version of ptxas and nvlink.
PiperOrigin-RevId: 215911973
This allows repositories that depend on TensorFlow to execute
'bazel run @org_tensorflow//:configure -- --workspace $(pwd)'
to configure TensorFlow.
END_PUBLIC
Before this change, the .tf_configure.bazelrc ended up in the bazel exec root, and 'bazel clean' would undo the configuration.
PiperOrigin-RevId: 215209207
Building binaries with XLA support does not enable it by default, it
simply makes it accessible via default binary builds.
PiperOrigin-RevId: 214942824
/.bazelrc is not gitignored anymore so this should help in case the
import line is accidentally committed. Bazel 0.18.0 will support a new
'try-import' statement that should be used once 0.18.0 has been out long
enough.
Signed-off-by: Jason Zaman <jason@perfinion.com>
This change reduce the size of //tensorflow/tools/pip_package:simple_console_windows's zip file from 1000027677 bytes to 47690474 bytes for a CPU build. For GPU build, it will avoid going over 4GB when multiple CUDA compatibility are specified.
To fix#22390
PiperOrigin-RevId: 214764423
Allow TF_SYSTEM_LIBS env var to split on spaces or commas.
Write PREFIX and related variables to bazelrc as well.
Signed-off-by: Jason Zaman <jason@perfinion.com>
The commit contains following components to support TensorFlow on ROCm platform
- bazel build system
- continuous integration logic
Authors:
- Jack Chung: jack.chung@amd.com
- Jeffrey Poznanovic: Jeffrey.Poznanovic@amd.com
- Peng Sun: Peng.Sun@amd.com
This improves build times when using the downloaded clang toolchain.
Additionally, remove '-B/usr/bin' flags from the cuda CROSSTOOL when using
the downloaded toolchain.
It was forcing 'clang' to first search for the linker in '/usr/bin',
preventing downloaded LLD from being selected.
PiperOrigin-RevId: 211430374
Please approve this CL. It will be submitted automatically, and its GitHub pull request will be marked as merged.
Imported from GitHub PR #21373
A new version of nsync is released, this updates the workspace.bzl and adds nsync to the libs that can be unbundled. I've packaged nsync on Gentoo Linux and tested the full unbundled build.
The CMake build is also updated to the latest version, the patch is no longer required and is removed. I have tested the cmake build on my machine but the CI needs to test windows since I can't.
The last commits update ./configure to handle adding TF_SYSTEM_LIBS and an option to disable stripping.
@m3bm3b, can you review too?
Copybara import of the project:
- 3d224b29fcf699b84cf538e2d16e3dd4e78e6a91 workspace: update nsync dep to 1.20.1 by Jason Zaman <jason@perfinion.com>
- 34536b88bedc5a83ad939438a49a3cc53aea210c CMake: nsync: update to 1.20.1 by Jason Zaman <jason@perfinion.com>
- f2444ba218ae280e03710f936b249b6b650ea3dd third_party: unbundle nsync by Jason Zaman <jason@perfinion.com>
- 4386d8c4b01020ba1c979258e5102ac1947d2574 configure: Add TF_SYSTEM_LIBS handling by Jason Zaman <jason@perfinion.com>
- f3fd1f110781cfb8cc941241a69eb41114787bfd Merge 4386d8c4b01020ba1c979258e5102ac1947d2574 into a05c7... by Jason Zaman <jasonzaman@gmail.com>
COPYBARA_INTEGRATE_REVIEW=https://github.com/tensorflow/tensorflow/pull/21373 from perfinion:nsync 4386d8c4b01020ba1c979258e5102ac1947d2574
PiperOrigin-RevId: 209201545
* Added nGraph bridge as a third_party to be built with TensorFlow based on user selection.
* Added a limited set of C++ unit tests to verify the correctness of the computation
I was using bazel 0.12.0 and was getting the error:
file '@bazel_tools//tools/cpp:windows_cc_configure.bzl' does not contain symbol 'setup_vc_env_vars'
PiperOrigin-RevId: 205316270
This simplifies ./configure process on Windows, we can remove them when we actually support the corresponding feature on Windows.
PiperOrigin-RevId: 204804112
After this change, to build TensorFlow with GPU support on Windows, you just need to follow the same procedure as Linux.
Also re-enable remote cache since the bug in b/111106393 has been fixed by the new wrapper script.
The clean-up of the wrapper scripts also made the build around 5 mins faster than using Bazel's msvc wrapper script.
PiperOrigin-RevId: 204105368
* Add KinesisDataset support for tensorflow Dataset
This fix is an attempt to add Kinesis support
for tensorflow's Dataset. Kinesis is provided by
AWS as a managed data streaming service. It is
similiar to Apache Kafka, often used in places
where maintaining a independent Kafka cluster on AWS
is not desirable or possible.
This fix adds the Kinesis support for tensorflow Dataset.
Similiar to the Kafka integration in tensorflow,
KinesisDataset outputs tf.string for records.
Test cases have also been added, which could be invoked manually.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Expose KinesisDataset in dataset_ops.cc
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Expose KinesisDataset in python wrapper
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Add test cases for KinesisDataset
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Update AWS library include files
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Add Bazel BUILD files
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Rename s3_crypto to aws_crypto
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Rename with_s3_support to with_aws_support
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Selectively add kinesis to tensorflow/contrib/BUILD
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Set different partition key and pylint fix.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Add missing modules in cmake's python_modules.txt
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Address review feedback
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
These rules currently get written by configure.py script to WORKSPACE
file which is not ideal since (1) WORKSPACE file is tracked by git and
(2) we require users to manually delete the rules in order to
update/regenerate them.
Moving these rules into an external repo that is generated based on
several ENV variables set by the configure.py script. Modifying any
of these ENV variables will cause the rules to be updated.
PiperOrigin-RevId: 199388460
Revert #18413. Too many internal test failures due to the name scope change caused by this change.
Revert #18192. Cannot use re2::StringPiece internally. Need alternative for set call. Will pull and clean this up in a separate change.
PiperOrigin-RevId: 197991247
