Why and How Developers Extend TensorFlow With Custom Kernels

Content Overview Prerequisites Define the op interface Implement the kernel for the op Multi-threaded CPU kernels GPU kernels Build the op library Compile the op using your system compiler (TensorFlow binary installation) Compile the op using bazel (TensorFlow source installation) Use the op in Python Verify that the op works Prerequisites Define the op interface Implement the kernel for the op Multi-threaded CPU kernels GPU kernels Build the op library Compile the op using your system compiler (TensorFlow binary installation) Compile the op using bazel (TensorFlow source installation) Use the op in Python Verify that the op works Note: To guarantee that your C++ custom ops are ABI compatible with TensorFlow's official pip packages, please follow the guide at Custom op repository. It has an end-to-end code example, as well as Docker images for building and distributing your custom ops. Note: To guarantee that your C++ custom ops are ABI compatible with TensorFlow's official pip packages, please follow the guide at Custom op repository. It has an end-to-end code example, as well as Docker images for building and distributing your custom ops. Note: Custom op repository Custom op repository If you'd like to create an op that isn't covered by the existing TensorFlow library, we recommend that you first try writing the op in Python as a composition of existing Python ops or functions. If that isn't possible, you can create a custom C++ op. There are several reasons why you might want to create a custom C++ op: It's not easy or possible to express your operation as a composition of existing ops. It's not efficient to express your operation as a composition of existing primitives. You want to hand-fuse a composition of primitives that a future compiler would find difficult fusing. It's not easy or possible to express your operation as a composition of existing ops. It's not efficient to express your operation as a composition of existing primitives. You want to hand-fuse a composition of primitives that a future compiler would find difficult fusing. For example, imagine you want to implement something like "median pooling", similar to the "MaxPool" operator, but computing medians over sliding windows instead of maximum values. Doing this using a composition of operations may be possible (e.g., using ExtractImagePatches and TopK), but may not be as performance- or memory-efficient as a native operation where you can do something more clever in a single, fused operation. As always, it is typically first worth trying to express what you want using operator composition, only choosing to add a new operation if that proves to be difficult or inefficient. To incorporate your custom op you'll need to: Register the new op in a C++ file. Op registration defines an interface (specification) for the op's functionality, which is independent of the op's implementation. For example, op registration defines the op's name and the op's inputs and outputs. It also defines the shape function that is used for tensor shape inference. Implement the op in C++. The implementation of an op is known as a kernel, and it is the concrete implementation of the specification you registered in Step 1. There can be multiple kernels for different input / output types or architectures (for example, CPUs, GPUs). Create a Python wrapper (optional). This wrapper is the public API that's used to create the op in Python. A default wrapper is generated from the op registration, which can be used directly or added to. Write a function to compute gradients for the op (optional). Test the op. We usually do this in Python for convenience, but you can also test the op in C++. If you define gradients, you can verify them with the Python tf.test.compute_gradient_error. See relu_op_test.py as an example that tests the forward functions of Relu-like operators and their gradients. Register the new op in a C++ file. Op registration defines an interface (specification) for the op's functionality, which is independent of the op's implementation. For example, op registration defines the op's name and the op's inputs and outputs. It also defines the shape function that is used for tensor shape inference. Implement the op in C++. The implementation of an op is known as a kernel, and it is the concrete implementation of the specification you registered in Step 1. There can be multiple kernels for different input / output types or architectures (for example, CPUs, GPUs). Create a Python wrapper (optional). This wrapper is the public API that's used to create the op in Python. A default wrapper is generated from the op registration, which can be used directly or added to. Write a function to compute gradients for the op (optional). Test the op. We usually do this in Python for convenience, but you can also test the op in C++. If you define gradients, you can verify them with the Python tf.test.compute_gradient_error. See relu_op_test.py as an example that tests the forward functions of Relu-like operators and their gradients. tf.test.compute_gradient_error relu_op_test.py Prerequisites Some familiarity with C++. Must have installed the TensorFlow binary, or must have downloaded TensorFlow source, and be able to build it. Some familiarity with C++. Must have installed the TensorFlow binary, or must have downloaded TensorFlow source, and be able to build it. TensorFlow binary downloaded TensorFlow source Define the op interface You define the interface of an op by registering it with the TensorFlow system. In the registration, you specify the name of your op, its inputs (types and names) and outputs (types and names), as well as docstrings and any attrs the op might require. attrs To see how this works, suppose you'd like to create an op that takes a tensor of int32s and outputs a copy of the tensor, with all but the first element set to zero. To do this, create a file named zero_out.cc. Then add a call to the REGISTER_OP macro that defines the interface for your op: int32 zero_out.cc REGISTER_OP #include "tensorflow/core/framework/op.h" #include "tensorflow/core/framework/shape_inference.h" using namespace tensorflow; REGISTER_OP("ZeroOut") .Input("to_zero: int32") .Output("zeroed: int32") .SetShapeFn([](::tensorflow::shape_inference::InferenceContext* c) { c->set_output(0, c->input(0)); return Status::OK(); }); #include "tensorflow/core/framework/op.h" #include "tensorflow/core/framework/shape_inference.h" using namespace tensorflow; REGISTER_OP("ZeroOut") .Input("to_zero: int32") .Output("zeroed: int32") .SetShapeFn([](::tensorflow::shape_inference::InferenceContext* c) { c->set_output(0, c->input(0)); return Status::OK(); }); This ZeroOut op takes one tensor to_zero of 32-bit integers as input, and outputs a tensor zeroed of 32-bit integers. The op also uses a shape function to ensure that the output tensor is the same shape as the input tensor. For example, if the input is a tensor of shape [10, 20], then this shape function specifies that the output shape is also [10, 20]. ZeroOut to_zero zeroed Note: The op name must be in CamelCase and it must be unique among all other ops that are registered in the binary. Note: The op name must be in CamelCase and it must be unique among all other ops that are registered in the binary. Note: Implement the kernel for the op After you define the interface, provide one or more implementations of the op. To create one of these kernels, create a class that extends OpKernel and overrides the Compute method. The Compute method provides one context argument of type OpKernelContext*, from which you can access useful things like the input and output tensors. OpKernel Compute Compute context OpKernelContext* Add your kernel to the file you created above. The kernel might look something like this: #include "tensorflow/core/framework/op_kernel.h" using namespace tensorflow; class ZeroOutOp : public OpKernel { public: explicit ZeroOutOp(OpKernelConstruction* context) : OpKernel(context) {} void Compute(OpKernelContext* context) override { // Grab the input tensor const Tensor& input_tensor = context->input(0); auto input = input_tensor.flat (); // Create an output tensor Tensor* output_tensor = NULL; OP_REQUIRES_OK(context, context->allocate_output(0, input_tensor.shape(), &output_tensor)); auto output_flat = output_tensor->flat (); // Set all but the first element of the output tensor to 0. const int N = input.size(); for (int i = 1; i 0) output_flat(0) = input(0); } }; #include "tensorflow/core/framework/op_kernel.h" using namespace tensorflow; class ZeroOutOp : public OpKernel { public: explicit ZeroOutOp(OpKernelConstruction* context) : OpKernel(context) {} void Compute(OpKernelContext* context) override { // Grab the input tensor const Tensor& input_tensor = context->input(0); auto input = input_tensor.flat (); // Create an output tensor Tensor* output_tensor = NULL; OP_REQUIRES_OK(context, context->allocate_output(0, input_tensor.shape(), &output_tensor)); auto output_flat = output_tensor->flat (); // Set all but the first element of the output tensor to 0. const int N = input.size(); for (int i = 1; i 0) output_flat(0) = input(0); } }; After implementing your kernel, you register it with the TensorFlow system. In the registration, you specify different constraints under which this kernel will run. For example, you might have one kernel made for CPUs, and a separate one for GPUs. To do this for the ZeroOut op, add the following to zero_out.cc: ZeroOut zero_out.cc REGISTER_KERNEL_BUILDER(Name("ZeroOut").Device(DEVICE_CPU), ZeroOutOp); REGISTER_KERNEL_BUILDER(Name("ZeroOut").Device(DEVICE_CPU), ZeroOutOp); Important: Instances of your OpKernel may be accessed concurrently. Your Compute method must be thread-safe. Guard any access to class members with a mutex. Or better yet, don't share state via class members! Consider using a ResourceMgr to keep track of op state. Important: Instances of your OpKernel may be accessed concurrently. Your Compute method must be thread-safe. Guard any access to class members with a mutex. Or better yet, don't share state via class members! Consider using a ResourceMgr to keep track of op state. Important: Compute ResourceMgr Multi-threaded CPU kernels To write a multi-threaded CPU kernel, the Shard function in work_sharder.h can be used. This function shards a computation function across the threads configured to be used for intra-op threading (see intra_op_parallelism_threads in config.proto). work_sharder.h config.proto GPU kernels A GPU kernel is implemented in two parts: the OpKernel and the CUDA kernel and its launch code. Sometimes the OpKernel implementation is common between a CPU and GPU kernel, such as around inspecting inputs and allocating outputs. In that case, a suggested implementation is to: Define the OpKernel templated on the Device and the primitive type of the tensor. To do the actual computation of the output, the Compute function calls a templated functor struct. The specialization of that functor for the CPUDevice is defined in the same file, but the specialization for the GPUDevice is defined in a .cu.cc file, since it will be compiled with the CUDA compiler. Define the OpKernel templated on the Device and the primitive type of the tensor. To do the actual computation of the output, the Compute function calls a templated functor struct. The specialization of that functor for the CPUDevice is defined in the same file, but the specialization for the GPUDevice is defined in a .cu.cc file, since it will be compiled with the CUDA compiler. Here is an example implementation. // kernel_example.h #ifndef KERNEL_EXAMPLE_H_ #define KERNEL_EXAMPLE_H_ #include template struct ExampleFunctor { void operator()(const Device& d, int size, const T* in, T* out); }; #if GOOGLE_CUDA // Partially specialize functor for GpuDevice. template struct ExampleFunctor { void operator()(const Eigen::GpuDevice& d, int size, const T* in, T* out); }; #endif #endif KERNEL_EXAMPLE_H_ // kernel_example.h #ifndef KERNEL_EXAMPLE_H_ #define KERNEL_EXAMPLE_H_ #include template struct ExampleFunctor { void operator()(const Device& d, int size, const T* in, T* out); }; #if GOOGLE_CUDA // Partially specialize functor for GpuDevice. template struct ExampleFunctor { void operator()(const Eigen::GpuDevice& d, int size, const T* in, T* out); }; #endif #endif KERNEL_EXAMPLE_H_ // kernel_example.cc #include "kernel_example.h" #include "tensorflow/core/framework/op.h" #include "tensorflow/core/framework/shape_inference.h" #include "tensorflow/core/framework/op_kernel.h" using namespace tensorflow; using CPUDevice = Eigen::ThreadPoolDevice; using GPUDevice = Eigen::GpuDevice; REGISTER_OP("Example") .Attr("T: numbertype") .Input("input: T") .Output("input_times_two: T") .SetShapeFn([](::tensorflow::shape_inference::InferenceContext* c) { c->set_output(0, c->input(0)); return Status::OK(); }); // CPU specialization of actual computation. template struct ExampleFunctor { void operator()(const CPUDevice& d, int size, const T* in, T* out) { for (int i = 0; i is the datatype of the tensors. template class ExampleOp : public OpKernel { public: explicit ExampleOp(OpKernelConstruction* context) : OpKernel(context) {} void Compute(OpKernelContext* context) override { // Grab the input tensor const Tensor& input_tensor = context->input(0); // Create an output tensor Tensor* output_tensor = NULL; OP_REQUIRES_OK(context, context->allocate_output(0, input_tensor.shape(), &output_tensor)); // Do the computation. OP_REQUIRES(context, input_tensor.NumElements() ()( context->eigen_device (), static_cast (input_tensor.NumElements()), input_tensor.flat ().data(), output_tensor->flat ().data()); } }; // Register the CPU kernels. #define REGISTER_CPU(T) \ REGISTER_KERNEL_BUILDER( \ Name("Example").Device(DEVICE_CPU).TypeConstraint ("T"), \ ExampleOp ); REGISTER_CPU(float); REGISTER_CPU(int32); // Register the GPU kernels. #ifdef GOOGLE_CUDA #define REGISTER_GPU(T) \ /* Declare explicit instantiations in kernel_example.cu.cc. */ \ extern template class ExampleFunctor ; \ REGISTER_KERNEL_BUILDER( \ Name("Example").Device(DEVICE_GPU).TypeConstraint ("T"), \ ExampleOp ); REGISTER_GPU(float); REGISTER_GPU(int32); #endif // GOOGLE_CUDA // kernel_example.cc #include "kernel_example.h" #include "tensorflow/core/framework/op.h" #include "tensorflow/core/framework/shape_inference.h" #include "tensorflow/core/framework/op_kernel.h" using namespace tensorflow; using CPUDevice = Eigen::ThreadPoolDevice; using GPUDevice = Eigen::GpuDevice; REGISTER_OP("Example") .Attr("T: numbertype") .Input("input: T") .Output("input_times_two: T") .SetShapeFn([](::tensorflow::shape_inference::InferenceContext* c) { c->set_output(0, c->input(0)); return Status::OK(); }); // CPU specialization of actual computation. template struct ExampleFunctor { void operator()(const CPUDevice& d, int size, const T* in, T* out) { for (int i = 0; i is the datatype of the tensors. template class ExampleOp : public OpKernel { public: explicit ExampleOp(OpKernelConstruction* context) : OpKernel(context) {} void Compute(OpKernelContext* context) override { // Grab the input tensor const Tensor& input_tensor = context->input(0); // Create an output tensor Tensor* output_tensor = NULL; OP_REQUIRES_OK(context, context->allocate_output(0, input_tensor.shape(), &output_tensor)); // Do the computation. OP_REQUIRES(context, input_tensor.NumElements() ()( context->eigen_device (), static_cast (input_tensor.NumElements()), input_tensor.flat ().data(), output_tensor->flat ().data()); } }; // Register the CPU kernels. #define REGISTER_CPU(T) \ REGISTER_KERNEL_BUILDER( \ Name("Example").Device(DEVICE_CPU).TypeConstraint ("T"), \ ExampleOp ); REGISTER_CPU(float); REGISTER_CPU(int32); // Register the GPU kernels. #ifdef GOOGLE_CUDA #define REGISTER_GPU(T) \ /* Declare explicit instantiations in kernel_example.cu.cc. */ \ extern template class ExampleFunctor ; \ REGISTER_KERNEL_BUILDER( \ Name("Example").Device(DEVICE_GPU).TypeConstraint ("T"), \ ExampleOp ); REGISTER_GPU(float); REGISTER_GPU(int32); #endif // GOOGLE_CUDA // kernel_example.cu.cc #ifdef GOOGLE_CUDA #define EIGEN_USE_GPU #include "kernel_example.h" #include "tensorflow/core/util/gpu_kernel_helper.h" using namespace tensorflow; using GPUDevice = Eigen::GpuDevice; // Define the CUDA kernel. template __global__ void ExampleCudaKernel(const int size, const T* in, T* out) { for (int i = blockIdx.x * blockDim.x + threadIdx.x; i void ExampleFunctor ::operator()( const GPUDevice& d, int size, const T* in, T* out) { // Launch the cuda kernel. // // See core/util/gpu_kernel_helper.h for example of computing // block count and thread_per_block count. int block_count = 1024; int thread_per_block = 20; ExampleCudaKernel >>(size, in, out); } // Explicitly instantiate functors for the types of OpKernels registered. template struct ExampleFunctor ; template struct ExampleFunctor ; #endif // GOOGLE_CUDA // kernel_example.cu.cc #ifdef GOOGLE_CUDA #define EIGEN_USE_GPU #include "kernel_example.h" #include "tensorflow/core/util/gpu_kernel_helper.h" using namespace tensorflow; using GPUDevice = Eigen::GpuDevice; // Define the CUDA kernel. template __global__ void ExampleCudaKernel(const int size, const T* in, T* out) { for (int i = blockIdx.x * blockDim.x + threadIdx.x; i void ExampleFunctor ::operator()( const GPUDevice& d, int size, const T* in, T* out) { // Launch the cuda kernel. // // See core/util/gpu_kernel_helper.h for example of computing // block count and thread_per_block count. int block_count = 1024; int thread_per_block = 20; ExampleCudaKernel >>(size, in, out); } // Explicitly instantiate functors for the types of OpKernels registered. template struct ExampleFunctor ; template struct ExampleFunctor ; #endif // GOOGLE_CUDA Build the op library Compile the op using your system compiler (TensorFlow binary installation) You should be able to compile zero_out.cc with a C++ compiler such as g++ or clang available on your system. The binary PIP package installs the header files and the library that you need to compile your op in locations that are system specific. However, the TensorFlow python library provides the get_include function to get the header directory, and the get_lib directory has a shared object to link against. Here are the outputs of these functions on an Ubuntu machine. zero_out.cc C++ g++ clang get_include get_lib $ python >>> import tensorflow as tf >>> tf.sysconfig.get_include() '/usr/local/lib/python3.6/site-packages/tensorflow/include' >>> tf.sysconfig.get_lib() '/usr/local/lib/python3.6/site-packages/tensorflow' $ python >>> import tensorflow as tf >>> tf.sysconfig.get_include() '/usr/local/lib/python3.6/site-packages/tensorflow/include' >>> tf.sysconfig.get_lib() '/usr/local/lib/python3.6/site-packages/tensorflow' Assuming you have g++ installed, here is the sequence of commands you can use to compile your op into a dynamic library. g++ TF_CFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') ) TF_LFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))') ) g++ -std=c++14 -shared zero_out.cc -o zero_out.so -fPIC ${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -O2 TF_CFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') ) TF_LFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))') ) g++ -std=c++14 -shared zero_out.cc -o zero_out.so -fPIC ${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -O2 On macOS, the additional flag "-undefined dynamic_lookup" is required when building the .so file. .so Note on gcc version >=5: gcc uses the new C++ ABI since version 5. TensorFlow 2.8 and earlier were built with gcc4 that uses the older ABI. If you are using these versions of TensorFlow and are trying to compile your op library with gcc>=5, add -D_GLIBCXX_USE_CXX11_ABI=0 to the command line to make the library compatible with the older ABI. TensorFlow 2.9+ packages are compatible with the newer ABI by default. Note on gcc version >=5: gcc uses the new C++ ABI since version 5. TensorFlow 2.8 and earlier were built with gcc4 that uses the older ABI. If you are using these versions of TensorFlow and are trying to compile your op library with gcc>=5, add -D_GLIBCXX_USE_CXX11_ABI=0 to the command line to make the library compatible with the older ABI. TensorFlow 2.9+ packages are compatible with the newer ABI by default. gcc >=5 ABI 5 gcc4 gcc>=5 -D_GLIBCXX_USE_CXX11_ABI=0 Compile the op using bazel (TensorFlow source installation) If you have TensorFlow sources installed, you can make use of TensorFlow's build system to compile your op. Place a BUILD file with following Bazel build rule in the tensorflow/core/user_ops directory. tensorflow/core/user_ops load("//tensorflow:tensorflow.bzl", "tf_custom_op_library") tf_custom_op_library( name = "zero_out.so", srcs = ["zero_out.cc"], ) load("//tensorflow:tensorflow.bzl", "tf_custom_op_library") tf_custom_op_library( name = "zero_out.so", srcs = ["zero_out.cc"], ) Run the following command to build zero_out.so. zero_out.so $ bazel build --config opt //tensorflow/core/user_ops:zero_out.so $ bazel build --config opt //tensorflow/core/user_ops:zero_out.so For compiling the Example operation, with the CUDA Kernel, you need to use the gpu_srcs parameter of tf_custom_op_library. Place a BUILD file with the following Bazel build rule in a new folder inside the tensorflow/core/user_ops directory (e.g. "example_gpu"). Example gpu_srcs tf_custom_op_library tensorflow/core/user_ops load("//tensorflow:tensorflow.bzl", "tf_custom_op_library") tf_custom_op_library( # kernel_example.cc kernel_example.cu.cc kernel_example.h name = "kernel_example.so", srcs = ["kernel_example.h", "kernel_example.cc"], gpu_srcs = ["kernel_example.cu.cc", "kernel_example.h"], ) load("//tensorflow:tensorflow.bzl", "tf_custom_op_library") tf_custom_op_library( # kernel_example.cc kernel_example.cu.cc kernel_example.h name = "kernel_example.so", srcs = ["kernel_example.h", "kernel_example.cc"], gpu_srcs = ["kernel_example.cu.cc", "kernel_example.h"], ) Run the following command to build kernel_example.so. kernel_example.so $ bazel build --config opt //tensorflow/core/user_ops/example_gpu:kernel_example.so $ bazel build --config opt //tensorflow/core/user_ops/example_gpu:kernel_example.so Note: As explained above, if you are compiling with gcc>=5 add --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" to the Bazel command line arguments. Note: As explained above, if you are compiling with gcc>=5 add --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" to the Bazel command line arguments. Note: --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" Note: Although you can create a shared library (a .so file) with the standard cc_library rule, we strongly recommend that you use the tf_custom_op_library macro. It adds some required dependencies, and performs checks to ensure that the shared library is compatible with TensorFlow's plugin loading mechanism. Note: Although you can create a shared library (a .so file) with the standard cc_library rule, we strongly recommend that you use the tf_custom_op_library macro. It adds some required dependencies, and performs checks to ensure that the shared library is compatible with TensorFlow's plugin loading mechanism. Note: .so cc_library tf_custom_op_library Use the op in Python TensorFlow Python API provides the tf.load_op_library function to load the dynamic library and register the op with the TensorFlow framework. load_op_library returns a Python module that contains the Python wrappers for the op and the kernel. Thus, once you have built the op, you can do the following to run it from Python: tf.load_op_library load_op_library import tensorflow as tf zero_out_module = tf.load_op_library('./zero_out.so') print(zero_out_module.zero_out([[1, 2], [3, 4]]).numpy()) # Prints array([[1, 0], [0, 0]], dtype=int32) import tensorflow as tf zero_out_module = tf.load_op_library('./zero_out.so') print(zero_out_module.zero_out([[1, 2], [3, 4]]).numpy()) # Prints array([[1, 0], [0, 0]], dtype=int32) Keep in mind, the generated function will be given a snake_case name (to comply with PEP8). So, if your op is named ZeroOut in the C++ files, the python function will be called zero_out. PEP8 ZeroOut zero_out To make the op available as a regular function import-able from a Python module, it maybe useful to have the load_op_library call in a Python source file as follows: import load_op_library import tensorflow as tf zero_out_module = tf.load_op_library('./zero_out.so') zero_out = zero_out_module.zero_out import tensorflow as tf zero_out_module = tf.load_op_library('./zero_out.so') zero_out = zero_out_module.zero_out Verify that the op works A good way to verify that you've successfully implemented your op is to write a test for it. Create the file zero_out_op_test.py with the contents: zero_out_op_test.py import tensorflow as tf class ZeroOutTest(tf.test.TestCase): def testZeroOut(self): zero_out_module = tf.load_op_library('./zero_out.so') with self.test_session(): result = zero_out_module.zero_out([5, 4, 3, 2, 1]) self.assertAllEqual(result.eval(), [5, 0, 0, 0, 0]) if __name__ == "__main__": tf.test.main() import tensorflow as tf class ZeroOutTest(tf.test.TestCase): def testZeroOut(self): zero_out_module = tf.load_op_library('./zero_out.so') with self.test_session(): result = zero_out_module.zero_out([5, 4, 3, 2, 1]) self.assertAllEqual(result.eval(), [5, 0, 0, 0, 0]) if __name__ == "__main__": tf.test.main() Then run your test (assuming you have tensorflow installed): $ python zero_out_op_test.py $ python zero_out_op_test.py Originally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License. Originally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License. Originally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License. TensorFlow TensorFlow