diff --git a/README.md b/README.md index d239637..5e222b8 100644 --- a/README.md +++ b/README.md @@ -1,58 +1,60 @@ # Introduction -This project contains data compression ops and layers for TensorFlow. The -project website is at -[tensorflow.github.io/compression](https://tensorflow.github.io/compression). +This project contains data compression ops and layers for TensorFlow. You can use this library to build your own ML models with end-to-end optimized data compression built in. It's useful to find storage-efficient representations -of your data (features, examples, images, etc.) while only sacrificing a tiny +of your data (images, features, examples, etc.) while only sacrificing a tiny fraction of model performance. It can compress any floating point tensor to a much smaller sequence of bits. -Specifically, the -[EntropyBottleneck class](https://tensorflow.github.io/compression/docs/entropy_bottleneck.html) -in this library simplifies the process of designing rate–distortion optimized -codes. During training, it acts like a likelihood model. Once training is -completed, it encodes floating point tensors into optimal bit sequences by -automating the design of probability tables and calling a range coder -implementation behind the scenes. +Specifically, the entropy model classes in this library simplify the process of +designing rate–distortion optimized codes. During training, they act like +likelihood models. Once training is completed, they encode floating point +tensors into optimal bit sequences by automating the design of probability +tables and calling a range coder implementation behind the scenes. -For an introduction to lossy image compression with machine learning, take a -look at @jonycgn's -[talk on Learned Image Compression](https://www.youtube.com/watch?v=x_q7cZviXkY). +The main novelty of this method over traditional transform coding is the +stochastic minimization of the rate-distortion Lagrangian, and using nonlinear +transforms implemented by neural networks. For an introduction to this, consider +our [paper on nonlinear transform coding](https://arxiv.org/abs/2007.03034), or +watch @jonycgn's [talk on learned image +compression](https://www.youtube.com/watch?v=x_q7cZviXkY). ## Documentation & getting help -For usage questions and discussions, please head over to our -[Google group](https://groups.google.com/forum/#!forum/tensorflow-compression). +Please post all questions or comments in our [Google +group](https://groups.google.com/forum/#!forum/tensorflow-compression). Only +file Github issues for actual bugs or feature requests. If you post to the group +instead, you may get a faster answer, and you help other people find the +question or answer more easily later. -Refer to -[the API documentation](https://tensorflow.github.io/compression/docs/api_docs/python/tfc.html) +Refer to [the API +documentation](https://tensorflow.github.io/compression/docs/api_docs/python/tfc.html) for a complete description of the Keras layers and TensorFlow ops this package implements. - -There's also an introduction to our `EntropyBottleneck` class -[here](https://tensorflow.github.io/compression/docs/entropy_bottleneck.html), -and a description of the range coding operators -[here](https://tensorflow.github.io/compression/docs/range_coding.html). +***Note: the API docs have not been updated for the current beta release yet.*** ## Installation -***Note: Precompiled packages are currently only provided for Linux (Python 2.7, -3.3-3.6) and Darwin/Mac OS (Python 2.7, 3.7). To use these packages on Windows, -consider using a -[TensorFlow Docker image](https://www.tensorflow.org/install/docker) and -installing tensorflow-compression using pip inside the Docker container.*** +***Note: Precompiled packages are currently only provided for Linux and +Darwin/Mac OS. To use these packages on Windows, consider using a [TensorFlow +Docker image](https://www.tensorflow.org/install/docker) and installing +tensorflow-compression using pip inside the Docker container.*** Set up an environment in which you can install precompiled binary Python packages using the `pip` command. Refer to the [TensorFlow installation instructions](https://www.tensorflow.org/install/pip) for more information on how to set up such a Python environment. -The current version of tensorflow-compression requires TensorFlow 1.15. For -TensorFlow 1.14 or earlier, see our -[previous releases](https://github.com/tensorflow/compression/releases). You can +The current stable version of TFC (1.3) requires TensorFlow 1.15. The current +beta release of TFC (2.0b1) is built for TensorFlow 2.3. For versions compatible +with TensorFlow 1.14 or earlier, see our [previous +releases](https://github.com/tensorflow/compression/releases). The following +instructions are for the stable release. To install the beta, replace the +version numbers with 2.0b1 and 2.3, respectively. + +You can install TensorFlow from any source. To install it via `pip`, run the following command: ```bash @@ -67,7 +69,7 @@ for CPU-only. Then, run the following command to install the tensorflow-compression pip package: ```bash -pip install tensorflow-compression +pip install tensorflow-compression==1.3 ``` To test that the installation works correctly, you can run the unit tests with @@ -86,7 +88,7 @@ and then run the `pip install` command inside the Docker container, not on the host. For instance, you can use a command line like this: ```bash docker run tensorflow/tensorflow:1.15.0-py3 bash -c \ - "pip install tensorflow-compression && + "pip install tensorflow-compression==1.3 && python -m tensorflow_compression.python.all_test" ``` This will fetch the TensorFlow Docker image if it's not already cached, install @@ -95,16 +97,15 @@ the pip package and then run the unit tests to confirm that it works. ### Anaconda It seems that [Anaconda](https://www.anaconda.com/distribution/) ships its own -binary version of TensorFlow which is incompatible with our pip package. It -also installs Python 3.7 by default, which we currently don't support on Linux. -To solve this, make sure to use Python 3.6 on Linux, and always install -TensorFlow via `pip` rather than `conda`. For example, this creates an Anaconda -environment with Python 3.6 and CUDA libraries, and then installs TensorFlow -and tensorflow-compression with GPU support: +binary version of TensorFlow which is incompatible with our pip package. To +solve this, always install TensorFlow via `pip` rather than `conda`. For +example, this creates an Anaconda environment with Python 3.6 and CUDA +libraries, and then installs TensorFlow and tensorflow-compression with GPU +support: ```bash conda create --name ENV_NAME python=3.6 cudatoolkit=10.0 cudnn conda activate ENV_NAME -pip install tensorflow-gpu==1.15 tensorflow-compression +pip install tensorflow-gpu==1.15 tensorflow-compression==1.3 ``` ## Usage @@ -194,24 +195,24 @@ This section describes the necessary steps to build your own pip packages of tensorflow-compression. This may be necessary to install it on platforms for which we don't provide precompiled binaries (currently only Linux and Darwin). -We use the Docker image `tensorflow/tensorflow:custom-op-ubuntu16` for building -pip packages for Linux. Note that this is different from -`tensorflow/tensorflow:devel`. To be compatible with the TensorFlow pip package, -the GCC version must match, but `tensorflow/tensorflow:devel` has a different -GCC version installed. +We use the custom-op Docker images (e.g. +`tensorflow/tensorflow:nightly-custom-op-ubuntu16`) for building pip packages +for Linux. Note that this is different from `tensorflow/tensorflow:devel`. To be +compatible with the TensorFlow pip package, the GCC version must match, but +`tensorflow/tensorflow:devel` has a different GCC version installed. For more +information, refer to the [custom-op +instructions](https://github.com/tensorflow/custom-op). Inside a Docker container from the image, the following steps need to be taken. -1. Install the TensorFlow pip package. -2. Clone the `tensorflow/compression` repo from GitHub. -3. Run `:build_pip_pkg` inside the cloned repo. +1. Clone the `tensorflow-compression` repo from GitHub. +2. Run `:build_pip_pkg` inside the cloned repo. For example: ```bash sudo docker run -v /tmp/tensorflow_compression:/tmp/tensorflow_compression \ - tensorflow/tensorflow:custom-op-ubuntu16 bash -c \ - "pip install tensorflow==1.15 && - git clone https://github.com/tensorflow/compression.git + tensorflow/tensorflow:nightly-custom-op-ubuntu16 bash -c \ + "git clone https://github.com/tensorflow/compression.git /tensorflow_compression && cd /tensorflow_compression && bazel run -c opt --copt=-mavx :build_pip_pkg" @@ -228,7 +229,7 @@ pip install /tmp/tensorflow_compression/tensorflow_compression-*.whl Then run the unit tests (Do not run the tests in the workspace directory where `WORKSPACE` of `tensorflow_compression` repo lives. In that case, the Python interpreter would attempt to import `tensorflow_compression` packages from the -source tree rather than from the installed package system directory): +source tree, rather than from the installed package system directory): ```bash pushd /tmp python -m tensorflow_compression.python.all_test @@ -256,5 +257,6 @@ for more information. * Sung Jin Hwang (github: [ssjhv](https://github.com/ssjhv)) * Nick Johnston (github: [nmjohn](https://github.com/nmjohn)) * David Minnen (github: [minnend](https://github.com/minnend)) +* Eirikur Agustsson (github: [relational](https://github.com/relational)) Note that this is not an officially supported Google product. diff --git a/build_pip_pkg.py b/build_pip_pkg.py index f03642a..ae4ebe7 100644 --- a/build_pip_pkg.py +++ b/build_pip_pkg.py @@ -64,6 +64,7 @@ def main(srcdir): packages=setuptools.find_packages(), install_requires=[ 'scipy >= 1', + 'tensorflow_probability >= 0.9', ], script_args=['sdist', 'bdist_wheel'], # Add in any packaged data. @@ -77,11 +78,10 @@ def main(srcdir): 'Intended Audience :: Education', 'Intended Audience :: Science/Research', 'License :: OSI Approved :: Apache Software License', - 'Programming Language :: Python :: 3.3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Scientific/Engineering :: Mathematics', 'Topic :: Software Development :: Libraries :: Python Modules', 'Topic :: Software Development :: Libraries', diff --git a/tensorflow_compression/__init__.py b/tensorflow_compression/__init__.py index 58029e3..9ac2890 100644 --- a/tensorflow_compression/__init__.py +++ b/tensorflow_compression/__init__.py @@ -17,10 +17,10 @@ try: import tensorflow as _tensorflow _tf_version = [int(v) for v in _tensorflow.version.VERSION.split(".")] - assert _tf_version[0] == 2 and _tf_version[1] == 1 + assert _tf_version[0] == 2 and _tf_version[1] == 3 except (ImportError, AssertionError): raise RuntimeError( - "For tensorflow_compression, please install TensorFlow 2.1.") + "For tensorflow_compression, please install TensorFlow 2.3.") # pylint: disable=wildcard-import from tensorflow_compression.python.distributions.deep_factorized import * diff --git a/tensorflow_compression/python/all_test.py b/tensorflow_compression/python/all_test.py index 5a08f21..e550ee5 100644 --- a/tensorflow_compression/python/all_test.py +++ b/tensorflow_compression/python/all_test.py @@ -21,6 +21,15 @@ import tensorflow.compat.v1 as tf # pylint: disable=wildcard-import +from tensorflow_compression.python.distributions.deep_factorized_test import * +from tensorflow_compression.python.distributions.helpers_test import * +from tensorflow_compression.python.distributions.round_adapters_test import * +from tensorflow_compression.python.distributions.uniform_noise_test import * + +from tensorflow_compression.python.entropy_models.continuous_batched_test import * +from tensorflow_compression.python.entropy_models.continuous_indexed_test import * +from tensorflow_compression.python.entropy_models.universal_test import * + from tensorflow_compression.python.layers.entropy_models_test import * from tensorflow_compression.python.layers.gdn_test import * from tensorflow_compression.python.layers.parameterizers_test import * @@ -30,7 +39,7 @@ from tensorflow_compression.python.ops.math_ops_test import * from tensorflow_compression.python.ops.padding_ops_test import * from tensorflow_compression.python.ops.range_coding_ops_test import * -from tensorflow_compression.python.ops.round_ops_test import * +from tensorflow_compression.python.ops.soft_round_ops_test import * from tensorflow_compression.python.ops.spectral_ops_test import * from tensorflow_compression.python.util.packed_tensors_test import * diff --git a/tensorflow_compression/python/distributions/BUILD b/tensorflow_compression/python/distributions/BUILD index e75dfb6..4645b0f 100644 --- a/tensorflow_compression/python/distributions/BUILD +++ b/tensorflow_compression/python/distributions/BUILD @@ -1,5 +1,5 @@ package( - default_visibility = ["//:__subpackages__"], + default_visibility = ["//visibility:public"], ) licenses(["notice"]) # Apache 2.0 diff --git a/tensorflow_compression/python/entropy_models/BUILD b/tensorflow_compression/python/entropy_models/BUILD index 0511025..f279645 100644 --- a/tensorflow_compression/python/entropy_models/BUILD +++ b/tensorflow_compression/python/entropy_models/BUILD @@ -1,5 +1,5 @@ package( - default_visibility = ["//:__subpackages__"], + default_visibility = ["//visibility:public"], ) licenses(["notice"]) # Apache 2.0 diff --git a/tensorflow_compression/python/layers/BUILD b/tensorflow_compression/python/layers/BUILD index ab9b58d..13a872b 100644 --- a/tensorflow_compression/python/layers/BUILD +++ b/tensorflow_compression/python/layers/BUILD @@ -1,5 +1,5 @@ package( - default_visibility = ["//:__subpackages__"], + default_visibility = ["//visibility:public"], ) licenses(["notice"]) # Apache 2.0 diff --git a/tensorflow_compression/python/ops/BUILD b/tensorflow_compression/python/ops/BUILD index 1ecab1a..3559230 100644 --- a/tensorflow_compression/python/ops/BUILD +++ b/tensorflow_compression/python/ops/BUILD @@ -1,5 +1,5 @@ package( - default_visibility = [":__subpackages__"], + default_visibility = ["//visibility:public"], ) licenses(["notice"]) # Apache 2.0 diff --git a/tensorflow_compression/python/util/BUILD b/tensorflow_compression/python/util/BUILD index 9f005ab..e0ec209 100644 --- a/tensorflow_compression/python/util/BUILD +++ b/tensorflow_compression/python/util/BUILD @@ -1,5 +1,5 @@ package( - default_visibility = ["//:__subpackages__"], + default_visibility = ["//visibility:public"], ) licenses(["notice"]) # Apache 2.0 diff --git a/workspace.bzl b/workspace.bzl index 08dc71c..e47b0bd 100644 --- a/workspace.bzl +++ b/workspace.bzl @@ -22,7 +22,7 @@ def _tensorflow_pip_impl(ctx): "tensorflow?: %s" % include_path.stderr) if "linux" in ctx.os.name: - library_filename = "libtensorflow_framework.so.1" + library_filename = "libtensorflow_framework.so.2" elif "mac" in ctx.os.name: library_filename = "libtensorflow_framework.dylib"