Support for macOS #24

lifeiteng · 2020-11-21T05:52:14Z

Pass reverb/cc/...

//reverb/cc:chunk_store_test                                    (cached) PASSED in 0.5s
//reverb/cc:client_test                                         (cached) PASSED in 0.1s
//reverb/cc:rate_limiter_test                                   (cached) PASSED in 1.5s
//reverb/cc:reverb_service_impl_test                            (cached) PASSED in 0.6s
//reverb/cc:sampler_test                                        (cached) PASSED in 4.2s
//reverb/cc:table_test                                          (cached) PASSED in 3.1s
//reverb/cc:tensor_compression_test                             (cached) PASSED in 0.2s
//reverb/cc:writer_test                                         (cached) PASSED in 1.2s
//reverb/cc/platform:net_test                                   (cached) PASSED in 0.2s
//reverb/cc/platform:server_test                                (cached) PASSED in 0.3s
//reverb/cc/platform:tfrecord_checkpointer_test                 (cached) PASSED in 1.7s
//reverb/cc/platform:thread_test                                (cached) PASSED in 0.2s
//reverb/cc/selectors:fifo_test                                 (cached) PASSED in 0.2s
//reverb/cc/selectors:heap_test                                 (cached) PASSED in 0.2s
//reverb/cc/selectors:lifo_test                                 (cached) PASSED in 0.2s
//reverb/cc/selectors:prioritized_test                          (cached) PASSED in 0.3s
//reverb/cc/selectors:uniform_test                              (cached) PASSED in 0.2s
//reverb/cc/support:cleanup_test                                (cached) PASSED in 0.2s
//reverb/cc/support:intrusive_heap_test                         (cached) PASSED in 0.4s
//reverb/cc/support:periodic_closure_test                       (cached) PASSED in 0.3s
//reverb/cc/support:queue_test                                  (cached) PASSED in 0.2s

Fix py bind
Fix all tests
Final

lifeiteng · 2020-11-21T05:52:46Z

#11

lifeiteng · 2020-11-21T11:56:34Z

I'm trying to Fix python tests

$ bash oss_build.sh --python 3.7

####=======Testing reverb//client_test.py=======####
Traceback (most recent call last):
  File "reverb//client_test.py", line 24, in <module>
    from reverb import client
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/__init__.py", line 27, in <module>
    from reverb import item_selectors as selectors
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/item_selectors.py", line 19, in <module>
    from reverb import pybind
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/pybind.py", line 1, in <module>
    import tensorflow as _tf; from .libpybind import *; del _tf
ImportError: dlopen(/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/libpybind.so, 2): Library not loaded: @loader_path/../_solib_darwin/_U_S_Sreverb_Scc_Creverb_Uservice_Ucc_Uproto___Ureverb_Scc/libreverb_service_cc_proto.so
  Referenced from: /Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/libpybind.so
  Reason: image not found
FAIL: reverb//client_test.py

===========Error Summary============
FAIL: reverb//platform/checkpointers_lib_test.py
FAIL: reverb//dataset_test.py
FAIL: reverb//tf_client_test.py
FAIL: reverb//pybind_test.py
FAIL: reverb//server_test.py
FAIL: reverb//rate_limiters_test.py
FAIL: reverb//client_test.py

ebrevdo · 2020-12-02T16:43:26Z

I'm trying to Fix python tests

$ bash oss_build.sh --python 3.7

####=======Testing reverb//client_test.py=======####
Traceback (most recent call last):
  File "reverb//client_test.py", line 24, in <module>
    from reverb import client
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/__init__.py", line 27, in <module>
    from reverb import item_selectors as selectors
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/item_selectors.py", line 19, in <module>
    from reverb import pybind
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/pybind.py", line 1, in <module>
    import tensorflow as _tf; from .libpybind import *; del _tf
ImportError: dlopen(/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/libpybind.so, 2): Library not loaded: @loader_path/../_solib_darwin/_U_S_Sreverb_Scc_Creverb_Uservice_Ucc_Uproto___Ureverb_Scc/libreverb_service_cc_proto.so
  Referenced from: /Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/libpybind.so
  Reason: image not found
FAIL: reverb//client_test.py

===========Error Summary============
FAIL: reverb//platform/checkpointers_lib_test.py
FAIL: reverb//dataset_test.py
FAIL: reverb//tf_client_test.py
FAIL: reverb//pybind_test.py
FAIL: reverb//server_test.py
FAIL: reverb//rate_limiters_test.py
FAIL: reverb//client_test.py

Looks like you added some proper dylib renaming. Is this still an issue?

oss_build.sh

reverb/cc/platform/default/build_rules.bzl

reverb/cc/platform/default/repo.bzl

reverb/pip_package/build_pip_package.sh

lifeiteng · 2020-12-03T11:25:24Z

I'm trying to Fix python tests

$ bash oss_build.sh --python 3.7

####=======Testing reverb//client_test.py=======####
Traceback (most recent call last):
  File "reverb//client_test.py", line 24, in <module>
    from reverb import client
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/__init__.py", line 27, in <module>
    from reverb import item_selectors as selectors
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/item_selectors.py", line 19, in <module>
    from reverb import pybind
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/pybind.py", line 1, in <module>
    import tensorflow as _tf; from .libpybind import *; del _tf
ImportError: dlopen(/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/libpybind.so, 2): Library not loaded: @loader_path/../_solib_darwin/_U_S_Sreverb_Scc_Creverb_Uservice_Ucc_Uproto___Ureverb_Scc/libreverb_service_cc_proto.so
  Referenced from: /Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/libpybind.so
  Reason: image not found
FAIL: reverb//client_test.py

===========Error Summary============
FAIL: reverb//platform/checkpointers_lib_test.py
FAIL: reverb//dataset_test.py
FAIL: reverb//tf_client_test.py
FAIL: reverb//pybind_test.py
FAIL: reverb//server_test.py
FAIL: reverb//rate_limiters_test.py
FAIL: reverb//client_test.py

Looks like you added some proper dylib renaming. Is this still an issue?

I think I did a dirty fix lifeiteng#1

lifeiteng · 2020-12-03T12:35:06Z

I'm trying to Fix python tests

$ bash oss_build.sh --python 3.7

####=======Testing reverb//client_test.py=======####
Traceback (most recent call last):
  File "reverb//client_test.py", line 24, in <module>
    from reverb import client
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/__init__.py", line 27, in <module>
    from reverb import item_selectors as selectors
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/item_selectors.py", line 19, in <module>
    from reverb import pybind
  File "/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/pybind.py", line 1, in <module>
    import tensorflow as _tf; from .libpybind import *; del _tf
ImportError: dlopen(/Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/libpybind.so, 2): Library not loaded: @loader_path/../_solib_darwin/_U_S_Sreverb_Scc_Creverb_Uservice_Ucc_Uproto___Ureverb_Scc/libreverb_service_cc_proto.so
  Referenced from: /Users/feiteng/opt/anaconda3/lib/python3.7/site-packages/reverb/libpybind.so
  Reason: image not found
FAIL: reverb//client_test.py

===========Error Summary============
FAIL: reverb//platform/checkpointers_lib_test.py
FAIL: reverb//dataset_test.py
FAIL: reverb//tf_client_test.py
FAIL: reverb//pybind_test.py
FAIL: reverb//server_test.py
FAIL: reverb//rate_limiters_test.py
FAIL: reverb//client_test.py

Looks like you added some proper dylib renaming. Is this still an issue?

Fix it in 32c3441

lifeiteng · 2020-12-03T13:20:19Z

bash oss_build.sh --python 3.6

Run Python tests...
===========Running Python tests============
####=======Testing reverb//platform/checkpointers_lib_test.py=======####
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:118] File already exists in database: reverb/cc/schema.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1379] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
run_python_tests.sh: line 32:  7580 Abort trap: 6           ${PYTHON_BIN_PATH} "${test_file}"
FAIL: reverb//platform/checkpointers_lib_test.py
####=======Testing reverb//dataset_test.py=======####
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:118] File already exists in database: reverb/cc/schema.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1379] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
run_python_tests.sh: line 32:  7585 Abort trap: 6           ${PYTHON_BIN_PATH} "${test_file}"
FAIL: reverb//dataset_test.py
####=======Testing reverb//tf_client_test.py=======####
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:118] File already exists in database: reverb/cc/schema.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1379] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
run_python_tests.sh: line 32:  7588 Abort trap: 6           ${PYTHON_BIN_PATH} "${test_file}"
FAIL: reverb//tf_client_test.py
...
===========Error Summary============
FAIL: reverb//platform/checkpointers_lib_test.py
FAIL: reverb//dataset_test.py
FAIL: reverb//tf_client_test.py
FAIL: reverb//pybind_test.py
FAIL: reverb//server_test.py
FAIL: reverb//rate_limiters_test.py
FAIL: reverb//client_test.py

lifeiteng · 2021-01-09T05:59:11Z

bazel_config=""
bazel test -c opt --copt=-mavx ${bazel_config} --test_output=errors //reverb/...

INFO: 8 processes: 8 darwin-sandbox.
INFO: Build completed successfully, 9 total actions
//reverb/cc:chunk_store_test                                    (cached) PASSED in 0.4s
//reverb/cc:client_test                                         (cached) PASSED in 0.2s
//reverb/cc:rate_limiter_test                                   (cached) PASSED in 1.5s
//reverb/cc:reverb_service_impl_test                            (cached) PASSED in 0.7s
//reverb/cc:sampler_test                                        (cached) PASSED in 3.9s
//reverb/cc:table_test                                          (cached) PASSED in 2.7s
//reverb/cc:tensor_compression_test                             (cached) PASSED in 0.1s
//reverb/cc:writer_test                                         (cached) PASSED in 1.2s
//reverb/cc/platform:net_test                                   (cached) PASSED in 0.1s
//reverb/cc/platform:server_test                                (cached) PASSED in 0.2s
//reverb/cc/platform:tfrecord_checkpointer_test                 (cached) PASSED in 1.1s
//reverb/cc/platform:thread_test                                (cached) PASSED in 0.1s
//reverb/cc/selectors:fifo_test                                 (cached) PASSED in 0.2s
//reverb/cc/selectors:heap_test                                 (cached) PASSED in 0.2s
//reverb/cc/selectors:lifo_test                                 (cached) PASSED in 0.2s
//reverb/cc/selectors:prioritized_test                          (cached) PASSED in 0.3s
//reverb/cc/selectors:uniform_test                              (cached) PASSED in 0.2s
//reverb/cc/support:cleanup_test                                (cached) PASSED in 0.1s
//reverb/cc/support:intrusive_heap_test                         (cached) PASSED in 0.1s
//reverb/cc/support:periodic_closure_test                       (cached) PASSED in 0.3s
//reverb/cc/support:queue_test                                  (cached) PASSED in 0.1s
//reverb:dataset_test                                                    PASSED in 20.0s
//reverb:pybind_test                                                     PASSED in 7.2s
//reverb:rate_limiters_test                                              PASSED in 6.9s
//reverb:server_test                                                     PASSED in 7.2s
//reverb:tf_client_test                                                  PASSED in 7.4s
//reverb/platform:checkpointers_lib_test                                 PASSED in 6.8s
//reverb:client_test                                                     PASSED in 8.9s
  Stats over 2 runs: max = 8.9s, min = 8.7s, avg = 8.8s, dev = 0.1s

INFO: Build completed successfully, 9 total actions

but bash oss_build.sh --python 3.6 still has many errors (python tests)

lifeiteng · 2021-01-09T15:51:44Z

bash oss_build.sh --python 3.6

Run Python tests...
===========Running Python tests============
####=======Testing reverb//platform/checkpointers_lib_test.py=======####
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:118] File already exists in database: reverb/cc/schema.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1379] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
run_python_tests.sh: line 32:  7580 Abort trap: 6           ${PYTHON_BIN_PATH} "${test_file}"
FAIL: reverb//platform/checkpointers_lib_test.py
####=======Testing reverb//dataset_test.py=======####
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:118] File already exists in database: reverb/cc/schema.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1379] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
run_python_tests.sh: line 32:  7585 Abort trap: 6           ${PYTHON_BIN_PATH} "${test_file}"
FAIL: reverb//dataset_test.py
####=======Testing reverb//tf_client_test.py=======####
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:118] File already exists in database: reverb/cc/schema.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1379] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
run_python_tests.sh: line 32:  7588 Abort trap: 6           ${PYTHON_BIN_PATH} "${test_file}"
FAIL: reverb//tf_client_test.py
...
===========Error Summary============
FAIL: reverb//platform/checkpointers_lib_test.py
FAIL: reverb//dataset_test.py
FAIL: reverb//tf_client_test.py
FAIL: reverb//pybind_test.py
FAIL: reverb//server_test.py
FAIL: reverb//rate_limiters_test.py
FAIL: reverb//client_test.py

Those are caused by (link ../_solib_darwin/ schema_cc_proto.so twice)

# reverb/cc/checkpointing/BUILD
reverb_cc_proto_library(
    name = "checkpoint_cc_proto",
    srcs = ["checkpoint.proto"],
    deps = [
        "//reverb/cc:schema_cc_proto",
    ],
)


$ otool -L /Users/feiteng/opt/anaconda3/lib/python3.6/site-packages/reverb/libpybind.so

/Users/feiteng/opt/anaconda3/lib/python3.6/site-packages/reverb/libpybind.so:
	bazel-out/darwin-opt/bin/reverb/libpybind.so (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../_solib_darwin/_U_S_Sreverb_Scc_Creverb_Uservice_Ucc_Uproto___Ureverb_Scc/libreverb_service_cc_proto.so (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../_solib_darwin/_U_S_Sreverb_Scc_Scheckpointing_Ccheckpoint_Ucc_Uproto___Ureverb_Scc_Scheckpointing/libcheckpoint_cc_proto.so (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../_solib_darwin/_U_S_Sreverb_Scc_Cschema_Ucc_Uproto___Ureverb_Scc/libschema_cc_proto.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libtensorflow_framework.2.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1677.104.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1677.104.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

$ otool -L /Users/feiteng/opt/anaconda3/lib/python3.6/site-packages/_solib_darwin/_U_S_Sreverb_Scc_Scheckpointing_Ccheckpoint_Ucc_Uproto___Ureverb_Scc_Scheckpointing/libcheckpoint_cc_proto.so

/Users/feiteng/opt/anaconda3/lib/python3.6/site-packages/_solib_darwin/_U_S_Sreverb_Scc_Scheckpointing_Ccheckpoint_Ucc_Uproto___Ureverb_Scc_Scheckpointing/libcheckpoint_cc_proto.so:
	bazel-out/darwin-opt/bin/reverb/cc/checkpointing/libcheckpoint_cc_proto.so (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../_solib_darwin/_U_S_Sreverb_Scc_Cschema_Ucc_Uproto___Ureverb_Scc/libschema_cc_proto.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libtensorflow_framework.2.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

lifeiteng · 2021-01-26T12:51:56Z

I'm busy a few weeks before the Spring Festival, and I will reply later.

tfboyd · 2021-02-24T16:55:30Z

@lifeiteng We are not giving up on this item. We did not setup our system to sync PRs which is why it is taking a bit (ok a lot) more time than one might expect. This is a big thing we really want for Reverb.

This is just a note to let you or anyone seeing this know that this is still being tracked and worked on.

lifeiteng · 2021-02-28T10:55:40Z

TF-Agents and ACME will work on OSX

yes, I have tested them.

lifeiteng · 2021-02-28T10:58:02Z

Should we configure CI(run tests Darwin/Linux) for the repo?

lifeiteng · 2021-02-28T11:03:57Z

@lifeiteng We are not giving up on this item. We did not setup our system to sync PRs which is why it is taking a bit (ok a lot) more time than one might expect. This is a big thing we really want for Reverb.

This is just a note to let you or anyone seeing this know that this is still being tracked and worked on.

OK, I'll replay review comments in 24 hours.

KaleabTessera · 2021-04-06T12:39:33Z

This would be an amazing feature to have!

reverb/cc/platform/default/repo.bzl

thisiscam · 2021-06-17T17:55:21Z

@tfboyd is there still plan to merge this?

andyk · 2021-09-27T22:00:48Z

Hi Reverb team, just wondering if you have any updates on a timeline for merging this?

sabelaraga · 2021-09-30T15:39:10Z

Hey, unfortunately, at the moment we don't have the bandwidth to merge and maintain the extra build for MacOSX.

One possible way forward would be to keep this in a fork. We would definitely add the link to it from the Reverb README.

Robokan · 2022-03-23T01:25:00Z

What is the status of this? Is it possible to try out the fork?

… found

lifeiteng · 2023-01-13T13:25:51Z

close this

lifeiteng mentioned this pull request Nov 21, 2020

Support for macOS #11

Open

lifeiteng force-pushed the osx branch from a7c2901 to 5ecfbff Compare November 21, 2020 11:51

lifeiteng force-pushed the osx branch from 5ecfbff to b6063a3 Compare November 21, 2020 15:42

ebrevdo self-requested a review December 2, 2020 16:38