An error occurred when running Bazel to build the PIP package script and install all DeepMind Lab dependencies

[o00000o]

[o00000o]

[tkoeppe]

Hm, it works for me now, could you just try again? Maybe run bazel clean --expunge first?

[o00000o]

Hm, it works for me now, could you just try again? Maybe run bazel clean --expunge first?

What do you mean? Can you describe it more clearly?The second is the dependency package I installed, I don't know if there is any missing, I have tried 3 times.Can you help me？

[GittiHab]

Hi, I ran into a few errors myself and found a few workarounds to finally get this thing installed. Below are the notes I made for myself -- maybe they are also helpful for you or anyone to come. This is for Python 3, should you for some reason require the Python 2.7 module, you can simply exchange py3 with py2 at the respective places.

Disclaimer: I have not yet tested whether the resulting module actually works.

---

Installing DeepMind Lab as a Python Module:

Generally, you can follow the instructions in the GitHub Repo:
https://github.com/deepmind/lab/tree/master/python/pip_package

However, by default, I encountered a few errors. To avoid these you may want to use these tips:

0. Dependencies that might be missing are: python-dev, libsdl2-dev and gettext.
1. Apparently, some hash for the jpeg_archive is wrong. Go into the WORKSPACE file and insert the new hash (printed on the command line) in the respective source.
   In my case it looked someting like this:

http_archive(
    name = "jpeg_archive",
    build_file = "@//:jpeg.BUILD",
    sha256 = "1e9793e1c6ba66e7e0b6e5fe7fd0f9e935cc697854d5737adec54d93e5b3f730",
    strip_prefix = "jpeg-9c",
    urls = ["http://www.ijg.org/files/jpegsrc.v9c.tar.gz"],
)

2. Some numpy header file isn't found. This one was a bit tricky. So here are all the steps I needed to fix this.
   First of all, note that I still use Python 3 as the target installation, so I want to run the command bazel build -c opt --python_version=PY3 //python/pip_package:build_pip_package.
   Before, I also exported the Python3 binary as export PYTHON_BIN_PATH="/usr/bin/python3".

Now, it turns out that for compilation the numpy headers of numpy for Python2.7 are required. So we need to get those from somewhere (maybe it works with Python 3 as well but I wouldn't know which flag to set or button to press). First, I had to install Python 2.7 using sudo apt-get install python-dev (which you might have done in step 0). Then I created a virtual environment with Python 2.7:

virtualenv -p /usr/bin/python venv

Activated it and installed numpy:

source venv/bin/activate
pip install numpy

I did all of this in the DM Lab folder (lab), in case you were wondering.

Now to the complicated part: The header files need to be discovered by the C-compiler (gcc in this case). Luckily, there is this thread on GitHub asking for exactly this. Three steps:

1. Run python in the console (inside the Python 2 virtual environment), import numpy import numpy as np, and print the import directory np.get_include() (source).
2. Head to the WORKSPACE file again and define a new local repository (or just copy and paste this, but take care to use the correct include path you just printed in your console) like this:

new_local_repository(
    name = "numpyh",
    path = "/home/username/somepath/lab/venv/lib/python2.7/site-packages/numpy/core/include",
    build_file_content = """
package(default_visibility = ["//visibility:public"])
cc_library(
    name = "headers",
    hdrs = glob(["**/*.h"])
)
"""
)

3. Use this to modify the build command in BUILD. For me the command throwing errors started on line 956. Here, I added "-Iexternal/numpyh", to copts and "@numpyh//:headers", to deps. So the definition looked like this afterwards:

cc_binary(
    name = "deepmind_lab.so",
    srcs = [
        "public/dmlab.h",
        "python/dmlab_module.c",
    ],
    copts = [
        "-fno-strict-aliasing",  ## ick ick ick
        "-std=c99",
        "-DDEEPMIND_LAB_MODULE_RUNFILES_DIR",
        "-Iexternal/numpyh",
    ],
    linkshared = 1,
    linkstatic = 1,
    visibility = [
        "//python/pip_package:__subpackages__",
        "//python/tests:__subpackages__",
    ],
    deps = [
        ":dmlablib",
        "@python_system//:python",
        "@numpyh//:headers",
    ],
)

Final note: When finally installing the pip module, the filename is spelled differently than in the GitHub instructions (not the upper case DeepMind_Lab instead of lower case):

pip install /tmp/dmlab_pkg/DeepMind_Lab-1.0-py3-none-any.whl --force-reinstall

* other dependencies I had to install from the apt-repo are: libsdl2-dev and gettext. Ignore this if you followed step 0.

[tkoeppe]

@o00000o: can you please describe the error in words rather than with a context-free screenshot? All I was were errors downloading libjpeg, which I can't reproduce. If you have other problems, can you please describe them in some detail?

[o00000o]

Hi, I ran into a few errors myself and found a few workarounds to finally get this thing installed. Below are the notes I made for myself -- maybe they are also helpful for you or anyone to come. This is for Python 3, should you for some reason require the Python 2.7 module, you can simply exchange py3 with py2 at the respective places.

Disclaimer: I have not yet tested whether the resulting module actually works.

Installing DeepMind Lab as a Python Module:

Generally, you can follow the instructions in the GitHub Repo:
https://github.com/deepmind/lab/tree/master/python/pip_package

However, by default, I encountered a few errors. To avoid these you may want to use these tips:

1. Dependencies that might be missing are: python-dev, libsdl2-dev and gettext.
2. Apparently, some hash for the jpeg_archive is wrong. Go into the WORKSPACE file and insert the new hash (printed on the command line) in the respective source.
   In my case it looked someting like this:

http_archive(
    name = "jpeg_archive",
    build_file = "@//:jpeg.BUILD",
    sha256 = "1e9793e1c6ba66e7e0b6e5fe7fd0f9e935cc697854d5737adec54d93e5b3f730",
    strip_prefix = "jpeg-9c",
    urls = ["http://www.ijg.org/files/jpegsrc.v9c.tar.gz"],
)

1. Some numpy header file isn't found. This one was a bit tricky. So here are all the steps I needed to fix this.
   First of all, note that I still use Python 3 as the target installation, so I want to run the command bazel build -c opt --python_version=PY3 //python/pip_package:build_pip_package.
   Before, I also exported the Python3 binary as export PYTHON_BIN_PATH="/usr/bin/python3".

Now, it turns out that for compilation the numpy headers of numpy for Python2.7 are required. So we need to get those from somewhere (maybe it works with Python 3 as well but I wouldn't know which flag to set or button to press). First, I had to install Python 2.7 using sudo apt-get install python-dev (which you might have done in step 0). Then I created a virtual environment with Python 2.7:

virtualenv -p /usr/bin/python venv

Activated it and installed numpy:

source venv/bin/activate
pip install numpy

I did all of this in the DM Lab folder (lab), in case you were wondering.

Now to the complicated part: The header files need to be discovered by the C-compiler (gcc in this case). Luckily, there is this thread on GitHub asking for exactly this. Three steps:

1. Run python in the console (inside the Python 2 virtual environment), import numpy import numpy as np, and print the import directory np.get_include() (source).
2. Head to the WORKSPACE file again and define a new local repository (or just copy and paste this, but take care to use the correct include path you just printed in your console) like this:

new_local_repository(
    name = "numpyh",
    path = "/home/username/somepath/lab/venv/lib/python2.7/site-packages/numpy/core/include",
    build_file_content = """
package(default_visibility = ["//visibility:public"])
cc_library(
    name = "headers",
    hdrs = glob(["**/*.h"])
)
"""
)

1. Use this to modify the build command in BUILD. For me the command throwing errors started on line 956. Here, I added "-Iexternal/numpyh", to copts and "@numpyh//:headers", to deps. So the definition looked like this afterwards:

cc_binary(
    name = "deepmind_lab.so",
    srcs = [
        "public/dmlab.h",
        "python/dmlab_module.c",
    ],
    copts = [
        "-fno-strict-aliasing",  ## ick ick ick
        "-std=c99",
        "-DDEEPMIND_LAB_MODULE_RUNFILES_DIR",
        "-Iexternal/numpyh",
    ],
    linkshared = 1,
    linkstatic = 1,
    visibility = [
        "//python/pip_package:__subpackages__",
        "//python/tests:__subpackages__",
    ],
    deps = [
        ":dmlablib",
        "@python_system//:python",
        "@numpyh//:headers",
    ],
)

Final note: When finally installing the pip module, the filename is spelled differently than in the GitHub instructions (not the upper case DeepMind_Lab instead of lower case):

pip install /tmp/dmlab_pkg/DeepMind_Lab-1.0-py3-none-any.whl --force-reinstall

• other dependencies I had to install from the apt-repo are: libsdl2-dev and gettext. Ignore this if you followed step 0.

Thank you very much. This problem has been bothering me for several days, but there is a new error when I run the "pip install -e ." command. Could you please help me？

[o00000o]

@o00000o: can you please describe the error in words rather than with a context-free screenshot? All I was were errors downloading libjpeg, which I can't reproduce. If you have other problems, can you please describe them in some detail?

Thank you very much, but there is a new problem running command "pip install -e ."

[tkoeppe]

Hm, I'm not sure right now if the PY2 version still works. DeepMind Lab should support PY2, but it's possible that other dependencies have dropped support. Coud you try the PY3 version first?

Regarding Numpy: there is indeed an issue in principle that you should compile DeepMind Lab against the native Numpy headers that are part of the actual system you'll use at runtime, which, if you're in virtualenv, might come from a PIP-installation. In principle there could be conflicts if you also have a system-wide installation of numpy that's different from the one pip would install. Our current Bazel build rules pick the version that's found by the Python interpreter that Bazel runs (https://github.com/deepmind/lab/blob/cf2f5250e1a00ecce37b3480df28c3a5dcd08b57/python_system.bzl#L42-L43).

[o00000o]

Hm, I'm not sure right now if the PY2 version still works. DeepMind Lab should support PY2, but it's possible that other dependencies have dropped support. Coud you try the PY3 version first?

Regarding Numpy: there is indeed an issue in principle that you should compile DeepMind Lab against the native Numpy headers that are part of the actual system you'll use at runtime, which, if you're in virtualenv, might come from a PIP-installation. In principle there could be conflicts if you also have a system-wide installation of numpy that's different from the one pip would install. Our current Bazel build rules pick the version that's found by the Python interpreter that Bazel runs (

https://github.com/deepmind/lab/blob/cf2f5250e1a00ecce37b3480df28c3a5dcd08b57/python_system.bzl#L42-L43

).

Thank you very much. I have solved the problem yesterday.But I have a new error when I install episodic curiosity and its pip dependencies with "pip install -e ." command. Can you help me？

[GittiHab]

@tkoeppe thanks for the hint with the Bazel build rule. It might be that my Bazel command uses the system Python which is essentially a clean installation as I exclusively use virtual environments... however, the headers it required always seemed to be from a numpy version that works with Python 2. Hence, my instruction. But I haven't used Bazel before so I might just be doing something wrong.

@o00000o regarding your screenshot, my guess is that there is a problem with the opencv-python module version. My recommendation is to head into setup.py and replace all >= with == in the install_requires list so that you can exactly reproduce the installation of the authors. However, I ran into further errors afterwards concerning a missing dm_control installation (which is easily worked around -- there is not dm_control version for Python 2 afaik) and then again regarding DM Lab. I can't help you on those though -- just so you are warned ;).

Generally, tkoeppe is right, that you should probably try to directly use Python 3. Are there instructions somewhere for this? Out of the box, it seems like that at least the dependencies need to be edited for this.

[o00000o]

@tkoeppe thanks for the hint with the Bazel build rule. It might be that my Bazel command uses the system Python which is essentially a clean installation as I exclusively use virtual environments... however, the headers it required always seemed to be from a numpy version that works with Python 2. Hence, my instruction. But I haven't used Bazel before so I might just be doing something wrong.

@o00000o regarding your screenshot, my guess is that there is a problem with the opencv-python module version. My recommendation is to head into setup.py and replace all >= with == in the install_requires list so that you can exactly reproduce the installation of the authors. However, I ran into further errors afterwards concerning a missing dm_control installation (which is easily worked around -- there is not dm_control version for Python 2 afaik) and then again regarding DM Lab. I can't help you on those though -- just so you are warned ;).

Generally, tkoeppe is right, that you should probably try to directly use Python 3. Are there instructions somewhere for this? Out of the box, it seems like that at least the dependencies need to be edited for this.

Thank you very much, I installed with Py2 successfully, but I made a syntax error when running the code. I think the code does not support Py2, so I installed with Py3 again, and there is a new problem in bazel build -c opt python/pip_package:build_pip_package, can you help me？

I'll use version install_requires of setup.py as you said.Is there any way to run setup.py directly.

[o00000o]

@tkoeppe感谢 Bazel 构建规则的提示。可能是因为我的 Bazel 命令使用系统 Python，它本质上是一个全新安装，因为我只使用虚拟环境......但是，它所需的标头似乎总是来自可与 Python 2 一起使用的 numpy 版本。因此，我的说明. 但我之前没有使用过 Bazel，所以我可能只是做错了什么。

@o00000o关于你的截图，我的猜测是 opencv-python 模块版本有问题。我的建议是头成setup.py和替换所有>=与==在install_requires列表中，这样就可以精确地再现了作者的安装。但是，之后我遇到了更多关于缺少dm_control安装的错误（这很容易解决 - Python 2 afaik 没有 dm_control 版本），然后再次关于 DM Lab。不过，我无法帮助您解决这些问题-只是警告您；)。

通常，tkoeppe 是正确的，您可能应该尝试直接使用 Python 3。是否有相关说明？开箱即用，似乎至少需要为此编辑依赖项。

Thank you very much for your help in installing and configuring my environment.ow I am using Anaconda to create virtual environment with PY3.7. There is an enum module problem in the installation process. Could you please help me, or have you ever met similar problems before.I have read a lot of information online that enum34 and PY3.7 encapsulate the enum module conflict, but I did not install enum34 module

.

[o00000o]

Hi, I ran into a few errors myself and found a few workarounds to finally get this thing installed. Below are the notes I made for myself -- maybe they are also helpful for you or anyone to come. This is for Python 3, should you for some reason require the Python 2.7 module, you can simply exchange py3 with py2 at the respective places.

Disclaimer: I have not yet tested whether the resulting module actually works.

Installing DeepMind Lab as a Python Module:

Generally, you can follow the instructions in the GitHub Repo: https://github.com/deepmind/lab/tree/master/python/pip_package

However, by default, I encountered a few errors. To avoid these you may want to use these tips:

1. Dependencies that might be missing are: python-dev, libsdl2-dev and gettext.
2. Apparently, some hash for the jpeg_archive is wrong. Go into the WORKSPACE file and insert the new hash (printed on the command line) in the respective source.
   In my case it looked someting like this:

http_archive(
    name = "jpeg_archive",
    build_file = "@//:jpeg.BUILD",
    sha256 = "1e9793e1c6ba66e7e0b6e5fe7fd0f9e935cc697854d5737adec54d93e5b3f730",
    strip_prefix = "jpeg-9c",
    urls = ["http://www.ijg.org/files/jpegsrc.v9c.tar.gz"],
)

1. Some numpy header file isn't found. This one was a bit tricky. So here are all the steps I needed to fix this.
   First of all, note that I still use Python 3 as the target installation, so I want to run the command bazel build -c opt --python_version=PY3 //python/pip_package:build_pip_package.
   Before, I also exported the Python3 binary as export PYTHON_BIN_PATH="/usr/bin/python3".

Now, it turns out that for compilation the numpy headers of numpy for Python2.7 are required. So we need to get those from somewhere (maybe it works with Python 3 as well but I wouldn't know which flag to set or button to press). First, I had to install Python 2.7 using sudo apt-get install python-dev (which you might have done in step 0). Then I created a virtual environment with Python 2.7:

virtualenv -p /usr/bin/python venv

Activated it and installed numpy:

source venv/bin/activate
pip install numpy

I did all of this in the DM Lab folder (lab), in case you were wondering.

Now to the complicated part: The header files need to be discovered by the C-compiler (gcc in this case). Luckily, there is this thread on GitHub asking for exactly this. Three steps:

1. Run python in the console (inside the Python 2 virtual environment), import numpy import numpy as np, and print the import directory np.get_include() (source).
2. Head to the WORKSPACE file again and define a new local repository (or just copy and paste this, but take care to use the correct include path you just printed in your console) like this:

new_local_repository(
    name = "numpyh",
    path = "/home/username/somepath/lab/venv/lib/python2.7/site-packages/numpy/core/include",
    build_file_content = """
package(default_visibility = ["//visibility:public"])
cc_library(
    name = "headers",
    hdrs = glob(["**/*.h"])
)
"""
)

1. Use this to modify the build command in BUILD. For me the command throwing errors started on line 956. Here, I added "-Iexternal/numpyh", to copts and "@numpyh//:headers", to deps. So the definition looked like this afterwards:

cc_binary(
    name = "deepmind_lab.so",
    srcs = [
        "public/dmlab.h",
        "python/dmlab_module.c",
    ],
    copts = [
        "-fno-strict-aliasing",  ## ick ick ick
        "-std=c99",
        "-DDEEPMIND_LAB_MODULE_RUNFILES_DIR",
        "-Iexternal/numpyh",
    ],
    linkshared = 1,
    linkstatic = 1,
    visibility = [
        "//python/pip_package:__subpackages__",
        "//python/tests:__subpackages__",
    ],
    deps = [
        ":dmlablib",
        "@python_system//:python",
        "@numpyh//:headers",
    ],
)

Final note: When finally installing the pip module, the filename is spelled differently than in the GitHub instructions (not the upper case DeepMind_Lab instead of lower case):

pip install /tmp/dmlab_pkg/DeepMind_Lab-1.0-py3-none-any.whl --force-reinstall

• other dependencies I had to install from the apt-repo are: libsdl2-dev and gettext. Ignore this if you followed step 0.

hi,Thank you so much for helping me before.Have you ever encountered this problem RuntimeError: Failed to connect RL API.If so, please help me