cargo run --features no_python --bin metta-repl before and after metta-repl

[gl-yziquel]

I do not know how much of a bug this is, but the user experience is suboptimal.

I just built the repo with cargo build.

1. If I launch metta-repl directly from target/debug, I get:

Fatal Error: ModuleNotFoundError: No module named 'hyperon'

2. If I then launch cargo run --features no_python --bin metta-repl, I get the REPL.

3. Then, if I launch metta-repl again, directly from target/debug, it works too.

The initialisation of the hyperon module from rust seems... weird.

[vsbogd]

Python module is used from REPL by default mostly because main users of MeTTa are using Python to integrate with neural net libraries. Thus if REPL is launched without "no_python" feature and Python part of the repo is not built and installed properly then REPL fails.

Then, if I launch metta-repl again, directly from target/debug, it works too.

After launching REPL with "no_python" property built binary is kept in target directory and second launch doesn't try loading Python part.

I agree that for pure Rust users it is surprising behavior and better way would be to replace "no_python" feature by "python" feature which enables using Python module. Also we have #432 which prevents REPL from starting in complex Python setup.

[gl-yziquel]

Well, I must say the build instructions to build from source are rather painful if you do not want to put a docker container in the middle. Such issues are, in my opinion, a byproduct of trying to build from source with the current instructions.

[luketpeterson]

One idea is that the repl's default config could build support for both and switch at runtime depending on the availability of a HyperonPy. The only downside would be a longer build time and larger binary size for the repl. But in dev that likely won't matter and, in any setting where it would matter, it's easy enough to configure.

That said, I wouldn't be against changing the default to no_python either.

[gl-yziquel]

That said, I wouldn't be against changing the default to no_python either.

I don't have objections to python being bound to metta. My concern is merely buildability. Each time I come back to MeTTa, some weird uncanny aspect of the build instructions blow up. And, here, the presence of an unactivated python 3.6 in pyenv got me in these problems with python not bound to MeTTa.

[gl-yziquel]

@vsbogd This is very much unclear. Build from source out of docker and with pyenv, I experience the following pain points:

1. When I launch python in my pyenv shell 3.11.8, I'm indeed able to import the python module I've built from source. import hyperon followed by hyperon.__version__ works.

2. cargo build --features python.git still creates a metta-repl that says Fatal Error: ModuleNotFoundError: No module named 'hyperon'.

How do I get the python module to be picked up by metta-repl ? Because the built module is clearly accessible from the python REPL (managed by pyenv) but not from the metta-repl cli. How is that set up ?

[vsbogd]

The point of PR #751 is to make REPL start without integrating Python by default. I.e. one can clone repo and after that execute:
cargo run to run MeTTa REPL.

If you need Python inside REPL you need build and install MeTTa Python library under your pyenv environment first. Usually it the following list of commands should be enough:

mkdir build
cd build
cmake ..
make
cd ..
python3 -m pip install -e ./python[dev]
cargo build --features python

To debug your issue further could you please:

1. do the steps above under pyenv you mentioned starting from clean repo, use git clean -xfd to clean up it
2. if it doesn't work then post the result of the following commands:

which python
python --version
which python3
python3 --version
python3 -m pip list | grep hyperon
ldd target/debug/metta-repl

[gl-yziquel]

Here goes:

mini-me@virtucon ~/h/b/hephaestus> pyenv shell 3.11.8
mini-me@virtucon ~/h/b/hephaestus> pyenv version
3.11.8 (set by PYENV_VERSION environment variable)
mini-me@virtucon ~/h/b/hephaestus> git clone https://github.com/trueagi-io/hyperon-experimental.git hyperon-experimental-main
mini-me@virtucon ~/h/b/hephaestus> cd hyperon-experimental-main
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> mkdir build
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> cd build
mini-me@virtucon ~/h/b/h/h/build (main)> cmake ..
-- The C compiler identification is GNU 13.2.0
-- The CXX compiler identification is GNU 13.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- HYPERONC_INSTALL_PREFIX = /home/mini-me/home/backyard/hephaestus/hyperon-experimental-main/build/hyperonc-install
-- HYPERONC_BINARY_DIR = /home/mini-me/home/backyard/hephaestus/hyperon-experimental-main/build/hyperonc-prefix/src/hyperonc-build
-- HYPERONPY_BINARY_DIR = /home/mini-me/home/backyard/hephaestus/hyperon-experimental-main/build/hyperonpy-prefix/src/hyperonpy-build
-- Configuring done (1.2s)
-- Generating done (0.0s)
-- Build files have been written to: /home/mini-me/home/backyard/hephaestus/hyperon-experimental-main/build
mini-me@virtucon ~/h/b/h/h/build (main)> make                
[  6%] Creating directories for 'hyperonc'       
[ 12%] No download step for 'hyperonc'                  
[ 18%] No update step for 'hyperonc'
[ 25%] No patch step for 'hyperonc'   
[ 31%] Performing configure step for 'hyperonc'
-- The C compiler identification is GNU 13.2.0
-- The CXX compiler identification is GNU 13.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
WARN: *** Conan 1 is legacy and on a deprecation path ***
[...]
-- Found optional-lite: 3.5.0 (found version "3.5.0") 
-- Configuring done (7.2s)
-- Generating done (0.0s)
-- Build files have been written to: /home/mini-me/home/backyard/hephaestus/hyperon-experimental-main/build/hyperonpy-prefix/src/hyperonpy-build
[ 87%] Performing build step for 'hyperonpy'
[ 50%] Building CXX object CMakeFiles/hyperonpy.dir/hyperonpy.cpp.o
[100%] Linking CXX shared module /home/mini-me/home/backyard/hephaestus/hyperon-experimental-main/python/hyperonpy.cpython-311-x86_64-linux-gnu.so
[100%] Built target hyperonpy
[ 93%] Performing install step for 'hyperonpy'
Skipping install step.
[100%] Completed 'hyperonpy'
[100%] Built target hyperonpy
mini-me@virtucon ~/h/b/h/h/build (main)> cd ..
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> python -c "import sys; print(sys.executable)"
/home/mini-me/.pyenv/versions/3.11.8/bin/python
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> python3 -m pip install -e ./python[dev]
[...]
Successfully installed build-0.10.0 hyperon-0.2.0+g8c1b66d4 iniconfig-2.0.0 pluggy-1.5.0 pytest-7.3.2
WARNING: There was an error checking the latest version of pip.
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> python3
Python 3.11.8 (main, Feb 16 2024, 14:33:34) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hyperon
>>> hyperon.__version__
'0.2.0+g8c1b66d4'
>>> 
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> cargo build --features python
[...]
   Compiling clap v4.5.10
   Compiling pep440_rs v0.3.12
   Compiling pyo3-macros v0.19.2
   Compiling metta-repl v0.2.0 (/home/mini-me/home/backyard/hephaestus/hyperon-experimental-main/repl)
   Compiling hyperon v0.2.0 (/home/mini-me/home/backyard/hephaestus/hyperon-experimental-main/lib)
   Compiling hyperonc v0.2.0 (/home/mini-me/home/backyard/hephaestus/hyperon-experimental-main/c)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 24.08s
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> target/debug/metta-repl 
Fatal Error: ModuleNotFoundError: No module named 'hyperon'
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main) [255]> which python
/home/mini-me/.pyenv/shims/python
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> python --version
Python 3.11.8
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> which python3
/home/mini-me/.pyenv/shims/python3
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> python3 --version
Python 3.11.8
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> python3 -m pip list | grep hyperon
hyperon                      0.2.0+g8c1b66d4     /home/mini-me/home/backyard/hephaestus/hyperon-experimental-main/python
WARNING: There was an error checking the latest version of pip.
mini-me@virtucon ~/h/b/h/hyperon-experimental-main (main)> ldd target/debug/metta-repl 
        linux-vdso.so.1 (0x00007ffceb72b000)
        libpython3.11.so.1.0 => /lib/x86_64-linux-gnu/libpython3.11.so.1.0 (0x0000753633000000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000075363393d000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000753632c00000)
        /lib64/ld-linux-x86-64.so.2 (0x0000753634142000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000753633852000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x0000753633833000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x0000753633806000)

I believe the libpython linkage information should instead point to ~/.pyenv/versions/3.11.8/lib/libpython3.11.so.

This is all the more confusing as PYO3_PRINT_CONFIG=1 cargo build shows lib_dir=/home/mini-me/.pyenv/versions/3.11.8/lib.

EDIT: Using patchelf --replace-needed libpython3.11.so.1.0 /home/mini-me/.pyenv/versions/3.11.8/lib/libpython3.11.so target/debug/metta-repl did the trick. Though it's mighty ugly. And undocumented in the readme.

[vsbogd]

I believe the libpython linkage information should instead point to ~/.pyenv/versions/3.11.8/lib/libpython3.11.so.

Right. The issue is REPL executable contains only library file name libpython3.11.so.1.0 not the absolute path. On the other hand PyEnv library is not located on the default system library loading path. Thus when REPL executable is loaded it is linked to the some library which is located on the system loading path. It is a duplicate of the #432 but here the system has same library version in a system path.

I would suggest closing this and continue discussion in #432.

[vsbogd]

EDIT: Using patchelf --replace-needed libpython3.11.so.1.0 /home/mini-me/.pyenv/versions/3.11.8/lib/libpython3.11.so target/debug/metta-repl did the trick. Though it's mighty ugly. And undocumented in the readme.

Yes, I think we need fixing #432 rather then mention this in README.md.

[gl-yziquel]

EDIT: Using patchelf --replace-needed libpython3.11.so.1.0 /home/mini-me/.pyenv/versions/3.11.8/lib/libpython3.11.so target/debug/metta-repl did the trick. Though it's mighty ugly. And undocumented in the readme.

Yes, I think we need fixing #432 rather then mention this in README.md.

I believe you could do both. I really lost months on that, with multitasking and all...