virtualenv site-packages not automatically included in sys.path

[snickell]

I'm trying to use a virtualenv with pycall. Python is loaded correctly, but sys.path does not include the site-packages from the virtualenv. Instead site-packages from the underlying python interpreter (in /opt/homebrew) is used.

For example, lets say I have a new version of urllib3 in my virtualenv:

ENV['PYTHON'] = "/Users/seth/src/code-dot-org/.venv/bin/python"
require 'pycall'
PyCall.import_module 'urllib3'
# => <module 'urllib3' from '/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/urllib3/__init__.py'>

# WRONG: Its using the system version of urllib3, not the version in the virtualenv

If I manually set PYTHONPATH, it will load from the virtualenv:

ENV['PYTHON'] = "/Users/seth/src/code-dot-org/.venv/bin/python"
require 'pycall'
ENV['PYTHONPATH'] = '/Users/seth/src/code-dot-org/.venv/lib/python3.12/site-packages'
PyCall.import_module 'urllib3'
# => <module 'urllib3' from '/Users/seth/src/code-dot-org/.venv/lib/python3.12/site-packages/urllib3/__init__.py'>

# CORRECT: Its using the new version of urllib from the venv

The problem with setting PYTHONPATH manually, is now I cannot invoke 3rd party python-using tools like aws cli from my process, because the manual PYTHONPATH will point them at my virtualenv.

[snickell]

Output from PYCALL_DEBUG_FIND_LIBPYTHON=1:

DEBUG(find_libpython) find_libpython("/Users/seth/src/code-dot-org/.venv/bin/python")
DEBUG(find_libpython) investigate_python_config("/Users/seth/src/code-dot-org/.venv/bin/python")
DEBUG(find_libpython) Candidate: /opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/Python
DEBUG(find_libpython) Trying to dlopen: /opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/Python
DEBUG(find_libpython) dlopen("/opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/Python") = #<Fiddle::Handle:0x0000000120b93798>

virtualenv is linking to /opt/homebrew python:

ls -l /Users/seth/src/code-dot-org/.venv/bin | grep python
# => python -> /opt/homebrew/opt/[email protected]/bin/python3.12

[snickell]

Further investigation:

1. sys.executable is returning the path to ruby: e.g. '/Users/seth/.rbenv/versions/3.0.5/bin/ruby', probably because we are using libpython not a python exe.
2. However, site.py from cpython (github source here) which is invoked automatically at python startup says that:

If a file named "pyvenv.cfg" exists one directory above sys.executable, sys.prefix and sys.exec_prefix are set to that directory and it is also checked for site-packages (sys.base_prefix and sys.base_exec_prefix will always be the "real" prefixes of the Python installation).

I have noticed that "editable" package installations (site-packages/*.pth files) are also not registered into the path from pycall.rb. This is consistent with site.py being invoked and not finding pyenv.cfg (present in my .venv directory). As a result, local site-packages are not properly registered.

I wonder: is there a way to set sys.executable to the local python bin? This would make site.py function as expected.

[snickell]

Here's a "semi proper" workaround that anyone else facing this can do:

path_to_your_venv = '/Users/seth/src/code-dot-org/.venv/lib/python3.12/site-packages' # your venv path here
require 'pycall'
site = PyCall.import_module('site')
site.addsitedir(path_to_your_venv)

This will properly setup your venv's site-packages, including registering .pth files (editable installs) etc.