Accessing the python environment after freeze

[tawjaw]

Hello, I am building an app using PyQt5. And I am using QProcess to run python scripts that are written by the user.
Is there a way to run the QProcess Python command using the frozen python environment?

I don't want the users to have install python or any libraries that they are using.
Write now after the freeze, the QProcess calls Python which runs the system's default installed python.

Thanks

[rokm]

Is there a way to run the QProcess Python command using the frozen python environment?

Not unless you have a python environment installed on the target system.

If you wanted to (re)use the embedded python interpreter that PyInstaller uses to run your frozen script, you would need to spawn the process (with QProcess) using sys.executable, and design your entry-point script to handle special command-line argument that signals that you want it to run the script from given file instead of the main frozen application (for example, implement optional --run-script file.py argument). Then load the provided file and execute the code using exec. Of course, all modules that could be used by such external .py files need to be collected when the main application is frozen (e.g., via hiddenimports).

[rokm]

To illustrate, consider the following example:

program.py

# program.py
import sys


def interactive_mode():
    # Interactive mode in embedded interpreter
    import code
    code.interact()


def run_script():
    # Run script
    with open(sys.argv[2], 'r') as fp:
        code = fp.read()
    exec(code)


def main():
    # Main codepath
    import signal
    from PySide2 import QtCore, QtWidgets  # Avoid importing in other modes

    signal.signal(signal.SIGINT, signal.SIG_DFL)

    app = QtWidgets.QApplication(sys.argv)
    window = QtWidgets.QWidget()
    window.setWindowTitle("Hello world")

    layout = QtWidgets.QVBoxLayout(window)
    button = QtWidgets.QPushButton("Run script")
    layout.addWidget(button)

    def run_script():
        filename, _ = QtWidgets.QFileDialog.getOpenFileName(None, "Run python script", None, "Python files (*.py)")
        if not filename:
            return

        process = QtCore.QProcess()
        if getattr(sys, 'frozen', False):
            # Frozen application
            print("Running script %s in instance of embedded interpreter" % (filename))
            process.start(sys.executable, ['--run-script', filename])
        else:
            print("Running script %s in regular python interpreter" % (filename))
            process.start(sys.executable, [filename, ])

        process.waitForStarted()
        # FIXME: implement busy dialog, etc.
        process.waitForFinished()

        print("Process stdout:\n%r" % process.readAllStandardOutput())
        print("Process stderr:\n%r" % process.readAllStandardError())

    button.clicked.connect(run_script)

    window.show()
    app.exec_()


if __name__ == '__main__':
    if len(sys.argv) == 2 and sys.argv[1] == '--interactive-mode':
        print("Running in interactive mode...")
        interactive_mode()
    elif len(sys.argv) == 3 and sys.argv[1] == '--run-script':
        print("Running in script-execution mode...")
        run_script()
    else:
        print("Running main codepath...")
        main()

Once this is frozen (pyinstaller --clean --noconfirm program.py), you can enter interactive mode by running it as dist/program/program --interactive-mode).

You can also have it run an arbitrary script by running it as dist/program/program --run-script <path-to-py-file>.

Or, if you run without any parameters, it shows a window with button. If you click on it, it allows you to select a script file, and then runs it using spawned instance of itself (with added --run-script argument).

[tawjaw]

thank you very much guys! this works. I tested it now.

would there be a way for a user to add a third library to the runtime? for example if they want to install a python library that they usually do with pip. Is there a way I can give them an option to import any library they install?

[rokm]

would there be a way for a user to add a third library to the runtime? for example if they want to install a python library that they usually do with pip. Is there a way I can give them an option to import any library they install?

You probably just need to ensure that the corresponding location is in sys.path, pretty much the same as you would with regular interpreter. Maybe you could prepare a directory within the frozen bundle that you always add to sys.path, and tell the user to install their libraries in there.

But depending on the complexity of libraries you expect users to install, it might be better to have them install an actual python environment and use that (because then they can install libraries using pip, without having to worry about dependencies, etc.).

[sreekarreddy21]

Hi,

I tried following the above suggestions, but I couldn't execute the 'run_script()' method which just reads a .py file. I am trying to invoke a python script based on the pyinstallers interpreter that further imports other scripts, along with some arguments. exec() isn't supporting these CLI commands. Is there any other way to execute the CLI commands with arguments in run_script()? Also, aren't we running two subprocesses, one using QProcess and the other using exec() this way? Please let me know.
Thanks.

[bwoodsend]

Depends on how you parse your arguments. The short answer is generally no but if you use a parser library which allows to parse the arguments from any list rather than sys.argv then you can use that. If your script is hardcoded to read from sys.argv then you're stuffed.

[xzhacker123]

Hi, I also encountered a similar problem, I am using MacOS, I freeze QProcess into a unix executable file mitm,

After I finished the main program freeze, the .app file was generated. I double-clicked to execute the .app file and the mitm file could not be started. There was no error message.
But when I use the terminal to execute the .app file, I can execute the main program and QProcess normally

self.p = QtCore.QProcess() 
mitmpath = os.path.join(project_dir, 'core/mitmproxy/mitm')

self.textedit.setText(mitmpath) #show the path of the ‘mitm’ being executed.
self.p.start(mitmpath, [‘8888’])

Double-click execution and terminal execution show the same path:
/Users/Alex/py/myProj/dist/myTools.app/Contents/MacOS/core/mitmproxy/mitm

This is my freeze command:

pyi-makespec -D --add-binary="core/mitmproxy/mitm:core/mitmproxy" --icon i.icns --name myTools --windowed main.py
pyinstaller myTools.spec

@rokm Hope to get your help, thank you very much.

[xzhacker123]

Thank you so much for your reply and advice.
I get standardOutput & standardOutputError ：

sh: mitmdump: command not found

/Users/Alex/py/myDebug/dist/myTools.app/Contents/MacOS/core/mitmproxy/mitm-3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) 
[Clang 6.0 (clang-600.0.57)]

This is the source code of my Unix file mitm:

import os
import sys
project_dir = os.path.dirname(os.path.abspath(__file__))
prot = sys.argv[1]
cmd = f"mitmdump -p {prot} -s {os.path.join(project_dir, 'script/addons/meta_proxy.py')} -k"
sys.stdout.write('{}-{}'.format(sys.executable, sys.version))
os.system(cmd)

Did I haven’t package the environment when I freeze ‘mitm’? Because I found that mitm is less than 10M

[rokm]

I think that when you're running from .app bundle, you do not have a shell (and actually, you have a very limited environment, environment-variable-wise). Therefore, mitmdump is not in the PATH and cannot be found. Try using full path to the executable instead.

[xzhacker123]

I import mitmdump method instead of os.system and then freezes mitm to execute successfully.
It seems that the os.system method cannot be easily used in QProcess.

import os
import sys
import shlex
from mitmproxy.tools.main import mitmdump #

project_dir = os.path.dirname(os.path.abspath(__file__))
prot = sys.argv[1]
mitmdump(args=shlex.split(f"-p {prot} -s {os.path.join(project_dir, 'script/addons/meta_proxy.py')} -k"))

[rokm]

It seems that the os.system method cannot be easily used in QProcess.

The problem is not QProcess per-se, but rather that an app bundle has different PATH (or the whole environment, really) than a console (POSIX) application. You can try displaying os.environ['PATH'] in the UI, and you'll see it's different (and does not include the directory where you have mitmdump executable). So you'd either need to use full absolute path to it, or modify os.environ['PATH'] to include its parent directory (either in your bundle code, or in your mitm code).

[xzhacker123]

Thank you very much for your answer, it helped me a lot！