the influence for limit the memory of the python interpreter to use pyinstaller

[zfy108]

my program depends on large memory, before using pyinstaller, there is some questions for memory error(1.0G), after using pyinstaller, there is no question any more (1.5G). after using pyinstaller ,my program can run successfully. but i don't konw why.
I want to know how to influence python interpreter for pyinstaller to enlarge memory
any infomation will be appreciated.

[bwoodsend]

I can't reproduce this. Can you give an example piece of code that does? My own attempt to reproduce this (below) works perfectly ok.

# test.py
# Allocate 10GB of memory.
a = bytes(10_000_000_000)
print(f"`a` is {len(a):3,} bytes in size.")

> pyinstaller test.py 
...
> ./dist/test/test 
`a` is 10,000,000,000 bytes in size.

[zfy108]

thanks for your reply
you need to use a 32-bit python interpreter and windows operating system.
I will be appreciated for your reply

[bwoodsend]

Again, I can't reproduce on 32-bit Python. You won't be allowed to use more than 2GB on anything 32 bit because a 32 bit integer is too small to hold a >2GB memory address but that's nothing to do with PyInstaller - it happens under regular unfrozen Python too.

[rokm]

If you are hitting the memory limit with unfrozen python but not with PyInstaller-frozen application, then the difference is that PyInstaller's bootloader is built with /LARGEADDRESSAWARE flag enabled, whereas python interpreter (python.exe) is not.

E.g.,

> dumpbin /headers pyinstaller\PyInstaller\bootloader\Windows-32bit\run.exe | more

Microsoft (R) COFF/PE Dumper Version 14.16.27045.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file pyinstaller\PyInstaller\bootloader\Windows-32bit\run.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               4 number of sections
        610625B7 time date stamp Sun Aug  1 06:40:23 2021
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             122 characteristics
                   Executable
                   Application can handle large (>2GB) addresses
                   32 bit word machine

OPTIONAL HEADER VALUES
             10B magic # (PE32)
           14.28 linker version
           21200 size of code
           1E200 size of initialized data
               0 size of uninitialized data
            8F94 entry point (00408F94)
-- More  --

vs:

> dumpbin /headers c:\Users\Rok\AppData\Local\Programs\Python\Python39-32\python.exe | more

Microsoft (R) COFF/PE Dumper Version 14.16.27045.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file c:\Users\Rok\AppData\Local\Programs\Python\Python39-32\python.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               5 number of sections
        612D3AA4 time date stamp Mon Aug 30 22:08:04 2021
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             102 characteristics
                   Executable
                   32 bit word machine

OPTIONAL HEADER VALUES
             10B magic # (PE32)
           14.29 linker version
            1400 size of code
           15200 size of initialized data
               0 size of uninitialized data
            12CC entry point (004012CC)
            1000 base of code
-- More  --

Although I wonder if we are supposed to declare support for large address spaces, when the shared libraries we use (e.g., python39.dll) do not...

[zfy108]

thanks for your reply
I know it won't be allowed to use more than 2GB on anything 32 bit because a 32 bit integer is too small to hold a >2GB memory address.
but， there is some questions for memory error(1.0G) on my program. whats more ,there is no question on my program after using pyInstaller. and the memory can reach 1.5G which is < 2.0G. therefore, i believe pyinstaller has an influence for the memory of the python interpreter.
I will be appreciated for your reply

[zfy108]

If you are hitting the memory limit with unfrozen python but not with PyInstaller-frozen application, then the difference is that PyInstaller's bootloader is built with /LARGEADDRESSAWARE flag enabled, whereas python interpreter (python.exe) is not.

E.g.,

> dumpbin /headers pyinstaller\PyInstaller\bootloader\Windows-32bit\run.exe | more

Microsoft (R) COFF/PE Dumper Version 14.16.27045.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file pyinstaller\PyInstaller\bootloader\Windows-32bit\run.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               4 number of sections
        610625B7 time date stamp Sun Aug  1 06:40:23 2021
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             122 characteristics
                   Executable
                   Application can handle large (>2GB) addresses
                   32 bit word machine

OPTIONAL HEADER VALUES
             10B magic # (PE32)
           14.28 linker version
           21200 size of code
           1E200 size of initialized data
               0 size of uninitialized data
            8F94 entry point (00408F94)
-- More  --

vs:

> dumpbin /headers c:\Users\Rok\AppData\Local\Programs\Python\Python39-32\python.exe | more

Microsoft (R) COFF/PE Dumper Version 14.16.27045.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file c:\Users\Rok\AppData\Local\Programs\Python\Python39-32\python.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               5 number of sections
        612D3AA4 time date stamp Mon Aug 30 22:08:04 2021
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             102 characteristics
                   Executable
                   32 bit word machine

OPTIONAL HEADER VALUES
             10B magic # (PE32)
           14.29 linker version
            1400 size of code
           15200 size of initialized data
               0 size of uninitialized data
            12CC entry point (004012CC)
            1000 base of code
-- More  --

Although I wonder if we are supposed to declare support for large address spaces, when the shared libraries we use (e.g., python39.dll) do not...

thanks a lot
I agree with you
I am appreciated for your reply