smoketests fails from build tree if python3.9 is not already installed

[jeremyd2019]

Updated smoketests.py in mingw-w64-python from the version in this repo and attempted to build with check enabled. Got this output:

check output

==> Starting check()...
Looking in links: c:/Users/ADMINI~1/AppData/Local/Temp/tmplubvt8of
Processing c:/users/administrator/appdata/local/temp/tmplubvt8of/setuptools-56.0.0-py3-none-any.whl
Processing c:/users/administrator/appdata/local/temp/tmplubvt8of/pip-21.1.3-py3-none-any.whl
Installing collected packages: setuptools, pip
  WARNING: The scripts pip3.9.exe and pip3.exe are installed in 'C:\Users\Administrator/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-21.1.3 setuptools-56.0.0
Collecting wheel
  Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.36.2
Processing c:/users/admini~1/appdata/local/temp/tmplfi0sqnk
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Building wheels for collected packages: cwrapper
  Building wheel for cwrapper (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: T:/mingw-w64-python3.9/src/build-aarch64/python.exe -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"'; __file__='"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d C:/Users/Administrator/AppData/Local/Temp/pip-wheel-mnp_u_w2
       cwd: C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/
  Complete output (13 lines):
  running bdist_wheel
  running build
  running build_ext
  building 'cwrapper' extension
  creating build
  creating build/temp.mingw_aarch64-3.9
  clang -fno-strict-aliasing -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -D__USE_MINGW_ANSI_STDIO=1 -IC:/msys32/clangarm64/include/ncurses -IT:/mingw-w64-python3.9/src/Python-3.9.6/Include -IT:/mingw-w64-python3.9/src/build-aarch64 -c cwrapper.c -o build/temp.mingw_aarch64-3.9/cwrapper.o
  writing build/temp.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.def
  creating build/lib.mingw_aarch64-3.9
  clang -shared -Wl,--enable-auto-image-base -pipe -Wl,--dynamicbase -pipe -Wl,--dynamicbase -pipe -Wl,--dynamicbase -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -D__USE_MINGW_ANSI_STDIO=1 -IC:/msys32/clangarm64/include/ncurses -s build/temp.mingw_aarch64-3.9/cwrapper.o build/temp.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.def -L. -L. -lpython3.9 -lm -lversion -lshlwapi -o build/lib.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.pyd
  lld: error: unable to find library -lpython3.9
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  error: command 'C:\\msys32\\clangarm64\\bin/clang.exe' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for cwrapper
  Running setup.py clean for cwrapper
Failed to build cwrapper
Installing collected packages: cwrapper
    Running setup.py install for cwrapper ... error
    ERROR: Command errored out with exit status 1:
     command: T:/mingw-w64-python3.9/src/build-aarch64/python.exe -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"'; __file__='"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record C:/Users/Administrator/AppData/Local/Temp/pip-record-_d0pvkfe/install-record.txt --single-version-externally-managed --compile --install-headers C:/msys32/clangarm64/include/python3.9/cwrapper
         cwd: C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/
    Complete output (13 lines):
    running install
    running build
    running build_ext
    building 'cwrapper' extension
    creating build
    creating build/temp.mingw_aarch64-3.9
    clang -fno-strict-aliasing -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -D__USE_MINGW_ANSI_STDIO=1 -IC:/msys32/clangarm64/include/ncurses -IT:/mingw-w64-python3.9/src/Python-3.9.6/Include -IT:/mingw-w64-python3.9/src/build-aarch64 -c cwrapper.c -o build/temp.mingw_aarch64-3.9/cwrapper.o
    writing build/temp.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.def
    creating build/lib.mingw_aarch64-3.9
    clang -shared -Wl,--enable-auto-image-base -pipe -Wl,--dynamicbase -pipe -Wl,--dynamicbase -pipe -Wl,--dynamicbase -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -DMS_DLL_ID="3.9" -DFIELD3=6000 -DORIGINAL_FILENAME="libpython3.9.dll" -D__USE_MINGW_ANSI_STDIO=1 -IC:/msys32/clangarm64/include/ncurses -s build/temp.mingw_aarch64-3.9/cwrapper.o build/temp.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.def -L. -L. -lpython3.9 -lm -lversion -lshlwapi -o build/lib.mingw_aarch64-3.9/cwrapper.cp39-mingw_aarch64.pyd
    lld: error: unable to find library -lpython3.9
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command 'C:\\msys32\\clangarm64\\bin/clang.exe' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: T:/mingw-w64-python3.9/src/build-aarch64/python.exe -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"'; __file__='"'"'C:/Users/Administrator/AppData/Local/Temp/pip-req-build-h2rjvhl0/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record C:/Users/Administrator/AppData/Local/Temp/pip-record-_d0pvkfe/install-record.txt --single-version-externally-managed --compile --install-headers C:/msys32/clangarm64/include/python3.9/cwrapper Check the logs for full command output.
E...............
======================================================================
ERROR: test_c_ext_build (__main__.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "T:\mingw-w64-python3.9\src\smoketests.py", line 218, in test_c_ext_build
    subprocess.check_call(
  File "T:/mingw-w64-python3.9/src/Python-3.9.6/Lib/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['T:/mingw-w64-python3.9/src/build-aarch64/python.exe', '-m', 'pip', 'install', 'C:/Users/ADMINI~1/AppData/Local/Temp/tmplfi0sqnk']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 16 tests in 77.263s

FAILED (errors=1)
==> ERROR: A failure occurred in check().

It seems to have been missing the -L option to find the libpython3.9.dll.a import library

[jeremyd2019]

All of the smoketests passed after installing the package on clangarm64 though, including building the cwrapper.

[naveen521kk]

I guess that running that directly from build tree is a special case when building the stdlib. So I think the -L isn't added there. I don't know what happens exactly, I will check later today.

[jeremyd2019]

I think this would only happen if python3.9 is not already installed, so libpython3.9.dll.a would not be on the search path.

[lazka]

My guess would be the relocation in sysconfigdata. We use sys.prefix there, which might be wrong in case it's not installed. I think we might also need to use sys.base_prefix for some cases since sys.prefix changes in virtualenvs... but that's a different problem.