DLL resolution in Python 3.8 on Windows has changed

[snorfalorpagus]

The resolution of DLLs has changed in Python 3.8 for Windows.

New in version 3.8: Previous versions of CPython would resolve DLLs using the default behavior for the current process. This led to inconsistencies, such as only sometimes searching PATH or the current working directory, and OS functions such as AddDllDirectory having no effect.

DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library. Note that Windows 7 users will need to ensure that Windows Update KB2533623 has been installed (this is also verified by the installer). (Contributed by Steve Dower in bpo-36085.)

• https://bugs.python.org/issue36085
• https://docs.python.org/3/whatsnew/3.8.html#bpo-36085-whatsnew
• https://docs.python.org/3/library/os.html#os.add_dll_directory

As a result the Appveyor job for 3.8 is failing:

https://ci.appveyor.com/project/sgillies/fiona/branch/master/job/5e9p53tga9ono4j3

python -c "import fiona"
python : Traceback (most recent call last):
At line:1 char:1
+ python -c "import fiona"
+ ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Traceback (most recent call last)::String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
  File "<string>", line 1, in <module>
  File "C:\Python38-x64\Lib\site-packages\fiona\__init__.py", line 85, in <module>
    from fiona.collection import BytesCollection, Collection
  File "C:\Python38-x64\Lib\site-packages\fiona\collection.py", line 9, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
ImportError: DLL load failed while importing ogrext: The specified module could not be found.
Command executed with exception:   File "<string>", line 1, in <module>
  File "C:\Python38-x64\Lib\site-packages\fiona\__init__.py", line 85, in <module>
    from fiona.collection import BytesCollection, Collection
  File "C:\Python38-x64\Lib\site-packages\fiona\collection.py", line 9, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
ImportError: DLL load failed while importing ogrext: The specified module could not be found.

I'm not sure what the correct thing to do here is. For wheels on Windows it's easy to call add_dll_directory with the correct path. But I'm not sure what to do for users not using a wheel - how could we know which path to add? I'd be interested to see how other libraries have handled this.

[achapkowski]

I'm seeing this in Python 3.6.8 as well.

[rbuffat]

@achapkowski I'm not sure that you observe the same issue, as 1) this is related to changes in Python 3.8 and 2) python 3.6 seems not to be the problem in the CI: https://ci.appveyor.com/project/sgillies/fiona/branch/master

The error message "ImportError: DLL load failed while importing ogrext: The specified module could not be found." is normally a general indication that there are problems with the gdal installation. E.g. fiona is not built for the installed gdal, or the gdal dlls are in the wrong place etc.

[achapkowski]

@rbuffat it's the same error message, i downgraded Fiona to 1.7.x and it works.

[gtmaskall]

@achapkowski - were you seeing this (with Python 3.6.8) in either ipython or a jupyter notebook?
I've just had a very similar issue with python 3.6.11 that is resolved by upgrading to python 3.6.12. The weird thing is that it only happened when using either ipython or jupyter notebooks, and not the straight python repl. I documented my research in this SO post.

@rbuffat , @snorfalorpagus - do the tests cover imports from ipython?

[achapkowski]

Yes I saw it in jupyter

[BC14w]

I'm getting similar issue here using gdal:

On Windows, with Python >= 3.8, DLLs are no longer imported from the PATH.
If gdalXXX.dll is in the PATH, then set the USE_PATH_FOR_GDAL_PYTHON=YES environment variable
to feed the PATH into os.add_dll_directory().

I don't understand what is the solution for this. could you elaborate

[rbuffat]

If you encounter an error message of the form of ImportError: DLL load failed while importing ogrext: The specified module could not be found. it is a general error that there is something wrong with your installation, which can have all sorts of causes.

@gtmaskall @achapkowski If you use conda, you should create an issue with the corresponding issue tracker, as this error message is most likely related to the environment and not Fiona itself (not excluding that there might be a bug within Fiona, but it is more likely that it is not).

• https://github.com/ContinuumIO/anaconda-issues/issues
• https://github.com/conda-forge/fiona-feedstock
  If you believe there is something wrong with Fiona, please open a new issue. If you are not using Python >= 3.8 and Windows this issue does not apply.

@BC14w I assume you are using the GDAL Python library. This python library is unrelated to Fiona. If this is the case, please consult GDAL's issue tracker:

• https://github.com/osgeo/gdal

[rootsmusic]

(@gtmaskall Good details in your SO post!) PyPl currently doesn't have a binary installer in Windows for fiona, which is noted within its installation notes.

[GuYith]

I found the same problem when I tried to "import fiona" in Jupyter Notebook.

As a matter of fact, I had no problem importing in Jupyter Notebook when I had configured the environment at the beginning. However, when I tried to use it again a few days later, the same error occurred. And I found, the import outside of Jupyter Notebook is correct.

I support the problem is not related to fiona itself.

[GuYith]

I found the same problem when I tried to "import fiona" in Jupyter Notebook.

As a matter of fact, I had no problem importing in Jupyter Notebook when I had configured the environment at the beginning. However, when I tried to use it again a few days later, the same error occurred. And I found, the import outside of Jupyter Notebook is correct.

I support the problem is not related to fiona itself.

I tried removing the kernel which include fiona, and installed it in Jupyter Notebook again.
The error disappeared. Although I don't know why.