Regressions in windows debug CI DLL load failed while importing _rclpy_pybind11

[sloretz]

🕵️‍♂️ I think is causing test lots of test regressions in windows CI.
https://ci.ros2.org/view/nightly/job/nightly_win_deb/2134/

Traceback (most recent call last):

  File "C:\Python38\lib\runpy.py", line 194, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File "C:\Python38\lib\runpy.py", line 87, in _run_code

    exec(code, run_globals)

  File "C:\Python38\lib\site-packages\pytest\__main__.py", line 5, in <module>

    raise SystemExit(pytest.console_main())

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 185, in console_main

    code = main()

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 143, in main

    config = _prepareconfig(args, plugins)

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 318, in _prepareconfig

    config = pluginmanager.hook.pytest_cmdline_parse(

  File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 265, in __call__

    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)

  File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec

    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)

  File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 55, in _multicall

    gen.send(outcome)

  File "C:\Python38\lib\site-packages\_pytest\helpconfig.py", line 100, in pytest_cmdline_parse

    config: Config = outcome.get_result()

  File "C:\Python38\lib\site-packages\pluggy\_result.py", line 60, in get_result

    raise ex[1].with_traceback(ex[2])

  File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 39, in _multicall

    res = hook_impl.function(*args)

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 1003, in pytest_cmdline_parse

    self.parse(args)

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 1283, in parse

    self._preparse(args, addopts=addopts)

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 1172, in _preparse

    self.pluginmanager.load_setuptools_entrypoints("pytest11")

  File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 287, in load_setuptools_entrypoints

    plugin = ep.load()

  File "C:\Python38\lib\importlib\metadata.py", line 77, in load

    module = import_module(match.group('module'))

  File "C:\Python38\lib\importlib\__init__.py", line 127, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load

  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load

  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load

  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked

  File "C:\Python38\lib\site-packages\_pytest\assertion\rewrite.py", line 170, in exec_module

    exec(co, module.__dict__)

  File "C:\ci\ws\install\Lib\site-packages\launch_testing_ros\__init__.py", line 20, in <module>

    from . wait_for_topics import WaitForTopics

  File "C:\Python38\lib\site-packages\_pytest\assertion\rewrite.py", line 170, in exec_module

    exec(co, module.__dict__)

  File "C:\ci\ws\install\Lib\site-packages\launch_testing_ros\wait_for_topics.py", line 21, in <module>

    from rclpy.executors import SingleThreadedExecutor

  File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 36, in <module>

    from rclpy.client import Client

  File "C:\ci\ws\install\Lib\site-packages\rclpy\client.py", line 22, in <module>

    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy

  File "C:\ci\ws\install\Lib\site-packages\rclpy\impl\implementation_singleton.py", line 32, in <module>

    rclpy_implementation = import_c_library('._rclpy_pybind11', package)

  File "C:\ci\ws\install\Lib\site-packages\rpyutils\import_c_library.py", line 39, in import_c_library

    return importlib.import_module(name, package=package)

  File "C:\Python38\lib\importlib\__init__.py", line 127, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

ImportError: DLL load failed while importing _rclpy_pybind11: The specified module could not be found.

The C extension 'C:\ci\ws\install\Lib\site-packages\rclpy\_rclpy_pybind11.cp38-win_amd64.pyd' failed to be imported while being present on the system. Please refer to 'https://docs.ros.org/en/{distro}/Guides/Installation-Troubleshooting.html#import-failing-even-with-library-present-on-the-system' for possible solutions

---
Finished <<< ament_package [1.11s]	[ with test failures ]

It seems that Windows can't find the correct path to _rclpy_pybind11 for the tests added with ros2/launch_ros#274.

Can I ask you to take a quick look? @sloretz You may have more context about what's going on.

Originally posted by @Blast545 in #814 (comment)

[sloretz]

@Blast545 - initial impressions are the test failures are caused by ros2/launch_ros#274 , but also not really that PRs fault.

Interesting to me is the rclpy tests are failing with errors like:

________________________ ERROR collecting test session ________________________
C:\Python38\lib\site-packages\pluggy\_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
C:\Python38\lib\site-packages\pluggy\_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
C:\Python38\lib\site-packages\_pytest\python.py:200: in pytest_collect_file
    module: Module = ihook.pytest_pycollect_makemodule(path=path, parent=parent)
C:\Python38\lib\site-packages\pluggy\_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
C:\Python38\lib\site-packages\pluggy\_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
..\..\install\Lib\site-packages\launch_testing\pytest\hooks.py:161: in pytest_pycollect_makemodule
    entrypoint = find_launch_test_entrypoint(path)
..\..\install\Lib\site-packages\launch_testing\pytest\hooks.py:155: in find_launch_test_entrypoint
    return getattr(path.pyimport(), 'generate_test_description', None)
C:\Python38\lib\site-packages\py\_path\local.py:704: in pyimport
    __import__(modname)
..\..\src\ros2\rclpy\rclpy\test\__init__.py:20: in <module>
    assert 'rclpy' not in sys.modules, 'rclpy should not have been imported before running tests'
E   AssertionError: rclpy should not have been imported before running tests

I haven't tested any of this, but my current guess is launch_testing is installing a pytest hook. That hook seems to be found when rclpy is tested, and that hook loads a plugin from launch_testing_ros. This is probably because the windows builds use --merge-install so by the time rclpy is tested, all packages upstream and downstream have been built. ros2/launch_ros#274 probably leads to that pytest hook importing rclpy, which means rclpy's assertions about it not being imported when its own tests are run are violated.

A way to test that theory is to remove this line https://github.com/ros2/launch_ros/blob/7a848e5da6f8774c28a537f4dcaf35d3b771d7bf/launch_testing_ros/launch_testing_ros/__init__.py#L20 and see if the job passes. That might be a good workaround for now.

That doesn't explain why ament_package's tests fail though.

[Blast545]

For ament_package the error is the following:

Starting >>> ament_package
Traceback (most recent call last):
  File "C:\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Python38\lib\site-packages\pytest\__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 185, in console_main
    code = main()
  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "C:\Python38\lib\site-packages\_pytest\helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "C:\Python38\lib\site-packages\pluggy\_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 1172, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 287, in load_setuptools_entrypoints
    plugin = ep.load()
  File "C:\Python38\lib\importlib\metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "C:\Python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\Python38\lib\site-packages\_pytest\assertion\rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "C:\ci\ws\install\Lib\site-packages\launch_testing_ros\__init__.py", line 20, in <module>
    from . wait_for_topics import WaitForTopics
  File "C:\Python38\lib\site-packages\_pytest\assertion\rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "C:\ci\ws\install\Lib\site-packages\launch_testing_ros\wait_for_topics.py", line 21, in <module>
    from rclpy.executors import SingleThreadedExecutor
  File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 36, in <module>
    from rclpy.client import Client
  File "C:\ci\ws\install\Lib\site-packages\rclpy\client.py", line 22, in <module>
    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
  File "C:\ci\ws\install\Lib\site-packages\rclpy\impl\implementation_singleton.py", line 32, in <module>
    rclpy_implementation = import_c_library('._rclpy_pybind11', package)
  File "C:\ci\ws\install\Lib\site-packages\rpyutils\import_c_library.py", line 39, in import_c_library
    return importlib.import_module(name, package=package)
  File "C:\Python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: DLL load failed while importing _rclpy_pybind11: The specified module could not be found.
The C extension 'C:\ci\ws\install\Lib\site-packages\rclpy\_rclpy_pybind11.cp38-win_amd64.pyd' failed to be imported while being present on the system. Please refer to 'https://docs.ros.org/en/{distro}/Guides/Installation-Troubleshooting.html#import-failing-even-with-library-present-on-the-system' for possible solutions
--- stderr: ament_package
Traceback (most recent call last):

  File "C:\Python38\lib\runpy.py", line 194, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File "C:\Python38\lib\runpy.py", line 87, in _run_code

    exec(code, run_globals)

  File "C:\Python38\lib\site-packages\pytest\__main__.py", line 5, in <module>

    raise SystemExit(pytest.console_main())

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 185, in console_main

    code = main()

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 143, in main

    config = _prepareconfig(args, plugins)

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 318, in _prepareconfig

    config = pluginmanager.hook.pytest_cmdline_parse(

  File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 265, in __call__

    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)

  File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec

    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)

  File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 55, in _multicall

    gen.send(outcome)

  File "C:\Python38\lib\site-packages\_pytest\helpconfig.py", line 100, in pytest_cmdline_parse

    config: Config = outcome.get_result()

  File "C:\Python38\lib\site-packages\pluggy\_result.py", line 60, in get_result

    raise ex[1].with_traceback(ex[2])

  File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 39, in _multicall

    res = hook_impl.function(*args)

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 1003, in pytest_cmdline_parse

    self.parse(args)

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 1283, in parse

    self._preparse(args, addopts=addopts)

  File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 1172, in _preparse

    self.pluginmanager.load_setuptools_entrypoints("pytest11")

  File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 287, in load_setuptools_entrypoints

    plugin = ep.load()

  File "C:\Python38\lib\importlib\metadata.py", line 77, in load

    module = import_module(match.group('module'))

  File "C:\Python38\lib\importlib\__init__.py", line 127, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load

  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load

  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load

  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked

  File "C:\Python38\lib\site-packages\_pytest\assertion\rewrite.py", line 170, in exec_module

    exec(co, module.__dict__)

  File "C:\ci\ws\install\Lib\site-packages\launch_testing_ros\__init__.py", line 20, in <module>

    from . wait_for_topics import WaitForTopics

  File "C:\Python38\lib\site-packages\_pytest\assertion\rewrite.py", line 170, in exec_module

    exec(co, module.__dict__)

  File "C:\ci\ws\install\Lib\site-packages\launch_testing_ros\wait_for_topics.py", line 21, in <module>

    from rclpy.executors import SingleThreadedExecutor

  File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 36, in <module>

    from rclpy.client import Client

  File "C:\ci\ws\install\Lib\site-packages\rclpy\client.py", line 22, in <module>

    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy

  File "C:\ci\ws\install\Lib\site-packages\rclpy\impl\implementation_singleton.py", line 32, in <module>

    rclpy_implementation = import_c_library('._rclpy_pybind11', package)

  File "C:\ci\ws\install\Lib\site-packages\rpyutils\import_c_library.py", line 39, in import_c_library

    return importlib.import_module(name, package=package)

  File "C:\Python38\lib\importlib\__init__.py", line 127, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

ImportError: DLL load failed while importing _rclpy_pybind11: The specified module could not be found.

The C extension 'C:\ci\ws\install\Lib\site-packages\rclpy\_rclpy_pybind11.cp38-win_amd64.pyd' failed to be imported while being present on the system. Please refer to 'https://docs.ros.org/en/{distro}/Guides/Installation-Troubleshooting.html#import-failing-even-with-library-present-on-the-system' for possible solutions

---
Finished <<< ament_package [1.11s]	[ with test failures ]

For some reason, when ament_package is tested _rclpy_pybind11 is tried to be importing for rclpy. That doesn't make too much sense to me, but in all the packages that depend on rclpy the error occurs.

[ivanpauno]

assert 'rclpy' not in sys.modules, 'rclpy should not have been imported before running tests'

is that assert also really needed?
I don't think it's an issue if rclpy got already imported.