[ci] macos_regular job fails: 'nbconvert --execute requires the jupyter_client package

[jameslamb]

Description

I've seen the following error on every run of the macos_regular Azure DevOps CI job in the last 12 hours.

[NbConvertApp] Converting notebook interactive_plot_example.ipynb to notebook
Traceback (most recent call last):
File "/Users/runner/runners/2.211.1/miniforge/envs/test-env/lib/python3.7/site-packages/traitlets/traitlets.py", line 645, in get
value = obj._trait_values[self.name]
KeyError: 'kernel_manager_class'

ImportError: cannot import name 'secure_write' from 'jupyter_core.paths' (/Users/runner/runners/2.211.1/miniforge/envs/test-env/lib/python3.7/site-packages/jupyter_core/paths.py)

ImportError: nbconvert --execute requires the jupyter_client package: pip install jupyter_client

Reproducible example

Example build from #5533: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=13799&view=logs&j=30b4943b-8000-5673-e2d8-b78860d98666&t=323d22c9-4fd4-5f79-2982-d153c24bec2f

Additional Comments

I suspect that this might be related to some combination of the following:

• the macos_regular job runs on Python 3.7 and uses conda + conda-forge
• conda-forge recently dropped support for Python 3.7
  • Dropping Python 3.7 conda-forge/conda-forge-pinning-feedstock#2623
  • Dropping Python 3.7 conda-forge/conda-forge-pinning-feedstock#2623
• at least one of the the many dependencies of ipywidgets or nbconvert put up a new release recently, which didn't get a Python 3.7 build as a result of conda-forge dropping that support
• the environment we're getting on this CI job has inconsistent versions of all the dependencies needed to nbvconvert Ipython notebooks

[jameslamb]

Yeah I can see on that build that we're getting very old versions of some notebook dependencies.

nbconvert-5.6.1            |     pyhd8ed1ab_2         373 KB  conda-forge
nbformat-5.7.0             |     pyhd8ed1ab_0         106 KB  conda-forge
notebook-5.7.11            |   py37hf985489_0         7.9 MB  conda-forge

For example, nbconvert 5.6.1 is from October 2019 (release history).

[jameslamb]

Updating to Python 3.9 in #5570 resulted in conda-forge returning the newest verisons of notebook, ipywidgets, and their dependencies. That resolved this issue.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.