recursive package dependencies

[dylex]

Problem

It seems that since #11, there is now a recursive dependency of jupyterlab_pygments on jupyterlab on jupyter_server on nbconvert on jupyterlab_pygments. Even though most of these are run-time dependencies (not build-time), this makes things complicated both for packagers and non-wheel installs on pip. Attempting a pip install --no-binary :all: jupyterlab results in:

LookupError: https://files.pythonhosted.org/packages/69/8e/8ae01f052013ee578b297499d16fcfafb892927d8e41c1a0054d2f99a569/jupyterlab_pygments-0.2.2.tar.gz (from https://pypi.org/simple/jupyterlab-pygments/) (requires-python:>=3.7) is already being built: jupyterlab-pygments from https://files.pythonhosted.org/packages/69/8e/8ae01f052013ee578b297499d16fcfafb892927d8e41c1a0054d2f99a569/jupyterlab_pygments-0.2.2.tar.gz (from nbconvert>=6.4.4->jupyter-server~=1.16->jupyterlab)

I'm not saying this is necessarily this package's fault, of course, and probably most people will be fine with wheels, but it seems worth bringing up.

Proposed Solution

If I understand #11 correctly it adds an extension. Is there a way to split off this extension part (and the dependency on jupyterlab) into a different package? Alternatively, maybe the nbconvert dependency could be removed or shifted onto a different package?

Additional context

This came up when trying to update to the latest jupyterlab_pygments with spack, which builds everything from source separately and needs to resolve dependencies, so the recursive dependencies prevent them from being installed at all (even though they are only runtime dependencies). The result is that an old version of jupyterlab_pygments (0.1.2) is installed instead, and this seems to be what some other packaging systems are doing, too.

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[kloczek]

Gentle ping .. any update or advise how to deal with this issue?? 🤔

[martinRenou]

Thanks for the ping.

Even though most of these are run-time dependencies (not build-time)

I'm not sure I understand the original issue. jupyterlab-pygments does not depend on jupyterlab, does it?

It has no dependency: https://github.com/jupyterlab/jupyterlab_pygments/blob/main/pyproject.toml#L28

The is a build-time dependency though, on JupyterLab, maybe that's the issue? jupyterlab-pygments requires jupyterlab at build time to build itself as a labextension. But there are apparently discussions to split the build tools out of jupyterlab.

[kloczek]

I'm not sure I understand the original issue. jupyterlab-pygments does not depend on jupyterlab, does it?

Indeed jupyterlab-pygments do not depend on jupyterlab but build of the jupyterlab-pygments depends on it

jupyterlab_pygments/pyproject.toml

Lines 1 to 4 in c3380f5

	[build-system]
	requires = [
	"hatchling>=1.5.0",
	"jupyterlab>=4.0.0,<5",

[kloczek]

Just tested build without that line and indeed looks it is used

+ /usr/bin/python3 -sBm build -w --no-isolation
* Getting build dependencies for wheel...
* Building wheel...
INFO:hatch_jupyter_builder.utils:Running jupyter-builder
INFO:hatch_jupyter_builder.utils:Building with hatch_jupyter_builder.npm_builder
INFO:hatch_jupyter_builder.utils:With kwargs: {'build_cmd': 'build:prod', 'npm': ['jlpm']}
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/usr/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/usr/lib/python3.10/site-packages/hatchling/build.py", line 58, in build_wheel
    return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard'])))
  File "/usr/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 147, in build
    build_hook.initialize(version, build_data)
  File "/usr/lib/python3.10/site-packages/hatch_jupyter_builder/plugin.py", line 94, in initialize
    raise e
  File "/usr/lib/python3.10/site-packages/hatch_jupyter_builder/plugin.py", line 89, in initialize
    build_func(self.target_name, version, **build_kwargs)
  File "/usr/lib/python3.10/site-packages/hatch_jupyter_builder/utils.py", line 106, in npm_builder
    npm_cmd = normalize_cmd(npm)
  File "/usr/lib/python3.10/site-packages/hatch_jupyter_builder/utils.py", line 210, in normalize_cmd
    raise ValueError(msg)
ValueError: Aborting. Could not find cmd (jlpm) in path. If command is not expected to be in user's path, use an absolute path.

[kloczek]

One sec .. this message is about jlpm command.
One sec will try to add it wo build env.

[kloczek]

Yeah .. and jlpm is part of the jupyterlab so as long that command is needed it is in loop on dependencies and this one is not worse because is not loop in install=-time dependencies but build as well.

[kloczek]

As this module is only sphinx theme I think that only solution is to remove it from jupyterlab and/or replace it by some other theme. 🤔

[kloczek]

I've double checked this ring of dependencies and it is a bit longer.

To build jupyterlab-pygments needs to be installed jupyterlab which on install time requires jupyter-server.
To install jupyter-server needs to be installed on install-time nbconvert and than on install-time nbconvert needs jupyterlab-pygments.

I'm going to check is is possible to build nbconvert without jupyterlab-pygments install-time dependencies 🤔

[kloczek]

And now I cannot build nbconvert ` because 502 gateway error on https://unpkg.com/ 😞

+ /usr/bin/python3 -sBm build -w --no-isolation
* Getting build dependencies for wheel...
* Building wheel...
Downloading CSS: https://unpkg.com/@jupyterlab/[email protected]/style/index.css
Failed to download css from https://unpkg.com/@jupyterlab/[email protected]/style/index.css: HTTP Error 520:
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/usr/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/usr/lib/python3.10/site-packages/hatchling/build.py", line 58, in build_wheel
    return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard'])))
  File "/usr/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 147, in build
    build_hook.initialize(version, build_data)
  File "/home/tkloczko/rpmbuild/BUILD/nbconvert-7.16.3/hatch_build.py", line 85, in initialize
    _get_css_files()
  File "/home/tkloczko/rpmbuild/BUILD/nbconvert-7.16.3/hatch_build.py", line 75, in _get_css_files
    _get_css_file(template_name, url, filename)
  File "/home/tkloczko/rpmbuild/BUILD/nbconvert-7.16.3/hatch_build.py", line 58, in _get_css_file
    raise OSError(msg) from None
OSError: Need CSS to proceed.

ERROR Backend subprocess exited when trying to invoke build_wheel

BTW is it possible to perform such build 100% offline? 🤔
If yes what needs to be installed locally?

[kloczek]

Correction it is http error 520 which is IIRC something with proxy.

[martinRenou]

jlpm is basically yarn, we could technically use yarn instead of jlpm wherever it's used, but jupyterlab is still needed as a dependency for building the labextension https://github.com/jupyterlab/jupyterlab_pygments/blob/main/package.json#L34.

Now, I could propose a change where the source distribution published on PyPi contains the labextension pre-built already. This would allow not depending on jupyterlab for building the source dist.
One could argue that the source dist would not really be the original source as some part of it is pre-built, but I personally don't have a strong feeling about this.

BTW is it possible to perform such build 100% offline?

As long as the build uses yarn/jlpm, we'll need an internet connection to download the npm dependencies.

[kloczek]

Now, I could propose a change where the source distribution published on PyPi contains the labextension pre-built already. This would allow not depending on jupyterlab for building the source dist.

No please don't do that.
Lessons from from few scandals exposing compromised manually generated dist resources caused that +3 years ago I made the decision to use as much as possible VCS resources as input source tar balls as it isl possible. Result is that on packaging all my python modules packaging procedures +95% are using now autogenerated from git tags tar balls

[tkloczko@pers-jacek SPECS]$ grep "Source:.*%{VCS}/" python-*spec -l | wc -l; ls -1 python-*spec |wc -l
1187
1243

Because I'm using the same methodology in case of other packages last scandal with xz caused that that many distros where forced to go back instantly to prev versions and because I've been using tar ball from git tag I've been sitting and watching everyone screaming around.

I'm 100% sure that it needs to be some clear way without pypi ..

[kloczek]

Why at all jupyterlab-pygments needs on pep517 build jupyterlab? 🤔
Looking on content seens like thois module packages only simple .py files and few .js/.json .. (my understanding is that for what is needed yarn.

BTW yarn .. are you aware of note from https://github.com/yarnpkg/yarn/blob/master/README.md? 🤔

[martinRenou]

Why at all jupyterlab-pygments needs on pep517 build jupyterlab? 🤔

As I said before:

but jupyterlab is still needed as a dependency for building the labextension https://github.com/jupyterlab/jupyterlab_pygments/blob/main/package.json#L34.

---

BTW yarn .. are you aware of note from https://github.com/yarnpkg/yarn/blob/master/README.md? 🤔

We're using yarn v3

[martinRenou]

You'll need to wait that #40 (comment) is resolved