Cannot install Monorepo deps without sourcecode for Dockerfile caching

[michaelshum321]

Description

Hi all,

I'm having difficulty setting up a proper Dockerfile for a service in a monorepo that doesn't require the source code for the services and libraries included in the monorepo.

The monorepo has a shared venv. There is a root pyproject.toml and a few packages/libraries and services nested, each with their own pyproject.toml.

My root pyproject.toml looks like this:

# ./pyproject.toml

[tool.poetry.group.main.dependencies]
# all the shared deps

[tool.poetry.group.my-library.dependencies]
my-library = { path = "packages/my-library", develop = true }

[tool.poetry.group.my-service.dependencies]
my-service = { path = "services/my-service", develop = true }

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

and my service will depend on a library as such:

# services/my-service/pyproject.toml
[tool.poetry.dependencies]
python = "3.11.4"
my-library = { path = "../../packages/my-library" }
# my-service's specific deps

I'm trying to optimize my Dockerbuild so that I don't have to rebuild it everytime I have some source changes. Here's what I'd like to do:

FROM python-base-image

RUN apt-get update && \
  apt-get install -y supervisor poppler-utils && \
  apt-get install --no-install-suggests --no-install-recommends --yes python3-venv pipx

ENV PATH="/root/.local/bin:${PATH}"
RUN pipx install poetry

# Copy poetry and pyproject files over so deps can be installed
COPY ./poetry.lock /home/app/poetry.lock
COPY ./poetry.toml /home/app/poetry.toml
COPY ./pyproject.toml /home/app/pyproject.toml

COPY ./packages/my-library/pyproject.toml /home/app/packages/my-library/pyproject.toml
COPY ./services/my-service/pyproject.toml /home/app/services/my-service/pyproject.toml


WORKDIR /home/app
RUN poetry lock --no-update
RUN poetry install --no-root --no-directory --only my-service,main

# Copy source code over
COPY ./packages /home/app/packages
COPY ./services/my-service /home/app/services/my-service 

My understanding is that --no-root --no-directory will allow me to install my service's deps, including my-library, without needing the source code. This way, I can utilize Docker layers so I don't have to rebuild when I have source code changes.

However, I keep getting this error, and I'm unsure of how to fix it - could I get some guidance?

#0 425.8   • Installing my-library (0.1.0 /home/app/packages/my-library)
#0 432.8 
#0 432.8   ChefBuildError
#0 432.8 
#0 432.8   Backend subprocess exited when trying to invoke build_editable
#0 432.8   
#0 432.8   Traceback (most recent call last):
#0 432.8     File "/root/.local/pipx/venvs/poetry/lib/python3.7/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module>
#0 432.8       main()
#0 432.8     File "/root/.local/pipx/venvs/poetry/lib/python3.7/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main
#0 432.8       json_out["return_val"] = hook(**hook_input["kwargs"])
#0 432.8                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 432.8     File "/root/.local/pipx/venvs/poetry/lib/python3.7/site-packages/pyproject_hooks/_in_process/_in_process.py", line 294, in build_editable
#0 432.8       return hook(wheel_directory, config_settings, metadata_directory)
#0 432.8              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 82, in build_editable
#0 432.8       return WheelBuilder.make_in(
#0 432.8              ^^^^^^^^^^^^^^^^^^^^^
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 88, in make_in
#0 432.8       wb.build(target_dir=directory)
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 118, in build
#0 432.8       self._add_pth(zip_file)
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 147, in _add_pth
#0 432.8       for include in self._module.includes:
#0 432.8                      ^^^^^^^^^^^^
#0 432.8     File "/root/.pyenv/versions/3.11.4/lib/python3.11/functools.py", line 1001, in __get__
#0 432.8       val = self.func(instance)
#0 432.8             ^^^^^^^^^^^^^^^^^^^
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/builder.py", line 97, in _module
#0 432.8       return Module(
#0 432.8              ^^^^^^^
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/utils/module.py", line 75, in __init__
#0 432.8       PackageInclude(
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/utils/package_include.py", line 31, in __init__
#0 432.8       self.check_elements()
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/utils/package_include.py", line 72, in check_elements
#0 432.8       raise ValueError(
#0 432.8   ValueError: /home/app/packages/my-library/my-library does not contain any element
#0 432.8   
#0 432.8 
#0 432.8   at ~/.local/pipx/venvs/poetry/lib/python3.7/site-packages/poetry/installation/chef.py:147 in _prepare
#0 432.8       143│ 
#0 432.8       144│                 error = ChefBuildError("\n\n".join(message_parts))
#0 432.8       145│ 
#0 432.8       146│             if error is not None:
#0 432.8     → 147│                 raise error from None
#0 432.8       148│ 
#0 432.8       149│             return path
#0 432.8       150│ 
#0 432.8       151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
#0 432.8 
#0 432.8 Note: This error originates from the build backend, and is likely not a problem with poetry but with my-library (0.1.0 /home/app/packages/my-library) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 --editable "/home/app/packages/my-library"'.

If I copy the sources over before running poetry install, I don't have the issue - but I can't utilize Docker layering.

Thank you guys very much!

PS - I'm also trying to use Docker's cache mount, but Poetry seems to ignore it per the Poetry Config. but that's for another day.

ENV POETRY_CACHE_DIR='/home/.cache/pypoetry'
# ...
RUN --mount=type=cache,target=$POETRY_CACHE_DIR poetry install --no-root --no-directory --only my-service,main

Workarounds

COPY the source for my library and service prior to poetry install - but this is undesired and slow

Poetry Installation Method

pipx

Operating System

Debian

Poetry Version

Poetry 1.5.1

Poetry Configuration

In docker container:


# poetry config --list
cache-dir = "/root/ cache/pypoetry" # /home/ .cache/pypoetry
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir)/virtualenvs" # /home/.cache/pypoetry/virtualenvs/
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "[project_name)-py{python_version}"

### Python Sysconfig

_No response_

### Example pyproject.toml

_No response_

### Poetry Runtime Logs

```bash session
n/a

[georgettica]

having something similar in latest aswell. originally the version was 1.5.1

[abn]

@georgettica can you try without this line?

RUN poetry lock --no-update

Locking requires a metadata build for path dependencies, and this in turn requires the source of the package.

[michaelshum321]

I think you meant to tag me but I'll try it out! if I want to install my deps per the lockfile, is there an alternative to doing that? @abn

[abn]

Oops, yes. I am guessing you want some form of layer caching here. You can sort of achieve that, the issue you are hitting is that for installing an editable package, Poetry needs to build wheel metadata. This could change in the future.

Here is a made-up example of working around this issue.

/tmp/foo$ tree .
.
├── bar
│   ├── bar
│   │   └── __init__.py
│   ├── poetry.lock
│   ├── pyproject.toml
│   └── README.md
├── Containerfile
├── foo
│   └── __init__.py
├── poetry.lock
├── pyproject.toml
└── README.md

4 directories, 9 files

FROM docker.io/python:3.13

RUN python -m pip install --root-user-action=ignore -q poetry

RUN install -d foo

RUN install -d foo/bar

COPY pyproject.toml poetry.lock foo/.

COPY bar/pyproject.toml bar/poetry.lock bar/README.md foo/bar/.
COPY bar/bar/__init__.py foo/bar/bar/.

WORKDIR foo

RUN poetry install --no-root

Note the addition of README.md and single __init__.py file.

[michaelshum321]

thanks @abn one more thing, would it be possible to have only a global/root-level poetry.lock? or do dependencies within the monorepo must have their own poetry.lock? thank you!

[abn]

Anything you manage and distribute independently should have its own pyproject.toml file. If you do not care about distributing them or developing them independently, I would imaging you can just use something like this

packages = [
    { include = "package", from="src" },
    { include = "lib_a", from="services/lib_a/src"}
]

And define all metadata in your root pyproject.toml. But all of this is dependent on how you use your monorepo.

A while ago, I created #2270. No progress has been officially been made yet. You can check out the plugin a user mentioned there (not affiliated with the Poetry Team).