dev: Create a Python package, build Docker images from it

[chishm]

What type of PR is this?

• feature
• dev (Internal development)

What this PR does / why we need it:

Build a Python package (sdist and wheel) that can be used to install Mealie, including the frontend SPA files. The package is then used to create the Docker images used for testing and release.

The advantage here is that the frontend code (Javascript) and backend code (Python) are both platform-independent, and so the same package can be installed into both the ARM64 and AMD64 Docker images. This saves having to build them twice.

Eventually, Mealie will be completely installable from PyPI as a Python package, as an option in addition to the Docker images. But more work is needed to achieve that goal and further PRs will follow this one. Until then, this PR stands on its own to benefit the E2E testing and CI release process, and can be merged now.

Changes in detail

The backend by default looks for the frontend SPA files in mealie/frontend (actually, the frontend sub-directory of the mealie Python package). This can still be overridden by setting STATIC_FILES.

There is a new top-level task in Taskfile.yml, py:package, that builds the Python packages. The generated frontend files are copied from frontend/dist to mealie/frontend by the task py:package:copy-frontend (called by py:package).

During the package build, a requirements.txt file is generated from Poetry's lock file. This ensures that exact pinned dependencies are installed.

I have documented the 2 ways to build Mealie in building-packages.md.

A few extra stages are added to the Dockerfile to allow setting an additional build context which holds the pre-built Python package (and associated requirements.txt). This can be done by using the docker build argument --build-context packages=dist, where dist is the directory holding a pre-built package. When this context is not set, the Dockerfile will build the package itself. When it is set, the build steps will be skipped.

The steps of the py:package task are replicated in the Dockerfile's frontend and backend build stages. This ensures that running docker-compose up will give the same result, regardless of whether the Docker image was built from a pre-built package or from scratch.

The GitHub workflows have been modified to make use of this. There is a new partial-package.yml workflow that incorporates the frontend build, adds a backend build, and stores the resulting package as an artifact for later jobs. The E2E testing workflow uses the package for its Docker build, ensuring that it gets tested. The partial-builder workflow also uses the package for its Docker image build and push, saving the effort of rebuilding the frontend and backend for each platform.

Finally, to make the Python package a bit nicer to use (not needing to modify PYTHONPATH), I've incorporated #4323 into this PR.

Which issue(s) this PR fixes:

This is a few tasks from discussion 4322.

Testing

• I successfully tested the built Python package on my development machine, outside of a Docker container.
• I used task docker:prod to build from scratch and test the Docker image.
• I used task docker:build-from-package to rebuild the Docker image (after deleting the previous one), followed by task docker:prod to test it.
• I ran the new GitHub workflow for pull requests on my fork of the mealie repository.
  • Have a look under the Artifacts header at backend-dist for the Python sdist, wheel, and requirements.

[boc-the-git]

Without wanting to speak to the overall validity of this PR (I've not looked in any details), I just wanted to note that the below is not an objective for the Mealie project:

Eventually, Mealie will be completely installable from PyPI as a Python package, as an option in addition to the Docker images.

[chishm]

Without wanting to speak to the overall validity of this PR (I've not looked in any details), I just wanted to note that the below is not an objective for the Mealie project:

Eventually, Mealie will be completely installable from PyPI as a Python package, as an option in addition to the Docker images.

I understand, but is it an anti-objective? That is, if a PR improves things overall (e.g. faster CI, or easier dev environment setup) will it be rejected just because it happens to make it easier to install Mealie from a package?

[chishm]

Now that #4329 has been merged, I've rebased this PR onto the latest mealie-next branch.

[boc-the-git]

Without wanting to speak to the overall validity of this PR (I've not looked in any details), I just wanted to note that the below is not an objective for the Mealie project:

Eventually, Mealie will be completely installable from PyPI as a Python package, as an option in addition to the Docker images.

I understand, but is it an anti-objective? That is, if a PR improves things overall (e.g. faster CI, or easier dev environment setup) will it be rejected just because it happens to make it easier to install Mealie from a package?

Sorry, I've been meaning to respond. No, not an anti-objective. However, if it requires any meaningful additional maintenance time from the team, it's not what we want to do. We are all for improvements! But we have limited time.

[chishm]

Sorry, I've been meaning to respond. No, not an anti-objective. However, if it requires any meaningful additional maintenance time from the team, it's not what we want to do. We are all for improvements! But we have limited time.

That's completely fair; I don't want to take your time away from adding features and fixing bugs. With this PR I've tried to keep the old developer workflows the same (to not burden developers with having to learn a new way of doing things), and I've made sure the GitHub workflows are completely automated, without needing any extra maintainer oversight.

[chishm]

There's some workflow checks that aren't completing ("Frontend and End-to-End Tests"), because they're defined in the mealie-next branch in this repo but have don't exist in the python-package-github branch in my fork (the tests are reworked into "End-to-End Tests"). See this example PR in my fork for an example of all the workflows running successfully.

[github-actions]

This PR is stale because it has been open 45 days with no activity.

[chishm]

Hi @boc-the-git and @hay-kot,

I'd like to check what are the chances of this getting merged? I'm happy to break it up into smaller pieces, if that makes it easier to review?

[chishm]

Thank you for the approval, @hay-kot . It looks like mealie-next has had a lot of development since I last updated this branch (which is great to see :-), but it meant I had to rebase and fix this branch, causing the approval to be invalidated.

As before, I've got the full GitHub workflow running and passing on my fork.

One thing to note is the GitHub action that installs Poetry now installs Poetry 2.0.1 (as of when I write this), which has some backwards incompatibilities with Poetry 1.8.3 (the old default). I've fixed this by installing the Poetry plugin poetry-plugin-export , which is needed for generating the requirements.txt and is no longer included by default. The way it's installed should mean that people using Poetry 1.8 are still OK, but in future you might want to consider adding an explicit requires-poetry = ">=2.0" and poetry-plugin-export = ">=1.9" to the pyproject.toml (maybe when using the new project.dependencies support).

[michael-genson]

LGTM too. If you could just fix the merge conflict (just update the version to 2.6.0) I can re-approve.

I'd just do it but then I couldn't approve it

[chishm]

Thanks for taking a look. I've rebased to fix the merge conflict and the latest test run succeeded.

[michael-genson]

Waiting on some admin tinkering; we have to remove the old status check requirements and add the new ones, since GH looks for required status checks by name

[chishm]

Waiting on some admin tinkering; we have to remove the old status check requirements and add the new ones, since GH looks for required status checks by name

Oh, that sounds like a bit of annoying work. Should I try to rework this to fit the old naming scheme?

[michael-genson]

No that's okay, it shouldn't be too bad!