feat: full edx-platform setup with tutor dev launch -m ...

[kdmccormick]

Preface

This is the part of effort to make it easier to work with mounted repositories.

@regisb , previously we had agreed that it'd be good to generate the .egg-info folder automatically using an init task. I realized the other day, though, that we may as well do the same thing for installing node_modules and regenerating static assets too! With this change, we can setup a dev env with a bind-mounted edx-platform in one command.

In the future, I am still interesting in removing the need to reinstall node_modules and regenerate static assets; for now, though, I think that this is an improvement, and its user interface is compatible with the long-term vision.

I did my best to rework the Open edX development docs around the simplified setup. Hopefully they feel simpler now.... very open to feedback here, though.

Finally, I expect that bindmount-canary might be controversial, but I'm not sure of a better solution--open to other suggestions 😄

Description

Before this commit, setting up an edx-platform development environment took multiple steps:

  tutor dev launch
  tutor dev run --mount=/path/to/edx-platform lms bash
  >> pip install -e .
  >> npm clean-install
  >> openedx-assets build --env=dev

This commit moves the steps under run into an init task, which is automatically run by launch. Thus, setup is now one command:

  tutor dev launch --mount=edx-platform

These extra init steps are only applicable when bind-mounting edx-platform (because bind-mounting the repository overrides some important artifacts that exist on the image, which must be re-generated). Thus, the new init tasks exists early if it detects that it is not operating on a bind-mounted repository.

Finally, we try to simplify the Open edX development docs so that it is clearer how bind-mounting fits into the development process.

Part of:

• Tutor Adoption 4: Devstack Parity: Improve Dev Workflows openedx-unsupported/wg-developer-experience#146
  Closes:
• Ensure that egg-info folder is created when edx-platform is mounted openedx-unsupported/wg-developer-experience#152

This works around (but does not close) these related issues:

• Move node_modules outside of edx-platform in Tutor's openedx image openedx-unsupported/wg-developer-experience#150
• Move artifacts outside of edx-platform in Tutor's openedx image openedx/wg-devops#22

Changed docs

[kdmccormick]

Something I just realized but don't have time to confirm right now: If -job services don't bindmount themes, then that'll cause an issue with static asset compilation for custom theme users, right?

[regisb]

This is a great step forward, I love it!

[regisb]

These latter two steps will be performed every time we run launch with a bind-mount. I don't have a better alternative, so let's keep it that way, but we should keep it in mind.

[kdmccormick]

Something we could do, eventually, would be to use a hash to ensure that these steps are not run extraneously.

• node_modules would be easy: upon running npm install, create a node_modules/.package_lock_hash file, containing a hash of package-lock.json. If package-lock.json matches node_modules/.package_lock_hash, then we can skip node_modules installation.
• assets would be more complicated, but still doable. The scripts/build-assets.sh has the means to a compile a list of all asset input files. Saving the combined hash of those somewhere would allow us to skip asset compilation when assets are unchanged.

Thing is, this is just reimplementing what Docker already does. That's why I'd rather move node_modules and assets outside of the bind-mount if at all possible.

[regisb]

If -job services don't bindmount themes, then that'll cause an issue with static asset compilation for custom theme users, right?

Yes, this is correct. So we should bind-mount the themes folder as well.

But we're not going to be able to solve all issues related to running a custom edx-platform repo. For instance, if the edx-platform python requirements are modified, then in theory we should also run pip install -- but we are not going to, because it would take forever.

[kdmccormick]

Yes, this is correct. So we should bind-mount the themes folder as well.

Sounds good, I will add that to the PR.

in theory we should also run pip install -- but we are not going to, because it would take forever.

Even if it were quick, I'm not sure we would want run pip install, because the changes to /openedx/venv would only exist in ephemeral container memory. The pip updates would be missing in other containers (cms, lms-worker, etc) and they would be blown away from lms whenever the platform got brought down.

So, changes to pip requirements require an image rebuild. I don't think that's a bad thing, I just think it means we need to make sure that developers can change pip requirements and then rebuild as little of the image as possible: openedx-unsupported/wg-developer-experience#162

[kdmccormick]

Alright @regisb , this is ready for another look.

[regisb]

This looks great to me! Do you want to add anything else before we merge?

[kdmccormick]

Nope, merge away 🚀