Pyproject toml

[oerc0122]

• Migration to pyproject.toml with meson.
• Migrated away from versioneer to custom version.py script.
• Migrated CI workflows to use cibuildwheel
• Move utilities related to build into <root>/build_utils.
• Updated testing workflows to add appropriate compilers.

Addresses #323

[ajjackson]

This is looking really promising! It's nice having the compilation stuff in the meson config file and then a fairly clean toml for main project.

We may have to ditch versioneer; will be worth seeing what other meson projects are using?

[ajjackson]

Have a look at this branch: https://github.com/pace-neutrons/Euphonic/tree/pyproject-toml-rebase

• I rebased on master (only tweak needed really was to add toolz to pyproject.toml as it was in the desynced setup.py)
• Hacked out versioneer for now. (Haven't even removed the files yet, just the references.)
• Added a lot of boilerplate to meson.build so that it includes all they Python source files

The last part seems to be essential but looks pretty ugly. I think the preferred pattern is to cascade it into more meson.build files and load them with the subdir directive?

Anyway it seems to build and euphonic-dos runs so that's a good start.

I can force-push to your branch if that's helpful/allowed

[ajjackson]

Some reflections on the overall process that got us here:

• This was quite painful and took more time than expected

  • None of the problems can be blamed on pyproject.toml / PEP517 / PEP518; they do actually simplify things. The trouble is finding the correct configuration for the "simple" way of doing things...

• We had hoped that by using (python-)meson the platform-dependent code would be pushed upstream and we wouldn't need to worry about it any more. In practice what happened is that platform-dependent fiddling was moved from the setup.py to the Github runners. That should be good news as it means we are less tied to particular build environments and e.g. a Mac user should now be able to build against non-Homebrew OpenMP if they can obtain it.

• But this necessitated another round of work on the Github Actions. Troubleshooting Github Actions is a bit soul-destroying because it can take a lot of cursing and iteration to get to the correct "simple, obvious-looking" result.

• There were some gotchas with the Github-provided Windows runners:

  • after activating the msvc environment, we got an inconsistent cl.exe and link.exe on the PATH, because a "Git" directory is given priority and ships with GNU link.exe
  • the solution is to use absolute paths, but then we discovered that different runners can have slightly different MSVC versions leading to flaky builds/tests. We have ended up with some compromise code that makes a few assumptions and tries to locate a reasonable MSVC installation before setting CC and CC_LD to absolute paths. It will need updating when the runners are significantly changed.
  • For a while we tried this third-party action which activates MSVC and sets some useful environment variables: https://dvdhrm.github.io/2021/04/21/meson-msvc-github-actions/ https://github.com/bus1/cabuild
    • But it's totally unofficial and has 1 star on Github. How is there not a "standard" tool for this? What are other C-compiling Windows projects doing instead?

• I'm puzzled that we had to dig so much and invent so much process/machinery to handle the task of versioning sdist and wheel built with python-meson. It seems like something "everyone" needs to do, yet the information is scattered around a bit. As far as I can see, "everyone" has either a) something simpler/less-capable than this (i.e. no "dirty" versioning) or b) something even more complicated (that suits their complicated project.) Perhaps it's worth a blog post or something to help people (i.e. our future selves) navigate our solution and the surrounding issues; but I suspect neither of us wants to look at this again for a while.

• It's a pity that Versioneer is currently coupled to setuptools, because it looks like their new "put these lines in your pyproject.toml" deployment is a nice improvement on the "vendoring" approach Euphonic was previously using.

• Making PRs from forks is usually a pretty painless development workflow, but when applied to Github Actions it is awful. PRs would not trigger test runs; we had to post links to action runs on the other repository; discrepancies in the git tags gave misleading results.

In future if contributing CI tweaks from a fork I would make a PR to another branch on the fork for purposes of discussion and automatic CI.

[mducle]

Troubleshooting Github Actions is a bit soul-destroying

Yes, totally agree... I found that using tmate-actions was helpful but it's still quite a painful process...

[ajjackson]

Yeah, that wouldn't have helped with everything but might have been handy in a couple of places 🤔

[ajjackson]

At this stage:

• test CI (here) and package-building CI (on the fork) look as expected
• test_release.yml hasn't been updated but that can be a separate PR. It will probably need further revision when the release is actually made...

I think it's a good time to merge to master and see what happens.

[ajjackson]

Ah, one thing to look at: we currently document an option to

pip install --install-option="--python-only" euphonic

https://euphonic.readthedocs.io/en/stable/installation.html#installing-euphonic-without-the-c-extension

I don't think that syntax works any more? But that's arguably an unrelated Pip/deprecation issue: https://pip.pypa.io/en/stable/news/#v23-1

[oerc0122]

The command is now:

pip install -Csetup-args="-Dpython_only=true" euphonic

Not sure if that'll work from PyPI though

[ajjackson]

That doc update looks great! And the command behaves as expected on my laptop.