Add Miniforge instructions for users

[jpivarski]

This is a question that @eduardo-rodrigues, @henryiii, @chrisburr, @HDembinski may want to weigh in on: whether we, as Scikit-HEP, should endorse Miniforge as the recommended way to install Python packages. I know that we have different preferences, but in my experience with tutoring, getting users started has been easiest with a conda environment, and now Miniforge even removes the aspect of having to change channels and be sure that conda-forge doesn't get mixed in with the default channel.

I was prompted to make this pull request by a conversation on Slack (link will only work if you're on IRIS-HEP's Slack). The people involved in that conversation were @alexander-held, @kpedro88, @Moelf, @matthewfeickert, @agoose77, @NJManganelli, @PerilousApricot, @tacaswell. (If you want to unwatch this PR, feel free to!)

I welcome any direct edits to this PR if you want to change anything.

[PerilousApricot]

Speaking from the peanut gallery, is there a technical benefit to using conda over pip+venv? What is the reason for recommending one over the other?

[henryiii]

My only worry is this the wrong model to follow for as soon as someone starts working on two or more projects. Dumping everything into a single environment and not maintaining a environment.yml file will cause issues - mamba makes it scale better than conda (which is unmanageably slow after the first update or so). Having environment files per project and never installing into base is a much better model (as is avoiding activating base at all by default - so "python" remains whatever it is normally if you don't type conda activate <x>). But I do understand this is likely the best "beginner" model for someone who hasn't started a project before.

[jpivarski]

Speaking from the peanut gallery, is there a technical benefit to using conda over pip+venv? What is the reason for recommending one over the other?

From my perspective, I'd say that it's harder to get the conda environment wrong. Or at least, easier to back out of and try again. What I'm suggesting is that we, as Scikit-HEP, recommend a Miniforge+mamba set-up for users who don't have reason to pick something else. That is, if they come to us asking, "How should I install Python packages?" We'd have an answer for them.

If it's pip+venv, which Python is it? Where are the pip packages going, ~/.user? It leaves too many open questions. That's my reasoning.

[henryiii]

Conda has some downsides - specifically, if you have to start compiling anything, conda is broken; conda isn't designed for users to work on packages, it's focused on distributing code to users, but not users compiling code. We probably get someone complaining weekly about pybind11 segfaulting and it's because they installed python through conda then expected to be able to build code (with the system compilers).

I think we still strongly support pip/venv users, it's just that this is a good "jumping in" point, and what most tutorials start with, which is fine. It's also the only proper way to get ROOT. ;)

[henryiii]

Well, venv goes wherever you tell it to; .venv is a standard location and picked up automatically by the Python launcher for UNIX. But yes, you have to get that Python somehow (brew for me on macOS, personally, and pyenv for specific versions like the recent 3.11 beta).

We could make this a tabbed page eventually with more guides, and a workshop could link to the one they wanted.

[jpivarski]

At the ROOT workshop, somebody said that ROOT supports 64-bit Windows now, so that may remove the only blocker to installing ROOT for Windows via conda-forge. In that case, we'd also be able to say, "only proper way to get ROOT" without the qualifier.

[jpivarski]

Also, if @tacaswell's guide to overlaying environments can be made to work with HEP things like LCG, CDMFS, or collaboration Python distributions, then the tabbed pages could be specific advanced instructions. The goal for them would be to get scientific Python and ROOT and collaboration-specific stuff.

[jpivarski]

This article is ready to merge. Here is an easy way to view it, with all of the drop-down text: https://github.com/scikit-hep/scikit-hep.github.io/blob/add-miniforge-instructions/pages/installing-packages.md

I do not know, however, if this article needs to be put into a "toc" file for it to end up on the website. If so, then that update should be added to this PR before merging.

Otherwise, let me know if you have any last objections. I think I've taken all of your suggestions on board.

[henryiii]

Two last closely connected ideas:

First, this seems like it might be part of something larger - the beginning of a tutorial, a group of setup pages, etc. Would it make sense to put it into a sub menu? We are getting a pretty large menu of top level things. One could easily argue about should contain citing, getting in touch, and code of conduct, but that didn't happen. While the developer pages are focused on developers, this is focused on users, so maybe a similar User Pages? That affects the URL, which should be fairly stable after merging, so good to think about now.

The other thing is that there really needs to be an overarching tutorial series for Scikit-HEP. There was a minor attempt at https://github.com/scikit-hep/scikit-hep-tutorials but it never really went very far. This is something I think that is becoming very important. Having a great, friendly introduction¹ to how all the libraries play a part would be important for our growth. I've been thinking it makes more sense to do that tutorial series as Jupyterbooks, but instead integrating it in to the website and writing it in markdown isn't unreasonable either, and would look better. The biggest issue is it's harder to make sure it continues to work. How would this tie into that effort?

Footnotes

1. I believe one of the main reasons Poetry got so popular is because it has beautiful documentation: https://python-poetry.org ↩

[henryiii]

Here is an easy way to view it

Or:

gh pr checkout 218
# bundle install if you haven't already
bundle exec jekyll serve

First thought: title is too long. How about Setup with Conda? Or Installing Packages?

[henryiii]

Second thought: Please don't add headings inside details! Those are already carefully styled.

[henryiii]

Also, nested details don't render well - I've never set that up. I think it's fine to have several un-nested details. Or we can add nested details styling...

[jpivarski]

I think that making a group of user pages, parallel to the developer pages, is a great idea! This could be the first of those.

If we don't put heading tags inside the details summary, how would we render them appropriately? Semantically, they are heading titles.

Maybe we could have normal headings and all the text contained within should be in a details block?

Your screenshot reminds me that this would sometimes be viewed in dark mode. I don't know that the comic looks so good that way. But I don't know if an alternative, except for maybe adding margins. (I doubt it would look good inverted.)

[jpivarski]

Also, nested details don't render well - I've never set that up. I think it's fine to have several un-nested details. Or we can add nested details styling...

Since the nested ones are at the ends of sections, they could be unnested without losing their meaning.

[henryiii]

[henryiii]

I've moved it to a "user information" folder. I'd really like to condense our menu a bit - having so many top level items makes each top level item less special. But we can do that later (and we don't have to structure the URL to place items in menus, which is nice.

I named and renamed a few things, feel free to change, I'm not particularly inspired with good names. Also on folder name (user), etc.

[henryiii]

I've also added (expand) / (close) via css instead of manually adding (click here) each time.

[HDembinski]

I use pip + venv and recommend this to my students. We have an official Python package manager and it works. I have no need for conda or mamba.

That being said, the cluster environment maintained by our institute is using conda, because of ROOT (my subgroup is not using ROOT, but others do). So, as usual, everything bad in the world that is happening is because of ROOT. /s

[eduardo-rodrigues]

This is a question that @eduardo-rodrigues, @henryiii, @chrisburr, @HDembinski may want to weigh in on: whether we, as Scikit-HEP, should endorse Miniforge as the recommended way to install Python packages. I know that we have different preferences, but in my experience with tutoring, getting users started has been easiest with a conda environment, and now Miniforge even removes the aspect of having to change channels and be sure that conda-forge doesn't get mixed in with the default channel.

Thank you for the ping. I have no strong opinion on the topic and tend to work with envs as simple as possible (same as Hans very often, it seems). As for a recommendation: we give various recommendations here and there, be it in READMEs or in the devs pages of our website, that it is in a way consistent to go a step further and provide guidelines/instructions for something that is important and viable such as what you wrote. BTW, this MR relates strongly to the long standing task #81. In short, I'm for it as long as we keep the idea of guidelines.

[eduardo-rodrigues]

First, this seems like it might be part of something larger - the beginning of a tutorial, a group of setup pages, etc. Would it make sense to put it into a sub menu? We are getting a pretty large menu of top level things. One could easily argue about should contain citing, getting in touch, and code of conduct, but that didn't happen. While the developer pages are focused on developers, this is focused on users, so maybe a similar User Pages? That affects the URL, which should be fairly stable after merging, so good to think about now.

Henry, agree that a little reorganisation is going to be necessaryat some point soon. I did a little bit already when I made https://scikit-hep.org/projusers have 2 sub-pages. To me getting in touch is something that should be readily visible, though, just as getting in touch. This is about communication and that ought to be as straighforward as possible. OK, going off topic anyway.

The other thing is that there really needs to be an overarching tutorial series for Scikit-HEP. There was a minor attempt at https://github.com/scikit-hep/scikit-hep-tutorials but it never really went very far. This is something I think that is becoming very important. Having a great, friendly introduction1 to how all the libraries play a part would be important for our growth. I've been thinking it makes more sense to do that tutorial series as Jupyterbooks, but instead integrating it in to the website and writing it in markdown isn't unreasonable either, and would look better. The biggest issue is it's harder to make sure it continues to work. How would this tie into that effort?

Yes, totally agree! scikit-hep-tutorials is dormant but we do have lot's of material for a major update. I guess people kind of put the project tutorials to the side since a while, being busy with developments. But I am convinced that tutorials are just about the most important game-changer for the project these days.

I also agree that Jupyterbooks is a good option. I would not have the tutorials on the website because it would couple too much and I like the fact that anyone can contribute to the tutorials repo, even a young student, and having to touch the website pages would give the wrong message. We should have a discussion on tutorials elsewhere ...

[henryiii]

FYI, personally, I almost never use conda too. I use venv or some other tool that creates venvs (virtualenv is faster, or PDM for an all-in-one packager with locking). The conda version of Python is broken for building packages, so it's nearly useless for me most of the time. I do use conda via environment.yml files for tutorials, though, since it's the easiest way to get everyone on the same page and quickly going. I also use it for multi-person analysis projects, which is why I think it's fine to have this here. And of course I use it when working on conda recipes. :)

I will likely reread and submit some small changes as a PR later, but I think having a conda guide is overall a good idea (I did rename it to "installing conda" since that's what it is about).

[eduardo-rodrigues]

FYI, personally, I almost never use conda too. I use venv or some other tool that creates venvs (virtualenv is faster, or PDM for an all-in-one packager with locking). The conda version of Python is broken for building packages, so it's nearly useless for me most of the time. I do use conda via environment.yml files for tutorials, though, since it's the easiest way to get everyone on the same page and quickly going. I also use it for multi-person analysis projects, which is why I think it's fine to have this here. And of course I use it when working on conda recipes. :)

Good points, agreed. I too use conda via ... as you. That's really super handy especially when coupled to using Binder.

[jpivarski]

I think the new screenshot looks beautiful, with the improved drop-downs. (Though a style="border: 2px white solid" might help the comic in dark mode while being invisible on a white background). I'm glad that it could be integrated into the website as a whole so well.

I'm also glad that so many of us have weighed in on it, and we seem to be in agreement that this recommendation—Miniforge with mamba—is the best generic choice for newcomers who will likely need ROOT. On particular systems, there may be a more robust way to get a Python executable; conda has downsides for users who will be compiling a lot, though that probably doesn't apply to anyone who needs this page (for what it's worth, I've never encountered issues with conda and compilation); and there is the issue of installing ROOT with the right Python version, which is only handled well by conda. It seems that for every one of us, there's something that we do differently that we are not recommending for newcomers, and we recognize that the situation is different when we have to give one, simple, generic set of instructions.

So it sounds like everybody is okay with this becoming the recommendation. Is that true? Are we ready to merge?

[henryiii]

Added a "rebase PR" button to the GitHub UI, and also set the GitHub UI to use the PR title for squashing.

[henryiii]

Ahh, so that was a black frame causing the line in the middle, not a clear part.

[jpivarski]

I'd mostly want to make sure everyone is okay with the permalink scheme - that's the one thing that's not easily changeable after merging. Otherwise I'm fine.

Yes, this is a forward-looking permalink. If you're having regrets about things like "Python version policy" (which ought to be under "Developer information," I think), isn't it possible to change the nesting in the sidebar while leaving the permalinks untouched? The sidebar is the more user-visible thing.

I've never encountered issues with conda and compilation

I'm going to guess you usually install ROOT, which means you have the compilers package? I think things work if you install the compilers package. Because you are now also taking over your compilers. This will break other things that are not compiling for Python, however, if it's in by default.

Effectively, I do. I didn't know there was a metapackage named "compilers" that depends on "cxx-compiler" that depends on "gxx_linux-64" (for me); I directly installed "gxx_linux-64" and equivalent for C and Fortran because I know I have Linux. And I use those compilers for everything that I compile, which is strictly within the nested doll of conda. (I would never compile system packages in this day and age...) So that's why it works for me.

style="border: 2px white solid"

You are welcome to style this any way you want - if there are any clear parts, maybe setting a white background is a good idea?

I just did. The PNG itself doesn't have transparency, but it uses a black border at the edge of the PNG to separate itself from the page, and that black border isn't visible in dark mode. I put a white border around the black border, so it works in dark mode and on white backgrounds. My browser is by default light mode, and I can attest that it looks good there.

[henryiii]

metapackage named "compilers" that depends

Technically, it also adds symlinks/variables so that 'c++' points at the conda compilers with the environment is active.

"Python version policy" (which ought to be under "Developer information"

I'd really like most or everything nested - if there's basically no top level links, then that makes mobile navigation easier, and ensure that someone is likely to open "about" or some menu. Then highlighted things (contact us, etc) could be linked to on the main page as well to help highlight them.

Anyway, I don't think "Python version policy" should be in Developer information for two reasons - one is that the developer guidelines are a product on there own, a better version of packaging.python.org (opinionated and easier to edit). Python version policy (and package requirements) are specific to us, so I think they should probably go under "Packages". The second thing is those guidelines are for users too, not just developers, so they don't fit under a developer heading.

I'll likely propose something.

[henryiii]

Okay, I put this in, let's see how we like it, and people can propose edits.

PS: Yes, pages can be "moved" in the navigation structure without changing the link.

[henryiii]

Menu restructure in #220.