Remove python/poetry venv autocreate feature

[jdx]

When I built the venv autoactivate I copied direnv's behavior which creates the venv or runs poetry install if it hasn't been done yet. I'm wondering if this is what we want though. Would it be simpler if rtx didn't do this and instead just activated an existing venv?

To be clear, I'm proposing removing the "creation" step, not the automatic activation (which I know is well liked). The creation is often the source of problems. Basically removing the python -m venv step and showing a warning if it doesn't exist.

Perhaps this could go behind a setting like MISE_PYTHON_VENV_AUTO_CREATE defaulting to disabled?

My thinking here is that because I'm not a python developer I often get the logic around venvs wrong. I'm simply not the best person to lead a project for python DX (just an unfortunate reality). I'm thinking rather than me make mistakes, just have mise do a bit less so I don't get the logic incorrect. I feel like probably 25% of mise issues are venv related. I often will fix it in one place but break it in another.

I'm leaning towards doing this over not doing this, maybe with some sort of tbd deprecation message. I very much welcome input from python users though.

On a related note, I also plan to move this config from the [tools] section to the [env] section like this:

[env]
mise.python.venv = ".venv"
mise.poetry.pyproject = "pyproject.toml"

My thinking this is more of an "environment" config and mostly unrelated to the tool (they are called "virtual environments" after all). This will also make it possible to use it with the system python which can't work in [tools] without significantly changing the meaning of the term "system" in mise.

Related issues:

• Automatic Python venv activation doesn't account for Framework builds on macOS #1354
• Can't use virtualenv with system python #1248
• python: do not forcefully create venv #1058
• new venv syntax #1303

[jdx]

I'll be truly shocked if anyone picks the first option here. If you really think that's the solution you're going to need to back it up :)

[mustafa0x]

On a related note, I also plan to move this config from the [tools] section to the [env] section like this:

[env]
"@python:venv" = ".venv"

Isn't [env] for "setting arbitrary environment variables"? This usage seems to be different though.

[jdx]

if I remove the autocreation, then the only thing that would do is effectively set VIRTUAL_ENV=.venv with a bit of logic to show a warning if it doesn't exist.

"@" and ":" are invalid characters in env var keys¹ so I plan to use that syntax to denote special behavior within env var config like this.

I would argue even with autocreation putting it in [tools] is the wrong place. It's not managing the python runtime at all.

Footnotes

1. I think technically you can potentially use these characters in env vars at least on some OS's, but it's not well supported in any shell I don't think so it's practically not supported. ↩

[jdx]

update: changed the toml syntax a bit to make it not require quotes

[mustafa0x]

Ok, so it'll just be a shorthand for setting a special environment variable. Makes sense, and that doesn't feel too much like implicit behavior / magic / overloading. Thanks.

[delicb]

I can stand behind this format. But VIRTUAL_ENV is not enough, PATH should be updated as well.

[jdx]

I can stand behind this format. But VIRTUAL_ENV is not enough, PATH should be updated as well.

yeah exactly right. There is very good support in the internals of mise for modifying PATH though so it's easy to make that work

[delicb]

Adding a setting with default being disabled would be fine (option no. 3) IMO. This is the option I am voting for.

However, I just had a different thought. I did not follow development for some time (just used it as is, until rename brought me back to this repo) so I did not know that there is not experimental support for tasks. As far as I understand, tasks are not executed directly. However, if there is a way to:

• execute a task on event (e.g. cd into directory and cd out)
• export environment variables from task

Then special handling of python virtual environments could boil down to something like:

[tasks.pyvenv]
description = 'Create and activate virtual env'

# you can specify a multiline script instead of individual commands
run = """
#!/usr/bin/env bash
[ ! -d venv ] && python -m venv .venv
source .venv/bin/activate
"""
export_vars = ["VIRTUAL_ENV", "PATH"]
run_on = ["enter"]

Of course, example above does not work now (export_vars and run_on), but if such features were possible, this could be used to tailor behavior as needed. Having ability to "call" some globally defined task (so that same task does not have to be repeated in every .mise.toml (this is inspired by direnv, which I was using heavily until I switched to this tool).

Again, I have no idea what the roadmap for tasks feature is, but I have a filling it could be helpful here and leave only bare python version management for mise core.

[jdx]

what you're describing is a feature we don't have, I think I would call it "hooks" if we did implement it. Tasks are a different use-case, they're for manually executed things where you explicitly run mise run .... Still, that's semantics, I could see us adding something exactly like this but we don't have it today.

There was just some conversation around a similar idea to build env vars based on external commands too which might be relevant: #1301

Templates even enable this today with some caveats (performance can be impacted since they get run basically every time mise starts)

[delicb]

Yep, I understand that there is no such a feature today, I was just imagining how it would look like and if it could be helpful in this discussion. "Hooks" is a good name for it. I assume that a lot of implementation with tasks would be shared.

Event if it is not there today, I imagine this is such a pressing issue that it has to be solved right now, it could wait for "hooks", if they are on roadmap.

[jdx]

wow unanimous so far. So I'll go with disabling by default unless some config is set. I think with this in place I think I'm finally ready to label it as non-experimental anymore.

Follow-up question though: what setting do you plan to use?

[AlexCzar]

In my use-cases it'll be mostly on by default. .mise.toml files are committed to repos and when I clone the repo it's nice that I don't have to do any manual setup, I just cd into it and mise takes care of everything.

I also broke the unanimity, sorry :). My argument is not my own use-case though, but backward-compatibility.
The behaviour is currently to auto-create, adding the setting arguably should break as little things as possible, which means that new setting's default should also be to auto-create.

Additionally, I want to mention a couple of issues I'm experiencing with the currently existing functionality.
It would be nice if they are addressed as part of the planned refactoring.

1. I have a project which has .mise.toml at the root level, which specifies python version and venv.
   I have a subdirectory in that project that has nothing to do with python but addresses some other aspect of the project (say, deployment). When I add .mise.toml to the subpdirectory, mise wants to set up venv there, too, even though that subdirectory's .mise.toml doesn't have any python stuff at all.
2. Mise tasks ignore venv. I tried defining something like python -m code_formatter as a mise task where code_formatter module is installed in the venv, but this fails because the python that runs is the base one, not the venv one. This is easy enough to work around, but annoying. I assume that everything that mise configures should be accessible to its tasks, like environment, PATH, etc.

[jdx]

I also broke the unanimity, sorry :). My argument is not my own use-case though, but backward-compatibility.

keep in mind this is an experimental feature explicitly to be clear that breaking changes should be expected

[jdx]

I am pretty sure moving the config to [env] will resolve your first issue. It will mean the tool config and virtualenv config can be in separate files.

I'm actually a little surprised your second one is happening—it definitely should be using the venv one. I'll need to investigate that but it just sounds like a straightforward bug. I'm probably not using the right variable or initializing the right thing.

[AlexCzar]

1. Totally, I just mentioned that breakage as an argument as to why I feel keeping auto-create on by default would be nice.
2. Good to hear that :)

[noirbizarre]

My 2 cts: I think moving this to env is a bad idea: initially env was for "environment variables": clear scope, well known naming.
Now it accepts some specials mise.* vars for env_files. While this is great to be able to support multiple env vars, it makes the scope and purpose of env blurry.

Adding virtualenv configuration into it while it's not strictly related to environment variables will just open the door to having a one-catch-all env section and add lots of confusion.

By consistency with the environment variable MISE_TOOL_OPTS__VIRTUALENV and thinking forward compatibility, I think virtualenv management is some tooling/plugin config and should go in a dedicated section for this tool, something like:

[tools]
python = "3.7"

[python]
virtualenv = ".venv"

I think it's more scalable for both other Python specific options or other plugin that will for sure have some specific fine-tuning to add later.

It also respects separation of concerns.

Note that while part of virtualenv management is about hijacking some environment variables, it's not just that. How can I specify that I want to have system-packages in the venv ? Another special syntax ?
What if we want to support some pre-installed dev tools ? What if later we want to distinguish "reuse virtualenv" and "create if missing" ?

The dedicated section allows this:

[python]
virtualenv = ".venv"
create-virtualenv = true
site-packages = true
packages = ["ipython", "debugpy"]

This would be harder to handle with special env processing.

In the case of Python, adding some poetry, pdm or any package manager specific settings would be easier as both same level options or nested table.

At last having a dedicated section with explicit options make it possible to have a clear schema for completion or validation.

And I didn't raise the fact that it introduce an order and precedence issue: when does the venv get activated: before env, after env processing, at the same time that other variables... ?

Note: having the possibility to have env and env_files processed before and after plugins activations would be just perfect and allow:

[env]
SOME_VAR_THAT_MIGHT_BE_NEEDED_BEFORE_ACTIVATION = "value"

[tool]
python = "3.11"

[python]
virtualenv = ".venv"

[lazyenv]  # just for the example, naming TBD
SOME_VAR = "{{ env.VIRTUAL_ENV }}/some/extra/path"

[jdx]

or I was also thinking of just making [env] itself optionally an array:

[[env]]
_.scripts = "script1.sh"
_.python.virtualenv = ".venv"

[[env]]
_.scripts = "script2.sh"
MY_VAR = "{{ env.VIRTUAL_ENV }}/somewhere"

[tools]
python = "3.11"

[jdx]

I kind of think this last one is the cleanest, any thoughts?

[jdx]

I think I might change the deprecation method for both the existing env_path and if we change from env.mise to env._.

There isn't any good reason to warn the user. I'll change the docs but keep parsing the old syntax indefinitely (or at least for a very long time).

That's normally how I deprecate things. I think with mise showing an actual warning should be reserved for things actually likely to cause a problem. Not just something to simply improve consistency.

Consistency is important, but I think in this case the remedy is worse.

[noirbizarre]

I do prefer _ over mise. I think it's clear it's a directive. I think this could work.

Glad I shared the idea !

or I was also thinking of just making [env] itself optionally an array:

[[env]]
_.scripts = "script1.sh"
_.python.virtualenv = ".venv"

[[env]]
_.scripts = "script2.sh"
MY_VAR = "{{ env.VIRTUAL_ENV }}/somewhere"

I do like the idea if it's optional so people not requiring multiple script can still use the basic form. Given it is totally possible to validate the file format, I don't have an issue with the 2 forms approach.

[[env._.script]]
run = "script1.sh"
[env]
_.python.virtualenv = ".venv"
MY_VAR = "{{ env.VIRTUAL_ENV }}/somewhere"
[[env._.script]]
run = "script1.sh"
[tools]
python = "3.11"

This one feels a bit to complicated to me. If we go the array way, [[env]] seems more natural and simple.

what's nice about this is you don't need to make up a name like "first/second". I don't love that you can't do it inside of the [env] block though. I also think [[env._.script]] looks really weird and probably nonsense to someone that doesn't really understand toml.

[env]
_.scripts = "script1.sh"
_.python.virtualenv = ".venv"
_.scripts_ = "script2.sh"
MY_VAR = "{{ env.VIRTUAL_ENV }}/somewhere"

[tools]
python = "3.11"

this is easy for me to parse since I would just strip off any extra "_" at the end of keys.

I do like this idea too and I though about it. It was the alternative to naming. _.script* are executed in order or appearance. It handles multiple scripts quite well and I don't believe we should aim for a solution that fit lots of scripts. If people need lots of scripts, chances are it can be merged as a single or 2. Multiple script are just there to be able to have scripts/module loading alternance, not to replace a shell so most of the time people will just have _.script, _.script_ and maybe _.script_X where X is a number. Good enough to handle has many cases as it should I think.
Note that I used the singular form. If we go the _.script* prefix, we don't need to have a separate plural form and for the basic 1 script case, _.script seems more fitted. But it's a detail, plural form works too.

[jdx]

right now I think I'm leaning towards [[env]] as the solution—of course [env] would continue to work as it does today. I would probably just put this in the docs under a "tip" callout or something that you can use [[env]] instead in the situation where you have multiple scripts.

Thankfully we've got time to come to a decision here, as you said this is an edge case and _.scripts isn't even supported yet.