pip config updated as global.index-url

[beck3905]

When I run partifact login my-repo my global pip configuration gets updated with index-url. Whenever I try to pip install if I have packages that are in a different index then pip can't find them. index-url becomes the default package instead of pypi.

Would it make more sense for partifact to update the global.extra-index-url config instead of global.index-url to allow it to work with multiple sources?

[davidsteiner]

Arguably, partifact should only be concerned with configuring poetry. We added in the pip logic with global.index-url as it makes sense for our unique setup internally.

Is the pip functionality useful for you at all? I'd be tempted to make it optional through a CLI argument with the default being not configuring pip. Alternatively, the CLI arg could control whether extra-index-url or index-url is set.

[beck3905]

The pip functionality is useful for me, actually, but right now it actually is breaking me. Here is my use case.

I depend on multiple packages from the public PyPI and from my private PyPI stored in CodeArtifact. My applications are serverless applications running in AWS Lambda. So when I am developing locally, I use poetry to manage my dependencies in my project, but I also use pip and pipx to manage system-wide dependencies on my local machine.

After I run partifact login I can manage my project dependencies easily with poetry and I appreciate the simplicity partifact provides. But then when I try to manage any system-wide dependencies, the changes to my global pip config made by partifact limit my ability to install any dependencies that aren't in my private PyPI in CodeArtifact. This is due to the global.index-url config and should be solved by using global.extra-index-url instead.

In addition, in my CI/CD pipeline I build Lamba layer packages. I do this by first running poetry export to generate a requirements.txt file and then I run pip install -t ... to install the dependencies into a local directory to be packaged. Again, due to the global.index-url setting I cannot install dependencies other than from my CodeArtifact PyPI. So I ended up writing a command to read the global.index-url config, set that value to the global.extra-index-url config, and then remove the global.index-url config.

TL;DR: pip functionality is useful for me, but is currently broken for me. Making them optional arguments and supporting extra-index-url instead of index-url should help.

[davidsteiner]

Thanks for the detailed write-up!

Are the packages that are not in your CodeArtifact repository in another private repository? For public packages, wouldn't having PyPI as an upstream repository for your private CodeArtifact repository suffice? Sorry, just trying to make sure I fully understand your use-case.

For the CI/CD use-case, are you exporting with --with-credentials? Again, we only ever use a single private repository, so there may be gotchas I'm not aware of, but I feel like the exported requirements file should be self-contained without relying on the global URLs in the pip config.

If you still think setting the global.extra-index-url is useful, my preference would be implementing this as an optional convenience through a CLI argument. What do you think?

[beck3905]

I could have PyPI as an upstream for my private CodeArtifact, but I don't really need the extra setup to do that because I can just hit PyPI directly and my private CodeArtifact directly.

I can use --with-credentials to export my requirements.txt and it gets added as an extra-index-url in my requirements.txt. However, the problem is still that the global.index-url setting is limiting pip to only looking in the single private CodeArtifact repository. Poetry doesn't add PyPI to the requirements.txt as an index url or extra index url.

It seems like if I do the --with-credentials in my export then everything I need will be in my requirements.txt as long as the global.index-url is not set in my pip config. So I think either removing that altogether or making it optional via CLI would address my use-case. Thanks

[davidsteiner]

In that case, I'd make it optional and not configure pip by default. We can introduce a --configure-pip flag to retain the current behaviour.

I can do it sometime this weekend. Alternatively, if you want to have a go at it, let me know.

[beck3905]

I think that will work. I just ran a test where I run pip config unset global.index-url and added --with-credentials to my export and it worked. Before, without the --with-credentials I was reading the global.index-url value and saving it to global.extra-index-url and then unsetting global.index-url. So this simplifies it for me. With your proposed change I won't have to mess with pip config at all.

Unfortunately, I don't have time to work on this right now. I have a working setup so it's no big rush. I appreciate your help and responsiveness.

[davidsteiner]

0.3 should address this. Any issues, let me know!

[jrobbins-LiveData]

While I don't fully understand the long thread I'm linking to here, it seems germane regarding how pip treats global.extra-index-url: pypa/pip#8606

This isn't a partifact concern, but it is relevant to trying to understand pip's approach to multiple code repositories. One important thing to note is that the entire python package namespace is a singleton. You need to give your python packages unique names. At any point, someone might publish to the pypi repository a package with your private package's name, and, depending on how pip resolves the package, you might get surprised if you rely on pip's search order!

For this reason, putting CodeArtifact "in front" of pypi ("upstream") seems to be a safer choice.