[WIP] vendor git

[xu-cheng]

I will add some explanation in later time.

[MikeMcQuaid]

I will add some explanation in later time.

Ok, 🆒. In the mean time: I definitely think we want/need to vendor Ruby but I'm much less convinced for Git (or Curl). I think we should lean on brew install git instead for a few reasons:

• Git updates quite often and has had a few major security issues that require a quick turnaround. As a result I think we should have a single Git formula and binary package
• Git (and particularly not a new Git) are not needed to install Git itself
• We support installing from URLs e.g. brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/master/Formula/git.rb. If we want to improve this flow we could parse the tap name (e.g. org/repo) from this URL so the from-source Tab is correct.
• Git is required to install Homebrew so by the stage someone would receive this: they already have Git installed. While we could tweak our install process to not require Git it feels like that's not necessarily advantageous given how much still relies on having Xcode and/or the CLT installed on our supported versions of OS X.

[xu-cheng]

First of all, I think we should first acknowledge that in long term we want to make the core code, i.e. Homebrew/brew, to become cross-platform. Therefore, for git and curl, we should consider old systems like Tigerbrew and Linuxbrew. In fact, those systems are more likely to use vendor git and curl due to the fact git and curl are missing or serious outdated in those systems.

Git updates quite often and has had a few major security issues that require a quick turnaround. As a result I think we should have a single Git formula and binary package

This can also be applied to ruby because it's bundled with openssl. Since we have to vendor ruby anyway, the differential maintaining cost is less than the gain. Even more, vendor git and curl will never be used outside Homebrew.

We support installing from URLs e.g. brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/master/Formula/git.rb. If we want to improve this flow we could parse the tap name (e.g. org/repo) from this URL so the from-source Tab is correct.

This won't work for following reasons:

• For old system like Tigerbrew and Linuxbrew, both git and curl have several hard dependencies. This renders brew install url unscalable. In comparison, vendor them as a single bundle is relative easier to handle.
• On old systems, we have no choose but to use curl --insecure to download curl at least, due to outdated curl and/or CA certs in those systems. If we use vendor-install, we can at least to guarantee the security by SHA256 checksum. While for brew install URL, there is no way to do it as the content of formula is changing all the time.

Git is required to install Homebrew so by the stage someone would receive this: they already have Git installed. While we could tweak our install process to not require Git it feels like that's not necessarily advantageous given how much still relies on having Xcode and/or the CLT installed on our supported versions of OS X.

This is not entirely true. We still support to install Homebrew using curl instead of git

Besides the reasons mentioned above, there is a more important issue. For approach like brew install URL, it will mean that Homebrew will depend on some files which can be unlinked during brew install/reinstall/upgrade. And in those cases, Homebrew will be likely broken. This issue becomes more severe, when git and curl actually depend on other formulae which can also be unlinked. For example, in Linuxbrew land, if you are using git and curl from Homebrew, brew upgrade will break horribly if any of git/curl/openssl get update. The only solution is to do manually brew fetch as well as careful sort the installation order. Deep down, it is a very bad idea to mix Homebrew's dependencies with tools depend on Homebrew to install. In comparison, vendor system offer a nice and clean boundary to avoid such loop dependencies.

[MikeMcQuaid]

First of all, I think we should first acknowledge that in long term we want to make the core code, i.e. Homebrew/brew, to become cross-platform. Therefore, for git and curl, we should consider old systems like Tigerbrew and Linuxbrew. In fact, those systems are more likely to use vendor git and curl due to the fact git and curl are missing or serious outdated in those systems.

I don't think the core code being cross platform necessitates that we must provide portable, binary, vendored dependencies for each of these tools on each of these platforms. I think Ruby is a special case because it's required to run any Homebrew/brew code and we've wanted to use Ruby 2 for a long time and it's the only way we can do so.

This can also be applied to ruby because it's bundled with openssl. Since we have to vendor ruby anyway, the differential maintaining cost is less than the gain. Even more, vendor git and curl will never be used outside Homebrew.

Ruby's OpenSSL is not used for downloads so it's not the same. Our vendored Ruby will not need to be updated on every (possibly any) OpenSSL update as a result.

This is not entirely true. We still support to install Homebrew using curl instead of git

In that case: why not install the Homebrew/homebrew-core tap in the same way? Then we can brew install curl git using checksums.

Deep down, it is a very bad idea to mix Homebrew's dependencies with tools depend on Homebrew to install. In comparison, vendor system offer a nice and clean boundary to avoid such loop dependencies.

I don't agree. The vendor system (even once it's been documented/rolled out to users) adds a lot of maintenance overhead and process changes.

Regardless: stepping aside for a minute: Homebrew/brew is not cross-platform yet and it's not being used by Tigerbrew. Linuxbrew requires you to sudo apt-get install build-essential curl git python-setuptools ruby to use Homebrew so there's other system-level dependencies there we're never going to provide. Linuxbrew is pretty far from using an unmodified Homebrew/brew. I think vendoring these just for versions of OS X we don't support is a mistake when they've been using the previous system fine. I'd like to see a clear user need for this before we add the maintenance overhead and I'm really not seeing one yet. Can you point to previously user-submitted issues in the past which have arisen from using e.g. brew install git/curl on older systems? Unless there's a sizeable number (or any) I'm not convinced this is solving a problem that exists.

[xu-cheng]

In that case: why not install the Homebrew/homebrew-core tap in the same way? Then we can brew install curl git using checksums.

I explored this in #36, which is way better than brew fetch URL. But it still fails to address other deps issue.

The vendor system (even once it's been documented/rolled out to users) adds a lot of maintenance overhead and process changes.

I acknowledge it introduces some overhead. But comparing with existed overhead for vendor ruby,
vendor git and curl won't add too much instead it will only solve more problems.

Linuxbrew requires you to sudo apt-get install build-essential curl git python-setuptools ruby

As far as from my daily usage, this has not been the case since Linuxbrew starts to do bottle. For now, Homeberw's hard deps are curl and git, in addition of Xcode/CLT if you cannot use bottle. Linuxbrew's hard deps are just curl, git, which makes it less deps than Homebrew.

Can you point to previously user-submitted issues in the past which have arisen from using e.g. brew install git/curl on older systems?

We can see plenty on Linuxbrew:
Linuxbrew/legacy-linuxbrew#883
Linuxbrew/legacy-linuxbrew#912
Linuxbrew/legacy-linuxbrew#832
Linuxbrew/legacy-linuxbrew#797
Linuxbrew/brew#47

cc @sjackman and @mistydemeo Who should have more experience than me why Homebrew depends on curl and git formula is a bad idea.

[MikeMcQuaid]

I explored this in #36, which is way better than brew fetch URL. But it still fails to address other deps issue.

I'm OK with tapping the core tap using curl if we use that for this system.

I acknowledge it introduces some overhead. But comparing with existed overhead for vendor ruby,
vendor git and curl won't add too much instead it will only solve more problems.

As mentioned: vendored Git and Curl will need updated far more frequently.

We can see plenty on Linuxbrew:

But none on Homebrew, which is my point and it's unclear whether we would see the same given the linking differences on both platforms. I don't mind if Linuxbrew uses this vendor system to install their curl and git and even if their code is installed in Homebrew/brew once they are using an unpatched Homebrew/brew but I think we're a long way off that point now and I don't want to add the overhead to Homebrew when it's fixing a problem our users are currently having.

If you want to get us to that stage the best thing we can do is:

• write documentation and tooling for the existing vendoring system for Ruby so we can ship it to users
• fix brew tests --generic
• work with Linuxbrew to get their patches into Homebrew/brew
• work with Linuxbrew to see if this vendoring system makes sense for their Git/Curl usage and, if so, get this code into core just for Linuxbrew
• once all the above is done and users have been using it for a while and the maintenance overhead is understood (e.g. there's been at least one Git or Curl security update): consider including this functionality into Homebrew

[xu-cheng]

I'm OK with tapping the core tap using curl if we use that for this system.

However, I have changed my mind on that. Using curl to tap core tap during Homebrew installation may make sense. But it's definitely a bad idea to do it automatically inside Homebrew. The reason is the same as brew install URL. We don't have any checksum to verify download file, especially we have to use --insecure on old systems. It's a giant security hole.

As mentioned: vendored Git and Curl will need updated far more frequently.

If it's a just git and curl version update, in long term, we can in fact to make it to be done fully automatically. e.g. auto update Homebrew/portable based on formulae in Homebrew/core. So I still don't think the overhead is as large as you thought would be.

But none on Homebrew, which is my point

That's because even we ship vendor git now, it will still be mostly used in old systems like Tigerbrew and Linuxbrew. There is a survival basis here. The majority of normal Homebrew users will always use git and curl from Apple. Make no mistake. Vendor system is never intended to be used by everyone.

it's unclear whether we would see the same given the linking differences on both platforms.

Even without linking issue and assume git formula and curl formula are self contained(which are not true in reality), it would still be a bad idea. For example, users can always run brew update to fix any unexpected bug with vendor git, but won't do much so if it is required to install git formula at first place. The major differences between vendor system and formula are:

• Vendor git and curl are always linked(even during update), while git and curl formula may be unlinked.
• Bash is the only dependency to install vendor tools, while installing formula depends on whole Homebrew codebase and even compilers if bottle cannot be used.

If you want to get us to that stage the best thing we can do is:

We can of course do this, as from the starting point vendor git and curl is meant to be mostly used by Linuxbrew and old systems like Tigerbrew. And they are not near urgent like Ruby does. But may I ask not to judge vendor as a bad idea for git and curl too soon?

[MikeMcQuaid]

However, I have changed my mind on that. Using curl to tap core tap during Homebrew installation may make sense. But it's definitely a bad idea to do it automatically inside Homebrew. The reason is the same as brew install URL. We don't have any checksum to verify download file, especially we have to use --insecure on old systems. It's a giant security hole.

I don't see a big difference between install and runtime but 👍 to having less code in Homebrew itself.

If it's a just git and curl version update, in long term, we can in fact to make it to be done fully automatically. e.g. auto update Homebrew/portable based on formulae in Homebrew/core. So I still don't think the overhead is as large as you thought would be.

If you get everything setup and written so it's automatic I'll be a lot more 👍 on this PR. I do think it needs done before this PR is merged, though. I still don't think the documentation or tooling is where is should be for the vendored Ruby and I think it's a shame we're focusing on this before that is done.

That's because even we ship vendor git now, it will still be mostly used in old systems like Tigerbrew and Linuxbrew. There is a survival basis here. The majority of normal Homebrew users will always use git and curl from Apple. Make no mistake. Vendor system is never intended to be used by everyone.

If it's not intended to be used on any Homebrew supported platforms: I'm not sure I understand why Homebrew maintainers are responsible for building and maintaining these formulae. Again, this is more arguments for it being at the bottom of the list I mentioned.

For example, users can always run brew update to fix any unexpected bug with vendor git, but won't do much so if it is required to install git formula at first place. The major differences between vendor system and formula are:

Vendor git and curl are always linked(even during update), while git and curl formula may be unlinked.
Bash is the only dependency to install vendor tools, while installing formula depends on whole Homebrew codebase and even compilers if bottle cannot be used.

Sure, I understand the differences, I'm just (still) not convinced the trade-offs are worth it for the moment. Ruby was/is different because we can't have brew update just break Homebrew for anyone on <10.9 whereas those on older versions have a curl and git situation which, although not ideal, is working for them now.

We can of course do this, as from the starting point vendor git and curl is meant to be mostly used by Linuxbrew and old systems like Tigerbrew. And they are not near urgent like Ruby does. But may I ask not to judge vendor as a bad idea for git and curl too soon?

If it's not used by the vast majority of Homebrew users and Tigerbrew and Linuxbrew aren't using an unmodified Homebrew/brew: it's far too soon to be including this.

As an aside, Tigerbrew is currently not using Homebrew/brew at all and we don't have access to a 10.4 VM for testing. If you want us to better support Tigerbrew getting those things fixed first is another requirement before we can sensibly include these.

It's obviously up to you what you work on and when you work on it but I'll be a very strong 👎 on merging this for Git or Curl for a long time and I've suggested in this issue a lot of things that are far higher priority (and actually blocking) this PR from being included. I don't want to monopolise conversation here but it does seem a bit premature to have this PR opened and definitely merged before more of those other things are addressed.

[xu-cheng]

If it's not intended to be used on any Homebrew supported platforms:

It is not intended widely used by average Homebrew users rather than not be used at all. There is a difference here. For example, no Xcode/CLT configuration is the same that not intended to be widely used yet.

Ruby was/is different because we can't have brew update just break Homebrew for anyone on <10.9 whereas those on older versions have a curl and git situation which, although not ideal, is working for them now.

I have stated enough reasons. So I don't want to repeat again that in long term vendor is the best solution from stability and robustness respect. Why we want to force certain users to use unideal solution when there is a much better one. One of biggest reasons that I'm 👎 on using formula to solve this problem is the fact it works terribly on Linuxbrew.

It's obviously up to you what you work on and when you work on it but I'll be a very strong 👎 on merging this for Git or Curl for a long time and I've suggested in this issue a lot of things that are far higher priority (and actually blocking) this PR from being included. I don't want to monopolise conversation here but it does seem a bit premature to have this PR opened and definitely merged before more of those other things are addressed.

I'm not suggesting to merge this immediately as it's labeled as WIP. And how a PR can be opened premature? I acknowledge there are much to do, which are what I'm actually working on, i.e, to make it perfection. But you are basically tell me not working on this and vendor is a bad idea without much consideration on its actual use context, which is very frustrated.

[MikeMcQuaid]

And how a PR can be opened premature?

If you're asking for review and discussion when there's other, blocking dependencies that do not have PRs.

But you are basically tell me not working on this

I said I'm not and can't tell you what to work on but: yes, I think working on this right now is not sensible when there's other work that needs done before this can be included in any form.

and vendor is a bad idea without much consideration on its actual use context, which is very frustrated.

I have considered it: you've just not convinced me that it's necessary for Homebrew users, particularly right now.

It's also frustrating to have things that are worked on, almost done and then neglected. Some examples:

• HOMEBREW_SANDBOX is still opt-in, almost working for homebrew/core but hasn't been finished off
• The New dependency implementation milestone had Broken opt link will cause build failed under sandbox legacy-homebrew#42312 added to it but has no activity in a year
• The vendored Ruby is still opt-in and lacking discussed documentation, a testing/roll-out plan and tooling. Arguably these should have all been done before it was merged but I wanted to try and compromise there (which may have been a mistake).

I can't really understand why we're spending time discussing the pros and cons of the vendor system when it's still not been tested by our users in production.

[ilovezfs]

HOMEBREW_SANDBOX is still opt-in

I think we're at the point where it should just be turned on now.

[MikeMcQuaid]

I think we're at the point where it should just be turned on now.

Need to fix or boneyard at least these formulae Homebrew/homebrew-core#342 and then enable it just for homebrew/core to start with.

[ilovezfs]

then enable it just for homebrew/core to start with.

There's something other than homebrew/core? 🙈

[MikeMcQuaid]

Closing this; it's marked "do not merge", "in progress" and has conflicts. I'd recommend an issue first to reach some consensus on how we want to implement this (if at all).