Option to use binary packages to run scripts

[ilyaglow]

Hi,

I noticed that if I want to run some Go code with tea using a command like: tea main.go the tea will start to download LLVM and compile Go from sources, which takes forever to finish and leads to out of memory errors on devices without many GBs of RAM.

Is binary packages something you can consider?

Thank you!

[mxcl]

I noticed that if I want to run some Go code with tea using a command like: tea main.go the tea will start to download LLVM and compile Go from sources, which takes forever to finish and leads to out of memory errors on devices without many GBs of RAM.

tea/cli is unable to compile from sources and only supports binary packages. There is no way I can see that tea main.go would cause llvm to be installed or invoked (edit: I'm an idiot, the go package depends on cc, so that's why)

The only way go would be compiled from source is if you clone pantry.core and invoke scripts/build.ts go.dev explicitly.

[ilyaglow]

Oh, I do apologise for the wrong statement. I've investigated this further and I see that the problem is not regarding compilation, but indeed installation of LLVM and it happens only on Linux (or at least not on MacOS). Consider the following Dockerfile:

FROM debian:latest

RUN apt-get update \
  && apt-get install -y curl \
  && curl -Lo tea-install.sh tea.xyz \
  && YES=1 sh tea-install.sh \
  && tea https://gist.githubusercontent.com/i0bj/2b3afbe07a44179250474b5f36e7bd9b/raw/colors.go --yellow \
  && du -d 1 -h /root/.tea > /root/du-result.txt

Once the image is built, you can list ~/.tea folder, which will contain llvm.org with the size of 2.8GB:

docker build -t tea-go-test .
docker run -it --rm more /root/du-result.txt
424K	/root/.tea/zlib.net
508M	/root/.tea/go.dev
2.8G	/root/.tea/llvm.org
13M	/root/.tea/openssl.org
1023M	/root/.tea/tea.xyz
4.3G	/root/.tea

As far as I understand, we need LLVM to be able to compile CGO code. Probably we can be more intelligent here and check first if project or script really needs CGO or not.

[mxcl]

Ah k! yes our go package.yml depends on “a c-compiler” because indeed, c-go may require it:

dependencies:
  # `cgo` requires a C compiler
  tea.xyz/gx/cc: c99

https://github.com/teaxyz/pantry.core/blob/main/projects/go.dev/package.yml

We have a few options:

• detect we need it rather than hard-require it (do you know how?)
• use a new system where instead of insisting on installing llvm any use of cc triggers tea to install llvm at that point

/cc @jhheider @jonchang

Additionally it would be great to split out llvm or at least try to reduce its size.

[jhheider]

• detect we need it rather than hard-require it (do you know how?)

That's probably changing https://github.com/teaxyz/pantry.core/blob/c73c34df95cb251f4f60999f0aca16769da1173e/projects/tea.xyz/gx/cc/package.yml#L22-L26 to use ln -s $(which tea) $bin_name, thereby causing cc, ld, et al. to use tea -X cc. Other option would be writing a linux cc/ld shim that uses tea +llvm.org clang.

[jonchang]

I don't know where subpackages are on the roadmap, but it would make sense to split up go such that cgo is a separate "subpackage" that does depend on llvm on Linux. So running tea -X go ... would work fine but tea -X cgo would pull down a compiler. Though in cgo's case it's likely that it only depends on libllvm or similar and doing subpackages for llvm would also be a good idea.

[mxcl]

@ilyaglow I have made cc a “companion” of go which means it will only be fetched if go is explicitly added to the env and magic is enabled

Your example will no longer fetch cc. In a situation like tea +go.dev fetching cc can be disabled with tea -m +go.dev to disable magic.

In the future we can make it so companions are in fact hooks that trigger installation as they are requested by other tools (or the user).

pkgxdev/pantry.core@61032cf