Update scripts to use cabal v1-... so they work on newer cabal

[anka-213]

Fixes build failures like the ubuntu failure here: https://github.com/GrammaticalFramework/gf-core/runs/2949099280?check_suite_focus=true

[anka-213]

Here's a successful ubuntu build: https://github.com/anka-213/gf-core/runs/2949499363?check_suite_focus=true

[anka-213]

We might want to migrate to the v2- versions of the cabal commands, but then we would not include the haskell-libraries in the deb package, since that workflow is no longer supported.

[johnjcamilleri]

We might want to migrate to the v2- versions of the cabal commands, but then we would not include the haskell-libraries in the deb package, since that workflow is no longer supported.

I have to say, I don't quite understand what you mean here. Of course we want to be able to produce a deb file containing the GF binary. Are you saying that the current deb file we produce also includes GF Haskell libraries?

[johnjcamilleri]

I guess this change means that if someone is using an old version of Cabal then the make commands won't work for them. Are the v1- commands old enough that this is safe to assume? Or is it worth making the Makefile query the Cabal version and decide whether to use the v1- prefix or not?

[anka-213]

Are you saying that the current deb file we produce also includes GF Haskell libraries?

That is what I was assuming (but didn't check), but maybe I misinterpreted what the cabal commands does.

[anka-213]

The v1- prefix was introduced in cabal-install 2.4.0.0 from 2018

2.4.0.0 Mikhail Glushenkov [email protected] September 2018
https://github.com/haskell/cabal/blob/f0d0594bae4f8e3e1ae1bdacef6217a4fc4e136d/cabal-install/changelog#L144

If we don't care about the makefile working for newer cabal, we could just update the debian build script which is only running on the cloud.

[johnjcamilleri]

It would be great if the Makefile could work for old Cabal, new Cabal, and even Stack. In the past I've used this, which I think can be easily adapted:

STACK=$(shell if hash stack 2>/dev/null; then echo "1"; else echo "0"; fi)

ifeq ($(STACK),1)
CMD=stack
else
CMD=cabal
endif

install:
    $CMD install

What do you think? I'm happy to work on this myself, although I'm not sure if I can push to your fork.

[anka-213]

I'm happy to work on this myself, although I'm not sure if I can push to your fork.

Yes, that would be nice!

I've checked this checkbox, so you should be able to push to my branch:

• Allow edits and access to secrets by maintainers

If that doesn't work I can give you full access to my fork.

[johnjcamilleri]

It would be great if the Makefile could work for old Cabal, new Cabal, and even Stack.

This is now done.

[johnjcamilleri]

We might want to migrate to the v2- versions of the cabal commands, but then we would not include the haskell-libraries in the deb package, since that workflow is no longer supported.

So I did some investigation. The deb file does indeed include GF Haskell libraries as *.hi files:

# dpkg-deb -c gf_3.10-2_amd64.deb | grep "\.hi"
-rw-r--r-- root/root     19160 2019-03-05 15:44 ./usr/lib/x86_64-linux-ghc-7.10.3/gf-3.10-21hcjqLQM7F05auNJPUt40/PGF/Internal.hi
-rw-r--r-- root/root     22545 2019-03-05 15:44 ./usr/lib/x86_64-linux-ghc-7.10.3/gf-3.10-21hcjqLQM7F05auNJPUt40/PGF/Haskell.hi
-rw-r--r-- root/root     57241 2019-03-05 15:44 ./usr/lib/x86_64-linux-ghc-7.10.3/gf-3.10-21hcjqLQM7F05auNJPUt40/PGF/Data.hi
-rw-r--r-- root/root     28620 2019-03-05 15:44 ./usr/lib/x86_64-linux-ghc-7.10.3/gf-3.10-21hcjqLQM7F05auNJPUt40/PGF/Macros.hi
-rw-r--r-- root/root     58289 2019-03-05 15:44 ./usr/lib/x86_64-linux-ghc-7.10.3/gf-3.10-21hcjqLQM7F05auNJPUt40/PGF/Binary.hi
-rw-r--r-- root/root     24958 2019-03-05 15:44 ./usr/lib/x86_64-linux-ghc-7.10.3/gf-3.10-21hcjqLQM7F05auNJPUt40/PGF/Optimize.hi
...

But I'm not sure this is necessarily intended. At least installing the deb package with dpkg -i does not seem to install these Haskell libraries in such that they are globally available to GHC.

Anyway, I think this is a separate issue. This PR itself can be merged AFAIC.

[anka-213]

This does however not solve #74. It only makes the package build, it still doesn't work properly with newer ubuntu.

[anka-213]

It successfully installs the package, but trying to run gf gives the error:

gf: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory

I'm using this Dockerfile for testing:

# syntax=docker/dockerfile:1
FROM ubuntu:20.04
COPY gf_3.10.4-1_amd64.deb /installer/
RUN dpkg -i /installer/gf_3.10.4-1_amd64.deb
RUN gf --version

[anka-213]

Probably all that's needed is to bump the ubuntu version on the runner. Let me try that.

[anka-213]

Here's the latest build: https://github.com/anka-213/gf-core/actions/runs/1003605632

Unfortunately it failed with a new error now:

gf: error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory

Before upgrading ubuntu, libatomic.so.1 was not a dependency:

# ldd $(which gf)
	linux-vdso.so.1 (0x00007fff80bce000)
	libpgf.so.0 => /lib/libpgf.so.0 (0x00007efc78a4e000)
	libgu.so.0 => /lib/libgu.so.0 (0x00007efc78813000)
	libtinfo.so.5 => not found
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007efc78808000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007efc78803000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efc787fd000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efc787d8000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007efc78754000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007efc78605000)
	libffi.so.6 => not found
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efc78413000)
	/lib64/ld-linux-x86-64.so.2 (0x00007efc78c82000)

after using ubuntu-latest on ci to build the package, it is a dependency for some reason:

# ldd $(which gf)
	linux-vdso.so.1 (0x00007ffd2e1a8000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7ef31e3000)
	libpgf.so.0 => /lib/libpgf.so.0 (0x00007f7ef31b1000)
	libgu.so.0 => /lib/libgu.so.0 (0x00007f7ef3174000)
	libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f7ef3144000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7ef3139000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f7ef3134000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7ef312c000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7ef3109000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f7ef3085000)
	libatomic.so.1 => not found
	libffi.so.7 => /lib/x86_64-linux-gnu/libffi.so.7 (0x00007f7ef3079000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7ef2e87000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f7ef3336000)

[anka-213]

That was easily resolved by letting apt install the dependencies:

# syntax=docker/dockerfile:1
FROM ubuntu:20.04
COPY gf_3.10.4-1_amd64.deb /installer/
RUN apt-get update
RUN apt-get install -y /installer/gf_3.10.4-1_amd64.deb
RUN gf --version

[johnjcamilleri]

Hmm, this fixes 20.04 but other versions don't work:

Ubuntu 18.04

 gf depends on libc6 (>= 2.29); however:
  Version of libc6:amd64 on system is 2.27-3ubuntu1.4.
 gf depends on libffi7 (>= 3.3~20180313); however:
  Package libffi7 is not installed.
 gf depends on libtinfo6 (>= 6); however:
  Package libtinfo6 is not installed.

Ubuntu 16.04

 gf depends on libc6 (>= 2.29); however:
  Version of libc6:amd64 on system is 2.23-0ubuntu11.3.
 gf depends on libffi7 (>= 3.3~20180313); however:
  Package libffi7 is not installed.
 gf depends on libgmp10; however:
  Package libgmp10 is not installed.
 gf depends on libtinfo6 (>= 6); however:
  Package libtinfo6 is not installed.

Maybe we need to have separate binaries for Ubuntu >= 20 and Ubuntu <= 18 ?

[anka-213]

Yes, that's probably the case. Ubuntu packages are often dependent on which ubuntu-version you have.

I wonder if we should create a ppa to publish the package on as well?

[johnjcamilleri]

From https://help.launchpad.net/Packaging/PPA:

Using a Personal Package Archive (PPA), you can distribute software and updates directly to Ubuntu users. Create your source package, upload it and Launchpad will build binaries and then host them in your own apt repository.

So if we use a PPA then we won't even need to build the binaries ourselves, which sounds like a plus.

[inariksit]

Shall we do this PPA thing then? Sounds like a good idea to me. Will one of you take up the task or shall I delegate it to someone at CCLAW?

[johnjcamilleri]

I don't think I have the time for it atm.

…

On Wed, 14 Jul 2021, 08:07 Inari Listenmaa, ***@***.***> wrote: Shall we do this PPA thing then? Sounds like a good idea to me. Will one of you take up the task or shall I delegate it to someone at CCLAW? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#116 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC37E53UBY6E6HWXJTSZODTXUSTPANCNFSM47RU67GQ> .

[inariksit]

@johnjcamilleri Ok, I'll delegate the task to someone!

[fititnt]

@anka-213

Yes, that's probably the case. Ubuntu packages are often dependent on which ubuntu-version you have.

I wonder if we should create a ppa to publish the package on as well?

While I sure would love official PPA (because requires less dependencies) either:

• https://flatpak.org/
• https://snapcraft.io/

Are likely to be easier from the point of view of maintainer (at cost of install size for who would not want to use the current version).

Also, with easier packaging (that abstract even dependencies), this could means allow put even more features for general public.

[johnjcamilleri]

While I believe that setting up a PPA is a good thing, it does seem like some extra work which comes with new problems and demands on our efforts. The simplest solution to the current problem is really to just build two binaries in the CI, for Ubuntu < 20 and >= 20 respectively. So I say we do this for the release, and the PPA effort can continue independently.

[fititnt]

@johnjcamilleri Not sure about the others, but simply 2 binaries (in special if want to release soon) seems to be a great win-win on short term!

Also, the idea of setup a PPA #125 is shiny new. I'm not sure if you here are all familiar with the GitHub actions and that stuff (I know Github Actions, Docker, etc, but not this type of binary distribution) but I think is possible to actually "plug even development releases" on some PPA or some of these distribution stores. But all these extra features could take weeks if want like make more automated. Ok, actually more than weeks, since to be sure if it works on automation pipeline, would need at least another release.

About Ubuntu

From https://ubuntu.com/about/release-cycle (in special if is not something more specialized, like PPA) I believe both 18.04 LTS and 20.04 LTS are good references to release today (eventually 22.04 LTS would be a thing, but at that point you here could decide to drop 16.04 support). Some Ubuntu-like flavors tend to to be based on some of these. Also, when "naming the Ubuntu binaries" the use of 18.04 / 20.04 as a hint can help users understand what is likely to work on first try, since the generic Ubuntu 32bit or Ubuntu 64 Bits is not very specific.

I know that do exist several ubuntu versions, but often the LTS ones (if need to restrict resources, like compile individually) tend to be a good focus. I'm not even sure if try to like setup automated testing for non-LTS version like the 21.04, will be easy to find docker images.

[anka-213]

I think we will also need to update debian/changes to make the ubuntu packages have the correct version numbers. I discovered that while trying to make a ppa.

[anka-213]

Hmm, the build was aborted after trying to build for 6 hours. https://github.com/GrammaticalFramework/gf-core/actions/runs/1054618733
I wonder what happened!

[anka-213]

I think this pull request is ready to merge now. It builds two versions for ubuntu, one for ubuntu 18.04 and one for 20.04. See this build for an example.
https://github.com/anka-213/gf-core/actions/runs/1063635053

I was looking into having it automatically upload the artifacts as release assets as well, but I've had some trouble with it and it probably belongs in a separate pull request.

Similarly, updating ubuntu/changelog with the new version does not belong in this PR.