Build conda packages for ARM

[jakirkham]

@show0k commented on Fri Sep 02 2016

Hi,
Did you ever think about building conda package for ARM architecture ?
Since August 2015, conda is officially supporting ARMv7l architecture.
It is possible to start ARM Docker container from an intel-64 arch thanks to qemu-arm-static. It could be used to build every conda-forge packages for arm platform with travis-ci or circle-ci.

Now a day I'm building almost every conda packages I need for my ARM board because there few official builds, but I think that it could have a growing ARM/Raspberry Pi community if there were more official or at least conda-forge packets.

Do you think that there is too few ARM users for conda package that it does not worth it ?

---

@jakirkham commented on Mon Sep 05 2016

There's a good chance that one will need to run such a Docker container in privileged mode. Not sure that CircleCI or other CIs support that.

---

@jakirkham commented on Mon Sep 05 2016

This may be a good tool for containerizing VMs.

---

@show0k commented on Mon Sep 12 2016

There is no need to run container in privileged mode. Usualy to run an ARM docker image on a x86_64 host it needs a kernel with binfmt_misc support, which is the case for travis-ci but not for CircleCI.

But, with a minor modification on qemu, it is possible to embed it on the image and run the ARM image without anything special on the host.

I made a POC image on show0k/miniconda-armv7.

---

@groutr commented on Sat Sep 24 2016

I wonder if, in this case, using the linaro tool chain to cross compile would be a good a option.

---

@show0k commented on Mon Sep 26 2016

I think that the Docker idea is better than the cross-compliation, it emulates the whole system, which is easier to integrate with the actual conda & conda-forge scripts.
In background, they both use Qemu for the ARM emulation, so the overhead is the same.

---

@pelson commented on Thu Sep 29 2016

I'm totally up for this. I've suffered conda build on the RaspberryPi and it was a terrible experience (conda build ... ... wait.... wait some more... go to sleep fail. conda build...... wait ... wait some more ... go to work .... fail). My recipes lived in https://github.com/pelson/raspberrypi-conda-recipes and I got quite a way into the stack (in fact, many of those recipes were the basis for conda-forge recipes).

The major things that I'd want to consider though:

• how we reduce the impact on build time for the existing conda-forge system
• whether we can improve on the existing noarch conda concept to save us having duplicate conda distributions for non-compiled things
• how we integrate this into conda-forge without breaking existing feedstocks (that may not yet compile on ARM)

---

@jakirkham commented on Thu Sep 29 2016

One idea I was toying with earlier was actually having another org for this. It may sound a bit extreme, but this is something that one of the Homebrew devs did when they wanted to support Mac OS 10.4 (Tiger). At least worth keeping in the back of our minds if nothing else.

xref: https://github.com/mistydemeo/tigerbrew

---

@show0k commented on Thu Sep 29 2016

The problem of build time is really an issue we have to deal with. Throw Qemu the build time on my i7-5557U@3Ghz laptop is two time slower than on Raspberry Pi 3.
I mainly use a Odroid XU4 which is far faster than a Raspberry Pi for testing new builds rules.

I think that we can use another free CI provider, in the same way you separate Linux and OSX build with CircleCi and Travis CI. This would solve the impact on build time for x64_86 build, and avoid labelling the existing build as faild.

Another solution could be to use a paid farm of ARM hardware which run Jenkins or Drone on it. Scaleway have great and cheap hardware, but I suppose that managing money and donations for build is out of the scope of conda-forge.

---

@pelson commented on Thu Sep 29 2016

Managing money is achievable (e.g. NumFOCUS), managing hardware is harder (and the software infrastructure around it) - it would ultimately take time away from other conda-forge activities. I have long wanted a CI service where we can attach donated systems (e.g. personal laptop cycles, corporately donated AWS instances etc.), but we aren't there yet 😉 .

Your Qemu experience doesn't sound like it was much fun 😄

---

@jakirkham commented on Tue Nov 15 2016

I'm going to move this over to the webpage repo for a few reasons.

1. This is an organization discussion.
2. It will get more visibility.

Hope that is ok. Thanks everyone for bring this up and providing interesting suggestions. Looking forward to seeing what others might contribute.

[jakirkham]

cc @maxnoe

[jjhelmus]

I'm been building a bunch of conda packages for the Raspberry Pi 2/3 (armv7l) and 1/0 (armv6l). They are available on the rpi channel on Anaconda.org.

The recipes were mostly derived from conda-forge (thanks all for such a great resources) and are available in the jjhelmus/rpi-recipes repo. I'll be adding docs, a README, proper license, etc and likely renaming this repo in the next week but wanted to give conda-forge folks a heads up.

[ChrisBarker-NOAA]

Is the pi identifiable as a unique platform by conda?

i.e. is there any reason these packages couldn't be put up on the same conda-forge channel?

Maybe there is no reason TO put them in the same channel -- but I kinda like the "branding" advantage of being able to use conda-forge for everything :-)

[dopplershift]

Well, unless I'm mistaken, one big difference is that @jjhelmus is building the packages manually--or at least locally. No raspberry pi CI services I'm aware of.

[ChrisBarker-NOAA]

yeah -- probably not.

But you can still push to the same anaconda.org channel by hand, yes?

Maybe we want to keep the channel "clean" in that sense, though.

[show0k]

@ChrisBarker-NOAA There is a tag for any platform, and as @jjhelmus said, it is the armv7l tag for Raspberry Pi 2/3 and armv6l for Raspberry Pi 1.
If building all conda-forge packages for Raspberry Pi is too much work for the impact, a solution could be too activate manually the armv7l building process.
In this way, the conda-forge project will get the hype effect (and the community) of Raspberry Pi with a a minimum work.

Moreover, to add a word on my first post on the technical solutions to build conda packages for ARM, I change a bit my mind. Qemu is a great project, but using it for ARM emulation will cause 5% of time some strange bugs which take lot of time to resolve.
Whatever the community of people which are involved (conda-forge or another) I think we have to host a CI software (circle-ci or travis) on ARM hardware.
Scaleway is the most famous (the only ?) cloud provider of ARM bare-metal server, it is quite cheap, and way much faster than a Raspberry Pi 3.

[jjhelmus]

The packages in the rpi channel were made locally by a two Raspberry Pi 3's and a Raspberry Pi 1 and Zero. Since they are not made as part of the standard conda-forge workflow I'm of the opinion that they should not be included in the conda-forge channel.

I think it would be really cool if linux-armv7l packages could be built by conda-forge using a CI solution (even an optional one) but do not have the time to work on the technical challenge around this at the moment.

@show0k Thanks for the details on QEMU and ARM bare metal hosting, I'll have the check out Scaleway's offerings. I know there are a few places that offer hosted Raspberry Pis which might be another option worth looking into.

[ChrisBarker-NOAA]

all sounds good.

@jjhelmus: I think it's really cool that you're doing this!

[jakirkham]

Just learned recently that Shippable supports ARMv8-A, which would provide us a way to build and upload packages for ARM. Not seeing anything about support for older ARM processors, but it's possible I'm missing something.

[scopatz]

@jakirkham Cool!

[zbeekman]

Is there ARM64/aarch64 support for conda somehow? (Even if I have to compile sources myself?)

I'm relatively new to conda/anaconda, and have been tasked with porting https://github.com/ParaToolsInc/taucmdr to aarch64. Thanks.

[jjhelmus]

Is there ARM64/aarch64 support for conda somehow?

conda and conda-build support the linux-aarch64 platform but there are not many sources of packages. Over the weekend, I created a set of aarch64 packages which should be sufficient to begin the building process. Specifically, that channel contains Python 3.7 versions of conda, conda-build, constructor and anaconda-client as well as their dependencies. In addition there is a gcc 7.3.0, binutils 2.29.1 based toolchain which targets glibc 2.17 as well as all the packages needed to rebuilt the toolchain from source.

If anyone would like to collaborate on getting aarch64 build support into conda-forge I would enjoy working together on such a task.

[zbeekman]

I’m a conda noob, and time is pretty tight these days. Happy to help a little bit but I need to be pointed in the right direction and might require some tutoring at first.

…

On Mon, Oct 22, 2018 at 6:23 PM Jonathan J. Helmus ***@***.***> wrote: Is there ARM64/aarch64 support for conda somehow? conda and conda-build support the linux-aarch64 platform but there are not many sources of packages. Over the weekend, I created a set of aarch64 packages <https://anaconda.org/jjhelmus/repo?label=aarch64_bootstrap> which should be sufficient to begin the building process. Specifically, that channel contains Python 3.7 versions of conda, conda-build, constructor and anaconda-client as well as their dependencies. In addition there is a gcc 7.3.0, binutils 2.29.1 based toolchain which targets glibc 2.17 as well as all the packages needed to rebuilt the toolchain from source. If anyone would like to collaborate on getting aarch64 build support into conda-forge I would enjoy working together on such a task. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#269 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAREPNa2gXboz7fRKlDIPIR1xsGreVpyks5unkVbgaJpZM4KzB4C> .

[ChrisBarker-NOAA]

@jjhelmus: wouldn't that require an ARM CI? or is it possible to cross-compile?

[jjhelmus]

Shippable provides an ARM based CI which is free for open source projects. I was hoping that it can be used by conda-forge for aarch64 builds.

[zbeekman]

See also: https://github.com/sedden/docker-rpi-plex-server/issues/1 It looks like you can *probably* do this with Docker on Travis-CI.

…

On Wed, Oct 24, 2018 at 5:38 PM Jonathan J. Helmus ***@***.***> wrote: Shippable provides an ARM based CI <http://docs.shippable.com/platform/tutorial/workflow/run-ci-builds-on-arm/> which is free for open source projects. I was hoping that it can be used by conda-forge for aarch64 builds. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#269 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAREPD9sxVxinTNI1rN1w-Z-M3NwRTYIks5uoN3jgaJpZM4KzB4C> .

[show0k]

Edit: I didnt' see @jjhelmus message, it sounds very good!

--
@zbeekman It was the purpose of my first message on that thread. I even did a docker image for ARM with conda (I didn't test if it still works).

But.. after many attempts, it has two drawbacks:

• It is very slow. Two x86 2Ghz vCPU are far slower than a Raspbery Pi 2 for instance. So more builds will hang after 1h of compilation on travis-ci
• Some builds fail due to qemu, it's rare but it happens and very hard (impossible ?) and annoying to fix.

I think the best solution is to use GitLab CI with ARM computers. With GitLab CI you can add any computer with docker installed as a worker, even of it's behind a NAT. An efficient solution could be to pay for ARM servers on Scaleway (the only ARM server provider as far as I know) and use the autoscaling feature of GitLab which works with docker machine (in a way similar than this) to start and stop ARM servers on demand.

As GitLab CI workers can work behind a NAT, many people can share a Raspberry Pi connected with Docker on it. But it has to be used only for testing builds and not packaging otherwise it could create security issues.

[hmaarrfk]

It seems that there are a few common key architectures that might be useful:

1. The RaspberryPi. Most people are on raspbian which targets armv7l (32 bit, hardware floating point)
2. aarch64 (64 bit, hardware floating point)

are there any other hardware platforms we should target?

[hmaarrfk]

@jjhelmus I was just able to request and obtained two node pools for aarch32 and aarch64 relatively quickly. I could request some for conda-forge "github" org too, but I figured I would ask somebody with actual rights before going ahead with that.

Xref: https://github.com/hmaarrfk/arm-conda-playground
This is mostly a skeleton to start buiding out the infrastructure we would need.

[hmaarrfk]

So I've kinda started to build some of the infrastructure this weekend.
You can find it at
https://github.com/archiconda/

The real repo that does all the work is; https://github.com/Archiconda/build-tools

I'm definitely stuck building out the compiler's dependencies: Archiconda/build-tools#2

definitely a work in progress, but a few critical things, like the structure to build out the docker image, forking conda-forge repos and automatic uploading to anaconda.org are more or less there

edit: I forgot to mention the most important thing: if anybody wants to spend their break helping out, just ask me and I'll gladly give you access. For now I'm starting out to build the necessary components manually!

[jjhelmus]

There are two primary ARM platforms that would be useful to support, 64-bit ARMv8 typically referred to as aarch64, and 32-bit ARMv7. There are a number of variations in floating point hardware support in ARMv7 cores. Many Linux distributions target the subset of these cores which provide at least VFP3-D16 hardware floating point. This subset is often referred to as armhf after the Debian port.

Raspberry Pis use a variety of CPU depending on the model. The 1 and Zeros have ARMv6, the original 2 has ARMv7 and the 2 version 1.2 and all 3's have a ARMv8 CPU. Although ARMv8 supports 64-bit mode most Raspberry Pi distributions boot into 32-bit mode and uname -p will report the processor as armv7l. Raspbian run the same binaries on all models and therefore targets ARMv6 with VFPv2 which is a key difference between Raspbian and the armhf port of Debian.

Conda packages could be made for armv6l for use on the Raspberry Pi 1 and Zero but I suspect armv7l and aarch64 packages would be more useful. There should be no issues using armv7l conda packages on Raspberry Pi 2 and 3 running Raspbian if the packages uses the same architecture compiler flags as the Debian armhf port.

Berryconda provides armv6l and armv7l packages that work on the various models of Raspberry Pis. The armv7l packages target a different hardware minimum than the Debian armhf port, specifically armv7l VFPv4 with NEON, and may not work on other armv7 systems. Berryconda uses system provided compilers. @gaiar has done some work on building armv7 compilers, see jjhelmus/berryconda#39.

I created an initial set of of packages for aarch64 which are available on Anaconda.org in the jjhelmus/label/aarch64_bootstrap channel. These include gcc 7.3.0 compilers which create binaries compatible with glibc 2.17 which is used in CentOS 7. 7 was the first CentOS release to support aarch64. The compilers were built on a Ubunutu 18.04 host so those packages must be run on a system with glibc 2.27 or newer but the other packages should work on anything newer than CentOS 7.

[hmaarrfk]

@jjhelmus it seems strange that the compiler won't run on the same machine as the target. My thought are that this makes it harder to test on CIs.

Is there any way to recompile the compilers so that they run on CentOS 7+?

[hmaarrfk]

For the record, I don't really care about saying that we only support ubuntu 18.04 and greater. 18.04 seems old enough for me ;)

[jmreicha]

@bradrydzewski Is there any (easy) way to get around the untrusted repositories cannot mount host volumes issue?

Seems like that is going to be a problem for doing Docker in Docker builds.

[bradrydzewski]

@jmreicha drone provides a plugin for building and publishing docker image. Check out http://plugins.drone.io/drone-plugins/drone-docker/. Since Drone Cloud is a shared environment we cannot allow host machine volumes mounts for security reasons.

[jmreicha]

Okay, thanks. I'm going to have to rethink how I am doing things a little bit.

[jmreicha]

So I made an interesting discovery, maybe somebody can tell me what's going on.

I have a build working with Drone, but the only way I have found to get it to complete instead of timing out at the one hour mark is to comment the run_conda_forge_build_setup line in the build_steps.sh script.

Once I remove that line my test build time drops down to 18 minutes. The weird thing is that line only seems to affect the aarch64 builds. Any ideas?

[isuruf]

Might be because it sets CONDA_BLD_PATH which is on a mounted volume.

[jmreicha]

More info. I used the azure pipelines code as a base for creating my drone configs.

There is a reference to run_conda_forge_build_setup in configure_feedstock which I basically copied into my code.

[jmreicha]

@isuruf Probably okay to skip then since Drone doesn't do anything with volume mounts?

[isuruf]

No. There's multiple things happening in that script and prints useful information. Can you debug what the issue is?

[jmreicha]

I'll see what I can do - I'm having trouble understanding where the build setup is coming from and what it is doing. Here's what I see in the conda smithy rerender.

appveyor:
  all_platforms:
  - win
  enabled: true
  image: Visual Studio 2017
  platforms: Windows
  secure:
    BINSTAR_TOKEN: tumuXLL8PU75WMnRDemRy02ruEq2RpNxeK3dz0MjFssnosPm2v4EFjfNB4PTotA1
azure:
  all_platforms:
  - linux
  - win
  - osx
  build_id: 690
  enabled: true
  force: true
  platforms: Linux,OSX,Windows
  project_id: 84710dde-1620-425b-80d0-4cf5baca359d
  project_name: feedstock-builds
  timeout_minutes: 360
  upload_packages: false
  user_or_org: conda-forge
build_setup: 'run_conda_forge_build_setup

  '
channels:
  sources:
  - conda-forge
  - defaults
  targets:
  - - conda-forge
    - main
circle:
  enabled: false
compiler_stack: comp7
configs:
- - linux_aarch64_
  - linux-aarch64
  - true
docker:
  command: bash
  executable: docker
  image: condaforge/linux-anvil-aarch64
  interactive: true
drone:
  all_platforms:
  - linux_aarch64
  enabled: true
  platforms: Linux
exclusive_config_file: /home/jmreicha/anaconda3/conda_build_config.yaml
fast_finish: ''
github:
  branch_name: master
  repo_name: nodejs-feedstock
  user_or_org: conda-forge
idle_timeout_minutes: null
linux:
  enabled: true
linux_aarch64:
  enabled: true
linux_ppc64le:
  enabled: true
maintainers:
- minrk
- msarahan
- pelson
max_py_ver: '37'
max_r_ver: '35'
min_py_ver: '27'
min_r_ver: '34'
noarch_python: false
osx:
  enabled: true
outputs:
- nodejs
...

What are some other things to look at to debug?

[isuruf]

That script is coming from conda-forge-ci-setup package. Try running the commands from that script instead of running the script and remove steps one by one to see what affects the build

[jmreicha]

Think I might have found the problem.

It looks like the build_setup script is setting the CPU count to 2 and it never gets updated after that.

I updated the local build.sh script to spit out the configured number of CPUs and you can see the results in my test build (line 584 and 585).

[isuruf]

What's a reasonable value for CPU_COUNT on drone?
In recipes we use make -j${CPU_COUNT}, so that there'll be parallel jobs, but not too much so that resources like memory are not exhausted.

[vielmetti]

The physical hardware is a 96-core, 128GB ThunderX, best described as "lots of weak cores". It's very hard for most build systems to schedule things so that they successfully use all 96 cores at once, and it's also super rare to absorb the whole memory footprint in a single go. I would crank it all the way up, see what you get, and then govern it back down as experience teaches you.

@packethost has other arm64 hardware in the available pool that has fewer core but better oomph per core, if/when that gets added to the @droneio system then CPU_COUNT could potentially vary between runs.

Basically I'd like to optimize for build times, if you end up getting throttled on something other than CPU that would be something to explore in more detail.

[jmreicha]

I think the build is using all the cores then if that value doesn't get set, and the build time is around 16-18 minutes, but I will do some more testing to verify.

[jmreicha]

I opened a PR to add Drone, please feel free to jump in if you have input/feedback. I could use a hand with the CI integration if anyone is interested.

[yunqu]

One thing to mention is that if you are not willing to touch QEMU / docker, you can always use the Amazon AWS to open an aarch64 instance for you. This way you save a lot of efforts when you come across potential QEMU bugs (we have seen QEMU bugs when cross compiling bazel on aarch64).

I am also extremely interested to see porting conda onto aarch64.

[vielmetti]

@yunqu - have those Bazel+QEMU bugs been reported? I would love to help track those down. I know that QEMU 3.x fixed a bunch of issues that were present in QEMU 2.x.

[yunqu]

@vielmetti I don't think so. Original issue here: bazelbuild/bazel#7135 That issue is not related to this issue though.

[isuruf]

Does anyone know if cloud.drone.io support jsonnet configuration files?

[vielmetti]

Hi @isuruf

I see docs on the jsonnet extension for Drone here

https://docs.drone.io/extend/config/jsonnet/

I've asked in their forum at https://discourse.drone.io/t/drone-cloud-and-jsonnet/4621 and will hope for an answer.

[vielmetti]

from Brad at Drone in the above forum:

Hey Ed,

Right now the only option is to use the jsonnet command line utility to generate a yaml file from jsonnet, and commit the generated yaml to your repository. This is what we do for all 100+ Drone repositories. We absolutely will support native jsonnet in the future, but we first need to work through some of the security implications – it is not considered safe to evaluate untrusted jsonnet files server-side [1].

[1] https://jsonnet.org/ref/bindings.html#server_side

[isuruf]

@vielmetti, is it possible to get access to an aarch64 server mainly for building packages that take longer than 1 hour given by Drone CI? (And also for debugging some package build failures)

[vielmetti]

Hi @isuruf - I'd be happy to review a proposal for access to an Arm server to help things out, either short term or long term.

https://github.com/worksonarm/cluster is our "Works on Arm" cluster project, and we regularly get proposals to support CI/CD and debugging efforts. To make a request just open an issue and it will get a look from me at Packet and from folks at Arm. There's a pretty good crew working on servers in this environment + I'm also happy to introduce you to other people with relevant interest and experience.