On what platform to run the Docker when setting the environment - Raspberry Pi or x86?

[developerdudeone]

Hi,

Thank you very much for your time and effort in allowing us to take advantage of these technologies.
I am new to containerized type of development. I realize my question may sound trivial.
Here it is what I want to achieve:

I would like to run a Cardano pool on two Raspberry Pis (8 GB RAM version)
I installed Docker then downloaded your Cardano docker and followed the instructions how to compile the entire setup.
However, it fails during the compilation on the Pi 4B with the following error messages as shown below.

My understanding is that we can run this on the Raspberry Pi itself, but maybe that's not the case due to my lack of experience with this technology.
Could you please clarify if this can be run on the Pi or I need to create the docker image on a Linux machine x86 and then copy it to the Pi?

All the best,
Alex

pi@raspberrypi:/usr/local/lib $ sudo ld.gold -ltsan --verbose
GNU ld (GNU Binutils for Raspbian) 2.31.1
Supported emulations:
armelf_linux_eabi
armelfb_linux_eabi
..................................................

attempt to open //usr/local/lib/arm-linux-gnueabihf/libtsan.so failed
attempt to open //usr/local/lib/arm-linux-gnueabihf/libtsan.a failed
attempt to open //lib/arm-linux-gnueabihf/libtsan.so failed
attempt to open //lib/arm-linux-gnueabihf/libtsan.a failed
attempt to open //usr/lib/arm-linux-gnueabihf/libtsan.so failed
attempt to open //usr/lib/arm-linux-gnueabihf/libtsan.a failed
attempt to open //usr/local/lib/libtsan.so failed
attempt to open //usr/local/lib/libtsan.a failed
attempt to open //lib/libtsan.so failed
attempt to open //lib/libtsan.a failed
attempt to open //usr/lib/libtsan.so failed
attempt to open //usr/lib/libtsan.a failed
attempt to open //usr/arm-linux-gnueabihf/lib/libtsan.so failed
attempt to open //usr/arm-linux-gnueabihf/lib/libtsan.a failed
ld.gold: cannot find -ltsan

[pascallapointe]

Hi Alex,

yes this repository support both ARM and X86 architecture, but I'm using it mostly on RaspberryPi (ARM).
I'm actually running a fresh compilation just to make sure that nothing is broken in there.

I never tested it under Raspbian, only under Ubuntu, but technically it shouldn't matter.

Just to make sure, have you ran the following command for building on RPI?

ARCHITECTURE= aarch64

docker build \ -t cardano_env:latest \ ./Dockerfiles/build_env/${ARCHITECTURE}

[developerdudeone]

Hi Pascal,

Thanks a lot for your answer.
Yes, I did set the architecture to aarch64.
For the most part it ran nicely up until it needed to link using the ld.gold
I ran it under the Raspbian Buster on the Pi 4B.

This Pi I have is pretty well configured with a M2 SATA SSD and overclocked to 2 Ghz.
I was so hopeful this will succeed. I went to the official Debian ARM repository and installed this package:

• libtsan0_8.3.0-6_arm64.deb Despite that it did not work...

I am curious if this build ran successfully under a Pi 4B running latest Ubuntu.
Did you try it on Pi with the Ubuntu 20.10?
I could eventually install Ubuntu 20.10 on this Pi but is a bit cumbersome, because I need to open the case again and put the micro sd card and somehow from there I need to figure out how to copy the content of micro sd into the M2 SATA SSD.

I will wait for your suggestions.

All the best,
Alex

[pascallapointe]

Yes the build was successful on the Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1028-raspi aarch64) (RPI4b 8GBram).
I ran it with --no-cache to build it from scratch. It took more than 8 hours, wasn't done yet when I left for work this morning..!

Did not tried on 20.10.

I checked on my monitoring and the PI did spiked enough to required the swap, so make sure you have a swap file enabled.

I really wonder why you encounter these issues, I'll need to do some research and tests. This is interesting.
Your issue isn't related to your hardware, I would look more on the OS side even if that doesn't make sense to me. Still, this is mostly the difference between your setup and mine.

In the mean time, I invite you to try something, IOHK developper have produce a fully static node binary available on a telegram channel, here the invite link: https://t.me/joinchat/FeKTCBu-pn5OUZUz4joF2w So this is a pre-compiled version of cardano-node and cardano-cli that include all required libraries, so it should run on your system.

On my side I'll give a try with Raspbian as soon as I have some free time. I only have a 4GBram RPI4 free for the test, so I hope that will do with a good old swap file.

Pascal

[pascallapointe]

One other thing I can think of is OS configuration regarding limits. That's just an idea

Example, the library file ibtsan.so could be too large to be opened in memory accordingly to your OS limit settings.

I'm not very savy in that field but just for the sake of comparison, here what I got on my system:

Command you need to run: ulimit -a

My system ulimit bellow:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30861
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 800000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 30861
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[developerdudeone]

Hi Pascal,

Thanks a lot for your answers.
Here it is the result of the command ulimit -a:

pi@raspberrypi:~ $ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 59507
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 95
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 59507
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
pi@raspberrypi:~ $

[developerdudeone]

Hi,

I just ran these two commands as shown below.
I wonder if I should increase the open files limit...

pi@raspberrypi:~ $ ulimit -Sn
1024
pi@raspberrypi:~ $ ulimit -Hn
1048576
pi@raspberrypi:~ $

[developerdudeone]

Hi,

I tried again a fresh compile and still got error as seen below.
I may need to switch this Pi to Ubuntu 20.04 and see if it works.
In the meantime if you have any suggestions about Raspbian I would appreciate if you could let me know.

All the best,
Alex

/usr/bin/ld.gold: error: cannot find -ltsan
collect2: error: ld returned 1 exit status
gcc' failed in phase Linker'. (Exit code: 1)
make[1]: *** [rts/ghc.mk:321: rts/dist/build/libHSrts_tsan-ghc8.6.5.so] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:127: all] Error 2
The command '/bin/sh -c make -j3' returned a non-zero code: 2
pi@raspberrypi:~/Downloads/cardano-docker-master $

[pascallapointe]

Yes if I think to something else I'll tell you, sorry that this is not working out of the box for you :/

So I assume that you also still had attempt to open //usr/arm-linux-gnueabihf/lib/libtsan.a failed error,
have you tried increassing open files limit just in case?

Also, I just notice that a lot of new docker for cardano-node popped up on github since I first publish mine.
I didn't checked if they support ARM system though, but maybe an other one might work.

I'll give a try on Rasbian Buster and tell you if I run in the same issue

[developerdudeone]

Hi Pascal,

I managed to install on my Pi4 Ubuntu 20.10.
I ran the docker build file and it took in total 16 hours to finish the entire build on Pi.
I am pretty excited that this is possible with such small piece of technology.

I have a question related to the tagging after the successful Docker node build.
You are mentioning in your documentation that we should use the version: 1.19.0
Is this the version of the Cardano node or your Docker build number?
According with your Git tag your build is at 1.1.0.

I don't want to screw anything lol, so I want to make sure I give the correct version there.
Also when I execute the command:

• docker commit cardano_env cardano_env_bkp
  I get an error saying such container does not exist. I apologize for my trivial question, but this is an ongoing learning process for me.

Basically, I want to create a backup of the current container and the do a: docker save image that I save it for later.
I also have seen there is another container named: focal
Is this container created by your Docker build?

If that is the case I assume this one should be saved too if I want to transfer this docker container into another Pi 4.
Could you clarify that, if I need to do a backup of it too?

Thanks a lot for your time.

All the best,
Alex

[pascallapointe]

Yes this is pretty amazing that it actually run on RPI!

The version 1.19.0 is just an example, your have to keep up with the official IOHK cardano-node latest release, which is actually 1.25.1. So every times they update cardano-node, your have to build again by applying the latest carano-node version to VERSION_NUMBER.

The good thing is that you don't have to rebuild cardano_env every time as this is the building environment. You only build a new cardano_node image.

The version of my docker file (1.1.0) is related only to my repository, this is why it doesn't follow cardano-node versions.

Actually the documentation of my repository is a little outdated and some information are missing about the correct folder structure. I can see you already spotted one by reading the pull-request you made. I'll go through it this week so you have the correct information.

About the docker commit command:
This command is use to save a running container state to a new image. Actually, if you want to make a backup of an image to prevent loosing it by building again, you must use docker tag.

You had an error because cardano_env is an image, not a running container. cardano_env is a base image, a version of Ubuntu containing all the dependancies necessary to build the cardano_node image.

So actually, by running docker images you have cardano_env:latest.
If you want to make a backup of that image you have 2 options:

1. Tag it with a new tag (it create a second tag, it doesn't erase the :latest tag): docker tag cardano_env:latest cardano_env:backup. This method is the preferred one for local backup, it's instantaneous.
2. Save it to your drive (that will create a tar archive): docker save cardano_env:latest | gzip > cardano_env_latest.tar.gz. This is the preferred method to send the image to an other RasperryPi or for external backup.

So by your next question, I can see that you mixup container and image (That's ok! Using container are a steep learning curve at first). So if you look at the top of the Dockerfile used to build the cardano_env image, you will see from ubuntu:focal. This is the base image. A vanilla Ubuntu 20.04 (focal).
And cardano_env is the base image of cardano_node. So docker download the base image from an external repository so it can build the carano_env image on top of it. This is why you can see it.

No need to backup this image.

When you will be done building cardano_node image, you will be able to save it with docker save command and send it to your other RaspberryPi.

Now, the difference between an image and a container, is that you can spin-up many container from the same image. If you check further in the documentation (check in standalone link at the bottom), you'll notice the docker run command. This is the command used to create a container from a docker image. So you first build the image, than you create container from these images. You can create many container running in parallel from the same image, you just have to give them different name.

If you used docker-compose to orchestrate all this, the container name is defined in the docker-compose.yaml file.

To see all running container, use docker ps, or docker container ls (alias)
You can add "-a" to list also stopped container. This is really usefull when you need to remove old containers.

Also, you will notice that Docker prevent you from removing an image when a container use it, docker ps -a will allow you to see stopped container and remove them.

I hope that help a little and answer some of your questions.
Once you'll master docker basis, it will become (almost) a no brainer to you. We all went by the painful learning curve at first :)
And I'm not done learning yet..!

I'll be there if you have other issues, cheers!

[developerdudeone]

Hi Pascal,

Thank you very much for taking time and clarifying these matters for me.
I am an experienced developer but not in the area of the Cloud technologies.
I am doing mostly EAI type of development with Oracle OSB and Mulesoft ESB.

This entire experiment about learning cryptocurrencies and their associated technologies is a great learning process for me and also a welcomed skill upgrade.
Yes, I was a bit confused about the concept of container vs image. I think your explanation clarified it for me really nicely.
Just few more questions I have if you don't mind:

My intention is to save this image as pristine is after this long build.
The reason behind it is to have it as is for my second Raspberry Pi on which I will set the Cardano Node Producer.
I understand from the Cardano's documentation that the configurations for a Relay Node and Node Producer are slightly different.
In that respect I do not want to proceed with the steps mentioned by you after the initial build is done; or should I do them regardless? Could you clarify this particular matter for me?

Again, I would like to express my thanks for your help.
If ever I could be useful with something in the future for your projects let me know.
I also have background in Java development.

All the best,
Alex

[pascallapointe]

Alright, I'll take note of that!

About keeping the image pristine/untouched

Once the initial build is done (cardano_env image), you can proceed with the next step without any risk.
The cardano_env will stay as is on your system and the next build will use it in a "read-only" fashion.

Every time cardano-node will require a version upgrade, you'll be able to build a new cardano_node image without the need of building again cardano_env image.

Once an image is created, it never change.

About docker image bakcup

I would recommend that you to tag cardano_env:latest as cardano_env:backup or "docker save" it to a TAR archive just to avoid loosing it by mistake.

Docker images are rarely loss unless you purposely delete them. But if you build a new version of them using the same name and tag, the new one will bear the name and tag as the older one will be displayed only by its hash ID, which looks like this: b32d948b47ff. Not very convenient to retrieve it if you need it again. So by giving them a unique tag, even if cardano_env:latest is overwrite, cardano_env:meaningful_other_tag will remain.

Image VS Container backup

You might later wish to save a running container state for backup purpose or any other reason. You can achieve that by using docker commit command. That will create a NEW image, not erase the one upon which the container as been created.

What you need in the end to operate with docker-composer

The image that you will send to your other RaspberryPi will be the cardano_node image along with the cardano_monitor image. The cardano_env is used only as a base image for cardano_node image. (You can notice it in in ./Dockerfiles/node/Dockerfile first line)

Relay node VS Block Producer node

Yes it is in deed just a matter of configuration that separate one from the other. Once you've build the cardano_node image, you'll be able to use it for creating both your relay container and your block producer container.

The magic happen when starting the node. In short, a block producer node will be started using 3 more files than a relay node. (vrf.skey, kes.skey and your operation certificate file)
You can check the different startup scripts to see the difference.

The container will 'bind' to a local config folder on your host system. This folder will hold the necessary configuration file. Then it's a matter of starting it with the right start script.

On the network side, the host that hold your producer node must not be reachable from public internet, or at least filter IP address to allow only your relay node to connect to it.

The other things to consider is the difference in topology file. A block producer will only contain your relay node in its topology.

Missing part in my instructions

I've recently made some change to the docker-compose.yaml file so you can access the database files and node socket directly on your host system.
This part is still missing in the instruction, so just add a db folder and a socket folder on the same level that you've created your config folder.

Security consideration

Docker service can be configured to be controlled remotely, Never enable that unless you really know what you are doing. Always connect to your host via ssh and control docker from there.

[developerdudeone]

Hi Pascal,

Thanks a lot for your detailed answer.
I have a question regarding this part from your documentation:

Node configuration
Now you've created yours images, it's time to create your config folder. Your container will bind to this folder, so you can access your configuration from within.

mkdir config
cd config

Question:

• Where do I create the config folder? Is there a particular path needs to be created on?

I assume this would be created in the folder path: /home/pi/ ; however I am not sure about this... could you clarify it?

All the best,
Alex

[pascallapointe]

Thanks for highlighting unclear parts, I just updated it to add more informations about folder structure and files permissions. You can see latest changes on master branch.

If you open docker-compose.yaml file you'll notice that the config folder path look like this: source: ./config

The dot (./) in the path makes the path relative to your current location. So the container will look in your current folder to find /config.

You can create a /home/pi/Cardano to hold all your subfolders:

/home/pi/Cardano/config
/home/pi/Cardano/socket
/home/pi/Cardano/db

Then all you need to do is launch the cmd shown in my instruction from /home/pi/Cardano folder. You also could edit docker-compose.yaml or the docker run command to add and absolute path to your config folder, that would allow you to launch the command from anywhere.