Removing the build of LLVM sources as part of nballerina build

[ayonam]

The nballerina build currently goes through a build of the LLVM sources followed by a build of the nballerina sources.

The nballerina build depends upon some headers from the LLVM sources and some headers that are generated as part of the build process and libraries that are generate by the build.

We need to explore if we can do away with the LLVM source build altogether and use the standard released binary bits of LLVM to carry out our build process. Probably there is a Developer's package also. This needs to be explored.

Alternately, if we can get the sources, build them once and keep them in the repository and use them as needed.

@shubhamnarlawar Can you please explore this?

@uthaiyashankar @sameerajayasoma @manuranga

[shubhamnarlawar]

I tried below process to achieve above requirement -

1. I have build and installed LLVM-11.0 and used only installation directory for building nballerina. There are 3 library/header support missing in llvm installation directory -

/usr/bin/x86_64-linux-gnu-ld: cannot find -lLLVMTestingSupport
/usr/bin/x86_64-linux-gnu-ld: cannot find -lgtest_main
/usr/bin/x86_64-linux-gnu-ld: cannot find -lgtest
collect2: error: ld returned 1 exit status
Makefile:95: recipe for target 'nballerinacc' failed

-------------------------------------------------------Build commads-------------------------------------------------------------------------

cmake -DCMAKE_BUILD_TYPE="Release"
-DCMAKE_INSTALL_PREFIX=/home/shubham/build/llvm/llvm_install
-DBUILD_SHARED_LIBS=True
-DLLVM_USE_SPLIT_DWARF=True
-DLLVM_OPTIMIZED_TABLEGEN=True
-DLLVM_BUILD_TESTS=True
/home/shubham/llvm-11.0/llvm-project-11.0.0/llvm

make install
make install-llvm-headers
make install-llvm-libraries

[shubhamnarlawar]

To tackle above issue, I tried to build gtest library using below link -

https://github.com/google/googletest.git

which solves support for - -lgtest_main and -lgtest and nails down above issue to -

/usr/bin/x86_64-linux-gnu-ld: cannot find -lLLVMTestingSupport
collect2: error: ld returned 1 exit status
Makefile:95: recipe for target 'nballerinacc' failed

To solve this issue, I tried to install below dev and tools package -

https://debian.pkgs.org/sid/debian-main-amd64/llvm-11-dev_11.0.0-5+b1_amd64.deb.html
https://debian.pkgs.org/sid/debian-main-amd64/llvm-11-tools_11.0.0-5+b1_amd64.deb.html

I could not find -lLLVMTestingSupportlibrary in any one above. If we can get this library support, then we can put llvm installation directory in nballerina which avoids building of LLVM everytime we build nballerina compiler.

[ayonam]

Shubham, Do we have these libraries in the build directory at least?  I would guess so because otherwise our current build process won't work.  Just confirming. If that be so, then we should check on the LLVM dev community why these three libraries are not installed as part of the install process. Also, we should check why at all we need these three libraries.  My hunch is that we should have no dependency on these three libraries.  Ayonam Ray Director R. R. Logic Systems Private Limited #89, 1st Floor, V. M. Cosma Karthik Nagar, Marathahalli Outer Ring Road Bengaluru - 560 037, INDIA Mobile: +91-98456-67483 Web: https://www.rrlogic.co.in/ Fax: +91-80-6688-5757 E-mail: [email protected] On 14/12/20 15:49, shubhamnarlawar wrote: I tried below process to achieve above requirement - I have build and installed LLVM-11.0 and used only installation directory for building nballerina. There are 3 library/header support missing in llvm installation directory - /usr/bin/x86_64-linux-gnu-ld: cannot find -lLLVMTestingSupport /usr/bin/x86_64-linux-gnu-ld: cannot find -lgtest_main /usr/bin/x86_64-linux-gnu-ld: cannot find -lgtest collect2: error: ld returned 1 exit status Makefile:95: recipe for target 'nballerinacc' failed -------------------------------------------------------Build commads------------------------------------------------------------------------- cmake -DCMAKE_BUILD_TYPE="Release" -DCMAKE_INSTALL_PREFIX=/home/shubham/build/llvm/llvm_install -DBUILD_SHARED_LIBS=True -DLLVM_USE_SPLIT_DWARF=True -DLLVM_OPTIMIZED_TABLEGEN=True -DLLVM_BUILD_TESTS=True /home/shubham/llvm-11.0/llvm-project-11.0.0/llvm make install make install-llvm-headers make install-llvm-libraries — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/ballerina-platform/nballerina/discussions/27#discussioncomment-204380", "url": "https://github.com/ballerina-platform/nballerina/discussions/27#discussioncomment-204380", "name": "View Discussion" }, "description": "View this Discussion on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[shubhamnarlawar]

Yes. We have these libraries in build process. I will drop a mail to llvm community why they are not being installed. First, I will check whether we really need these libraries for build process of nballerina.

[shubhamnarlawar]

Upon removal of above 3 flags -lLLVMTestingSupport -lgtest & -lgtest_main -> ballerina build is successful with below config options for LLVM one time build -

cmake -DCMAKE_BUILD_TYPE="Release" \
-DCMAKE_INSTALL_PREFIX=/home/shubham/build/llvm/llvm_install \
/home/shubham/llvm-11.0/llvm-project-11.0.0/llvm

[ayonam]

I think you need to remove only the -DLLVM_BUILD_TESTS=True flag. That is what is causing the requirement of those Tests related three libraries. Removing that can remove that dependency also. Just try with removing only this one. The other two flags are good to have.

[shubhamnarlawar]

Upon removal of unnecessary flags like -lLLVMTestingSupport -lgtest & -lgtest_main from the nballerina Makefile - We are able to build nballerina compiler by using below package -

https://debian.pkgs.org/sid/debian-main-amd64/llvm-11-dev_11.0.0-5+b1_amd64.deb.html

This suggests that nballerina build process can be completed without building LLVM.

We tried on below machine -

Distributor ID: Ubuntu
Description: Ubuntu 18.04 LTS
Release: 18.04
Codename: bionic

[ayonam]

Shubham, We need the CI process to pick this release from the website you mentioned, install it on the build machine, run the nballerina build process, run the tests and then possibly uninstall it.  The Makefile needs to be modified to point variables like LLVM_INCLUDES_DIR, etc. to point to the installed directories. On 16/12/20 19:15, shubhamnarlawar wrote: Upon removal of unnecessary flags like -lLLVMTestingSupport -lgtest & -lgtest_main from the nballerina Makefile - We are able to build nballerina compiler by using below package - https://debian.pkgs.org/sid/debian-main-amd64/llvm-11-dev_11.0.0-5+b1_amd64.deb.html This suggests that nballerina build process can be completed without building LLVM. We tried on below machine - Distributor ID: Ubuntu Description: Ubuntu 18.04 LTS Release: 18.04 Codename: bionic` — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/ballerina-platform/nballerina/discussions/27#discussioncomment-214880", "url": "https://github.com/ballerina-platform/nballerina/discussions/27#discussioncomment-214880", "name": "View Discussion" }, "description": "View this Discussion on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[ruvi-d]

Pull Request #52 addresses this

[ruvi-d]

@ayonam : can you mark this discussion as "answered" ?

[ayonam]

Done.