diff --git a/README.md b/README.md index 25b8953..7057ccc 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Documentation for E2SAR adopters is contained in the [wiki](https://github.com/J ### Via cloning -All binary artifacts in this project are stored using Git LFS and you must [install git lfs](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage?platform=linux) in order to properly check out their contents. +Binary artifacts in this project are stored using Git LFS and you must [install git lfs](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage?platform=linux) in order to properly check out their contents. Clone the project as shown below, to include the [UDPLBd](https://github.com/esnet/udplbd) repo contents (needed for the protobuf definitions located in udplbd/pkg/pb) as well as wiki/ and docs/ (which are separate repos maintaining doxygen documentation and the wiki): ```bash @@ -24,7 +24,7 @@ $ cd udplbd $ git fetch && get switch develop ``` -Also a section below discusses the development Docker image which can be used for modifying, compiling and testing the code inside a running container. +Also a [section below](#Development-Docker) discusses the development Docker image which can be used for modifying, compiling and testing the code inside a running container. ## Building the code @@ -51,7 +51,7 @@ $ pip install protobuf ``` Continue using the venv when compiling and testing e2sar. PSA: you can get out of the venv by running `deactivate` from inside the venv. -The other two large dependencies are the C++ Boost library and gRPC library. For some common distributions we have pre-compiled installation trees stored [under this directory](binary_artifacts/). Note that you must have git-lfs installed to access them. In many cases they m ay need to be built from scratch using instructions provided below. +The other two large dependencies are the C++ Boost library and gRPC library. Specific versions are defined in the top-level [meson.build](./meson.build) file. #### Installing gRPC and Boost as packages @@ -125,17 +125,17 @@ The `live` test suite requires a running UDPLBd and the setting of EJFAT_URI mus If you desire a custom installation directory you can add `--prefix=/absolute/path/to/install/root`. If you have a custom location for pkg-config scripts, you can also add `-Dpkg_config_path=/path/to/pkg-config/scripts` to the setup command. -### Building on older systems (e.g. RHEL8) +#### Building on older systems (e.g. RHEL8) Due to a much older g++ compiler on those systems meson produces incorrect ninja.build files. After the `setup build` step execute the following command to correct the build file: `sed -i 's/-std=c++11//g' build/build.ninja`. -### Building a Docker version +## Building Dockers There are several Docker files in the root of the source tree. They build various versions of the system for improved portability -#### Control Plane and other tools +### Control Plane and other tools -To build this docker (Dockerfile.cli) use the following command: +To build [this docker](Dockerfile.cli) use the following command: ``` $ docker build -t cli -f Dockerfile.cli . ``` @@ -157,7 +157,7 @@ $ docker run --rm ibaldin/e2sar:latest lbadm --version -u "ejfats://udplbd@192.1 ``` (Notice that default docker network plumbing probably isn't appropriate for listening or sending packets using snifgen.py). -#### Development Docker +### Development Docker We provide another Docker image which allows development inside the running container. It includes all necessary dependencies for a given version of E2SAR. VSCode can be attached to it to ease the development. @@ -188,25 +188,24 @@ You can install the code after compilation by running `meson install -C build` ( To create a source distribution you can run `meson dist -C build --no-tests` (the `--no-tests` is needed because GRPC headers won't build properly when distribution is generated). -# Testing +## Testing -## C++ +### C++ E2SAR code comes with a set of tests under [test/](test/) folder. It relies on Boost unit-testing framework as well as meson testing capabilities. The easiest way is to execute `meson test` or `meson test --suite unit` or `meson test --suite live`. The latter requires an instance of UDPLBd running and `EJFAT_URI` environment variable to be set to point to it (e.g. `export EJFAT_URI="ejfats://udplbd@192.168.0.3:18347/"). There is a [Jupyter notebook](scripts/notebooks/EJFAT/LBCP-tester.ipynb) which runs all the tests on FABRIC testbed. -## Python +### Python The C++ unit tests `e2sar_uri_test` and `e2sar_reas_test` have been reproduced in Python Jupyter notebooks, which can be found at [scripts/notebooks/pybind11_examples/](scripts/notebooks/pybind11_examples/). The Python `e2sar_lbcp_live_test` is currently under development. -## Scapy +### Scapy Scapy scripts are provided for sniffing/validating and generating various kinds of UDP packets. See this [folder](scripts/scapy/) for details. Make sure Scapy is installed (`pip install scapy`) and for most tasks the scripts must be run as root (both for sending and sniffing). More details on the use can be found in the [wiki](https://github.com/JeffersonLab/E2SAR/wiki/Code-and-Binaries). - ## Dealing with SSL certificate validation UDPLBd implements gRPC over TLS to support channel privacy and server authentication. Only server-side certificate for UDPLBd is required - the code does not rely on SSL client-side authentication. For testing you can generate UDPLBd certificate as follows: @@ -235,9 +234,9 @@ If using a private CA, the `-o` option should be used to point to the CA certifi See [lbadm](bin/lbadm.cpp) code on how the two options are implemented and used. Programmatically it is possible to supply client-side certs to E2SAR, however UDPLBd does not validate them. See [LBManager constructor](src/e2sarCP.cpp) for how that can be done. -# Generating Documentation +## Generating Documentation -## Doxygen +### Doxygen A [Doxygen configuration file](Doxyfile) is provided with the distribution. It will update the documentation under docs/ which is a submodule - a separate public repo just for the documentation. Once updated, to make the documentation go live on [GitHub pages site](https://jeffersonlab.github.io/E2SAR-doc/annotated.html) you can do as follows from E2SAR root (this assumes you have installed Doxygen 1.12.0 or later): ```bash @@ -252,7 +251,7 @@ $ git commit -S -m "Link docs/ commit to this commit of E2SAR" $ git push origin ``` -## Wiki +### Wiki The [Wiki](https://github.com/JeffersonLab/E2SAR/wiki) is also a submodule of the project (but is a separate repository) under wiki/. You can update it by doing the following: ```bash @@ -268,7 +267,7 @@ $ git push origin ``` Note that wiki uses `master` as its main branch, while docs/ and main E2SAR repo use `main`. -# Related information +## Related information - [UDPLBd repo](https://github.com/esnet/udplbd) (aka Control Plane) - [ejfat-rs repo](https://github.com/esnet/ejfat-rs) (command-line tool for testing) diff --git a/binary_artifacts/README.md b/binary_artifacts/README.md index 4a259a4..c7cf794 100644 --- a/binary_artifacts/README.md +++ b/binary_artifacts/README.md @@ -1,9 +1,5 @@ # Overview -This directory contains binary artifacts needed for developing and/or testing E2SAR. For exaple: - -- gRPC library and its dependencies compiled for specific platform -- Boost library compiled for specific platform - +This directory contains binary artifacts needed for developing and/or testing E2SAR. All artifacts are stored using Git LFS (and you must [install git lfs](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage?platform=linux) in order to properly check out their contents). diff --git a/binary_artifacts/boost-v1.85.0-rhel8.tar.gz b/binary_artifacts/boost-v1.85.0-rhel8.tar.gz deleted file mode 100644 index 3b6b3c8..0000000 --- a/binary_artifacts/boost-v1.85.0-rhel8.tar.gz +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:94e0f3423c61ee1270ce1126ac52b03dd1882bbbdc040930149e1d8415a1cae0 -size 25470209 diff --git a/binary_artifacts/boost-v1.85.0-rocky8.tar.gz b/binary_artifacts/boost-v1.85.0-rocky8.tar.gz deleted file mode 100644 index 4e08ade..0000000 --- a/binary_artifacts/boost-v1.85.0-rocky8.tar.gz +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f5ee42efa566332fa62de3224396de27388930228a5ece311c653d208fd0c519 -size 25861052 diff --git a/binary_artifacts/boost-v1.85.0-ubuntu22.tar.gz b/binary_artifacts/boost-v1.85.0-ubuntu22.tar.gz deleted file mode 100644 index 7720ffb..0000000 --- a/binary_artifacts/boost-v1.85.0-ubuntu22.tar.gz +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:07125c3ffd03b88c59735458217e65b4697f2529cf7b4380184f48190ff7453e -size 26554922 diff --git a/binary_artifacts/grpc-v1.54.1-rhel8.tar.gz b/binary_artifacts/grpc-v1.54.1-rhel8.tar.gz deleted file mode 100644 index c7a7237..0000000 --- a/binary_artifacts/grpc-v1.54.1-rhel8.tar.gz +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9826d2e146d8a0125e94599b89b59f2b3d7372a612b4a289f74608119aec7623 -size 30459657 diff --git a/binary_artifacts/grpc-v1.54.1-rocky8.tar.gz b/binary_artifacts/grpc-v1.54.1-rocky8.tar.gz deleted file mode 100644 index b57a7f2..0000000 --- a/binary_artifacts/grpc-v1.54.1-rocky8.tar.gz +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:5da4af47c638954cacc8e7aecbb496103811845b2afa843c96378bccd7b58ecf -size 52511428 diff --git a/binary_artifacts/grpc-v1.54.1-ubuntu20.tar.gz b/binary_artifacts/grpc-v1.54.1-ubuntu20.tar.gz deleted file mode 100644 index 54d9c40..0000000 --- a/binary_artifacts/grpc-v1.54.1-ubuntu20.tar.gz +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f074e035e9a363e10e9b329e625ccdd34784495b5ca221bceb982fb8a6614ee6 -size 75956544 diff --git a/binary_artifacts/grpc-v1.54.1-ubuntu22.tar.gz b/binary_artifacts/grpc-v1.54.1-ubuntu22.tar.gz deleted file mode 100644 index 99f1370..0000000 --- a/binary_artifacts/grpc-v1.54.1-ubuntu22.tar.gz +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:fde8d7d2cb92430533631488740ad276903a50f17e960f1e7b7769580f02734f -size 107086098 diff --git a/scripts/notebooks/EJFAT/E2SAR-development-tester.ipynb b/scripts/notebooks/EJFAT/E2SAR-development-tester.ipynb index bf99eba..4b7f575 100644 --- a/scripts/notebooks/EJFAT/E2SAR-development-tester.ipynb +++ b/scripts/notebooks/EJFAT/E2SAR-development-tester.ipynb @@ -461,9 +461,11 @@ }, { "cell_type": "markdown", - "id": "1ed0ed9d-161e-4397-9bff-780dda436823", + "id": "12b93d76-1c92-4065-bced-b250420b1381", "metadata": {}, "source": [ + "## Building, Running Unit and Live Tests\n", + "\n", "On RedHat/Rocky derivatives the build fails. Likely has to do with the outdated gcc/g++ and stdlib. E2SAR/build/ninja.build must be modified - all mentions of `-std=c++11` must be removed from it.\n", "\n", "To avoid this after the build step (and the next one after) the below command invokes `sed` to remove mentions of `c++11` from build.ninja file. This is a hack, don't just walk away, **RUN**!\n", @@ -471,14 +473,6 @@ "Also note that the install directory is set to `$(HOME)/e2sar-install`. So if you run `meson install -C build` this is where the code will end up." ] }, - { - "cell_type": "markdown", - "id": "12b93d76-1c92-4065-bced-b250420b1381", - "metadata": {}, - "source": [ - "## Building, Running Unit and Live Tests" - ] - }, { "cell_type": "code", "execution_count": null, @@ -496,7 +490,7 @@ " f\"cd E2SAR; PATH=$HOME/.local/bin:/usr/local/bin:$PATH BOOST_ROOT=/usr/local/ LD_LIBRARY_PATH=/usr/local/lib/ meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig/:/usr/lib/lib64/pkgconfig/ --prefix {home_location}/e2sar-install build && sed -i 's/-std=c++11//g' build/build.ninja\",\n", " f\"cd E2SAR/build; PATH=$HOME/.local/bin:/usr/local/bin:$PATH LD_LIBRARY_PATH=/usr/local/lib/ meson compile -j 8\",\n", " f\"cd E2SAR/build; PATH=$HOME/.local/bin:/usr/local/bin:$PATH LD_LIBRARY_PATH=/usr/local/lib/ meson test {e2sar_test_suite} --suite unit --timeout 0\",\n", - "# f\"cd E2SAR/build; EJFAT_URI='ejfats://udplbd@{cpnode_addr}:18347/lb/1?data=127.0.0.1&sync=192.168.88.199:1234' PATH=$HOME/.local/bin:/usr/local/bin:$PATH LD_LIBRARY_PATH=/usr/local/lib/ meson test {e2sar_test_suite} --suite live --timeout 0\"\n", + " f\"cd E2SAR/build; EJFAT_URI='ejfats://udplbd@{cpnode_addr}:18347/lb/1?data=127.0.0.1&sync=192.168.88.199:1234' PATH=$HOME/.local/bin:/usr/local/bin:$PATH LD_LIBRARY_PATH=/usr/local/lib/ meson test {e2sar_test_suite} --suite live --timeout 0 -j 1\"\n", "]\n", " \n", "execute_commands([sender], commands)" @@ -528,7 +522,7 @@ " f\"cd E2SAR; BOOST_ROOT=/usr/local/ PATH=$HOME/.local/bin:/usr/local/bin:$PATH LD_LIBRARY_PATH=/usr/local/lib/ meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig/:/usr/lib/lib64/pkgconfig/ --prefix {home_location}/e2sar-install build --wipe && sed -i 's/-std=c++11//g' build/build.ninja\",\n", " f\"cd E2SAR/build; PATH=$HOME/.local/bin:/usr/local/bin:$PATH LD_LIBRARY_PATH=/usr/local/lib/ meson compile -j 8\",\n", " f\"cd E2SAR/build; PATH=$HOME/.local/bin:/usr/local/bin:$PATH LD_LIBRARY_PATH=/usr/local/lib/ meson test {e2sar_test_suite} --suite unit --timeout 0\",\n", - "# f\"cd E2SAR/build; EJFAT_URI='ejfats://udplbd@{cpnode_addr}:18347/lb/1?data=127.0.0.1&sync=192.168.88.199:1234' PATH=$HOME/.local/bin:/usr/local/bin:$PATH LD_LIBRARY_PATH=/usr/local/lib/ meson test {e2sar_test_suite} --suite live --timeout 0\"\n", + " f\"cd E2SAR/build; EJFAT_URI='ejfats://udplbd@{cpnode_addr}:18347/lb/1?data=127.0.0.1&sync=192.168.88.199:1234' PATH=$HOME/.local/bin:/usr/local/bin:$PATH LD_LIBRARY_PATH=/usr/local/lib/ meson test {e2sar_test_suite} --suite live --timeout 0 -j 1\"\n", "]\n", " \n", "execute_commands([sender, recver], commands)"