Skip to content
Flow-IPC pipeline

*SAN suppressions made more flexible to control. TSAN suppressions far reduced for most tests. TSAN runs and passes transport_test exercise mode (was disabled). Suppression fils are less copy/pasty/redundant to each other. #490

Sign in to view logs

*SAN suppressions made more flexible to control. TSAN suppressions far reduced for most tests. TSAN runs and passes transport_test exercise mode (was disabled). Suppression fils are less copy/pasty/redundant to each other.

*SAN suppressions made more flexible to control. TSAN suppressions far reduced for most tests. TSAN runs and passes transport_test exercise mode (was disabled). Suppression fils are less copy/pasty/redundant to each other. #490

Workflow file for this run

.github/workflows/main.yml at 4338024
name: Flow-IPC pipeline
on:
push:
branches:
- main
pull_request:
branches:
- main
# To create the button that runs a workflow manually (has to be in `main` at least):
workflow_dispatch:
jobs:
doc:
strategy:
fail-fast: false
matrix:
compiler:
- id: clang-15
name: clang
version: 15
c-path: /usr/bin/clang-15
cpp-path: /usr/bin/clang++-15
build-cfg:
- id: release
conan-profile-build-type: Release
conan-preset: release
# Not using ubuntu-latest, so as to avoid surprises with OS upgrades and such.
runs-on: ubuntu-22.04
name: |
doc-${{ matrix.compiler.id }}-${{ matrix.build-cfg.id }}
steps:
- name: Update available software list for apt-get
run: sudo apt-get update
- name: Checkout `ipc` repository and submodules (like `flow`)
uses: actions/checkout@v4
with:
submodules: true
- name: Install Flow-IPC dependencies (like Graphviz) with apt-get
run: |
sudo apt-get install -y graphviz
- name: Install the latest version of Conan which is less than 2
run: |
pip install "conan<2"
- name: Create Conan profile
run: |
cat <<'EOF' > conan_profile
[settings]
compiler = ${{ matrix.compiler.name }}
compiler.version = ${{ matrix.compiler.version }}
compiler.cppstd = 17
compiler.libcxx = libstdc++11
arch = x86_64
os = Linux
build_type = ${{ matrix.build-cfg.conan-profile-build-type }}
[conf]
tools.build:compiler_executables = {"c": "${{ matrix.compiler.c-path }}", "cpp": "${{ matrix.compiler.cpp-path }}"}
[buildenv]
CC = ${{ matrix.compiler.c-path }}
CXX = ${{ matrix.compiler.cpp-path }}
[options]
ipc:doc = True
ipc:build = False
EOF
- name: Install Flow-IPC dependencies (like Doxygen) with Conan using the profile
run: |
conan install \
. \
--profile:build conan_profile \
--profile:host conan_profile \
--build missing
- name: Generate code documentation using Conan and Doxygen
run: |
conan build .
- name: Create documentation tarball (full, API-only, landing page)
run: |
cd ${{ github.workspace }}/doc/ipc_doc
${{ github.workspace }}/tools/doc/stage_generated_docs.sh ${{ github.workspace }}/build/${{ matrix.build-cfg.conan-profile-build-type }}
# Save runner space.
rm -rf generated
- name: Upload documentation tarball
uses: actions/upload-artifact@v3
with:
name: ipc-doc-${{ matrix.compiler.id }}-${{ matrix.build-cfg.id }}
path: |
${{ github.workspace }}/doc/ipc_doc.tgz
build:
strategy:
fail-fast: false
matrix:
compiler:
- id: gcc-9
name: gcc
version: 9
c-path: /usr/bin/gcc-9
cpp-path: /usr/bin/g++-9
- id: gcc-10
name: gcc
version: 10
c-path: /usr/bin/gcc-10
cpp-path: /usr/bin/g++-10
- id: gcc-11
name: gcc
version: 11
c-path: /usr/bin/gcc-11
cpp-path: /usr/bin/g++-11
- id: gcc-13
name: gcc
version: 13
c-path: /usr/bin/gcc-13
cpp-path: /usr/bin/g++-13
- id: clang-13
name: clang
version: 13
c-path: /usr/bin/clang-13
cpp-path: /usr/bin/clang++-13
- id: clang-15
name: clang
version: 15
c-path: /usr/bin/clang-15
cpp-path: /usr/bin/clang++-15
- id: clang-16
name: clang
version: 16
c-path: /usr/bin/clang-16
cpp-path: /usr/bin/clang++-16
install: True
- id: clang-17
name: clang
version: 17
c-path: /usr/bin/clang-17
cpp-path: /usr/bin/clang++-17
install: True
build-test-cfg:
- id: debug
conan-profile-build-type: Debug
conan-profile-jemalloc-build-type: Debug
- id: release
conan-profile-build-type: Release
conan-profile-jemalloc-build-type: Release
- id: relwithdebinfo
conan-profile-build-type: RelWithDebInfo
conan-profile-jemalloc-build-type: Release
- id: minsizerel
conan-profile-build-type: MinSizeRel
conan-profile-jemalloc-build-type: Release
- id: relwithdebinfo-asan
conan-profile-build-type: RelWithDebInfo
conan-profile-jemalloc-build-type: Release
conan-profile-custom-conf: |
# no-omit-frame-pointer recommended in (A|UB|M)SAN docs for nice stack traces.
tools.build:cflags = ["-fsanitize=address", "-fno-omit-frame-pointer"]
tools.build:cxxflags = ["-fsanitize=address", "-fno-omit-frame-pointer"]
tools.build:sharedlinkflags = ["-fsanitize=address"]
tools.build:exelinkflags = ["-fsanitize=address"]
conan-profile-custom-settings: |
compiler.sanitizer = address
conan-profile-custom-buildenv: |
CXXFLAGS = -fsanitize=address -fno-omit-frame-pointer
CFLAGS = -fsanitize=address -fno-omit-frame-pointer
LDFLAGS = -fsanitize=address
conan-custom-settings-defs: | # Could we not copy/paste these 4x?
data['compiler']['gcc']['sanitizer'] = ['None', 'address', 'thread', 'memory', 'undefined']
data['compiler']['clang']['sanitizer'] = ['None', 'address', 'thread', 'memory', 'undefined']
sanitizer-name: asan # Used as internal enum of sorts + name of sanitizer-related dirs.
no-lto: true # At least ASAN with clang + LTO => cryptic link error. No need for LTO when *SAN.
- id: relwithdebinfo-ubsan
conan-profile-build-type: RelWithDebInfo
conan-profile-jemalloc-build-type: Release
conan-profile-custom-conf: |
tools.build:cflags = ["-fsanitize=undefined", "-fno-omit-frame-pointer"]
tools.build:cxxflags = ["-fsanitize=undefined", "-fno-omit-frame-pointer"]
tools.build:sharedlinkflags = ["-fsanitize=undefined"]
tools.build:exelinkflags = ["-fsanitize=undefined"]
conan-profile-custom-settings: |
compiler.sanitizer = undefined
conan-profile-custom-buildenv: |
CXXFLAGS = -fsanitize=undefined -fno-omit-frame-pointer
CFLAGS = -fsanitize=undefined -fno-omit-frame-pointer
LDFLAGS = -fsanitize=undefined
conan-custom-settings-defs: |
data['compiler']['gcc']['sanitizer'] = ['None', 'address', 'thread', 'memory', 'undefined']
data['compiler']['clang']['sanitizer'] = ['None', 'address', 'thread', 'memory', 'undefined']
sanitizer-name: ubsan
no-lto: true # While UBSAN might work with LTO, I do not want the aggravation/entropy. Turn it off.
- id: relwithdebinfo-tsan
conan-profile-build-type: RelWithDebInfo
conan-profile-jemalloc-build-type: Release
conan-profile-custom-conf: |
tools.build:cflags = ["-fsanitize=thread"]
tools.build:cxxflags = ["-fsanitize=thread"]
tools.build:sharedlinkflags = ["-fsanitize=thread"]
tools.build:exelinkflags = ["-fsanitize=thread"]
conan-profile-custom-settings: |
compiler.sanitizer = thread
conan-profile-custom-buildenv: |
CXXFLAGS = -fsanitize=thread
CFLAGS = -fsanitize=thread
LDFLAGS = -fsanitize=thread
conan-custom-settings-defs: |
data['compiler']['gcc']['sanitizer'] = ['None', 'address', 'thread', 'memory', 'undefined']
data['compiler']['clang']['sanitizer'] = ['None', 'address', 'thread', 'memory', 'undefined']
unit-tests-params: | # As of this writing these trigger assert+bugs in TSAN runtime. TODO: Revisit.
--gtest_filter=-Shm_session_test.In_process_array:\
Shm_session_test.In_process_vector_offset_ptr:\
Shm_session_test.In_process_string_offset_ptr:\
Shm_session_test.In_process_list_offset_ptr:\
Shm_session_test.Multisession_in_process:\
Shm_session_test.Allocation_performance_five:\
Jemalloc_shm_pool_collection_test.Interface:\
Jemalloc_shm_pool_collection_test.Multiprocess:\
Jemalloc_shm_pool_collection_test.Owner_shm_pool_removal:\
Jemalloc_shm_pool_collection_test.Multithread_load:\
Borrower_shm_pool_collection_test.Interface:\
Borrower_shm_pool_collection_test.Multiprocess:\
Shm_pool_collection_test.Interface:\
Shm_pool_collection_test.Multiprocess:\
Shm_session_data_test.Multithread_object_database
skip-transport-tests-ex: true # TODO: Explain why briefly (even though there's prob a ticket).
sanitizer-name: tsan
no-lto: false # See the other *SAN; but for now I heed jkontrik's words to keep consistent with non-*SAN.
- id: relwithdebinfo-msan
conan-profile-build-type: RelWithDebInfo
conan-profile-jemalloc-build-type: Release
conan-profile-custom-conf: |
tools.build:cflags = ["-fsanitize=memory", "-fno-omit-frame-pointer", "-fsanitize-ignorelist=/tmp/msan_ignore_list.cfg"]
tools.build:cxxflags = ["-fsanitize=memory", "-fno-omit-frame-pointer", "-fsanitize-ignorelist=/tmp/msan_ignore_list.cfg"]
tools.build:sharedlinkflags = ["-fsanitize=memory"]
tools.build:exelinkflags = ["-fsanitize=memory"]
conan-profile-custom-settings: |
compiler.sanitizer = memory
conan-profile-custom-buildenv: |
CXXFLAGS = -fsanitize=memory -fno-omit-frame-pointer -fsanitize-ignorelist=/tmp/msan_ignore_list.cfg
CFLAGS = -fsanitize=memory -fno-omit-frame-pointer -fsanitize-ignorelist=/tmp/msan_ignore_list.cfg
LDFLAGS = -fsanitize=memory
conan-custom-settings-defs: |
data['compiler']['gcc']['sanitizer'] = ['None', 'address', 'thread', 'memory', 'undefined']
data['compiler']['clang']['sanitizer'] = ['None', 'address', 'thread', 'memory', 'undefined']
sanitizer-name: msan
no-lto: true # While MSAN might work with LTO, I do not want the aggravation/entropy. Turn it off.
# We concentrate on clang sanitizers; they are newer/nicer; also MSAN is clang-only. So gcc ones excluded.
# Attention! Excluding some sanitizer job(s) (with these reasons):
# - MSAN: MSAN protects against reads of ununitialized memory; it is clang-only (not gcc), unlike the other
# *SAN. Its mission overlaps at least partially with UBSAN's; for example for sure there were a couple of
# uninitialized reads in test code which UBSAN caught. Its current state -- if not excluded -- is as
# follows: 1, due to (as of this writing) building dependencies, including the capnp compiler binary used
# during our build process, with the same compiler build config as the real code, ignore-list entires had
# to be added to get past these problems. 2, before main() in *all* our demos/tests Boost was doing some
# global init which MSAN did not like and hence aborted before main(); these are now ignored as well.
# 3, this got us into main() at least, but immediately cryptic aborts started, seemingly again originating
# in Boost (but requires detailed investigation to really understand). At this point I (ygoldfel)
# disabled MSAN and filed a ticket. The overall status: MSAN is said to be useful, even with UBSAN active,
# but all resources including official docs indicate that it is a high-maintenance tool:
# *All* linked code, including libc and libstdc++/libc++ (the former in our case as of this writing),
# must be instrumented to avoid cryptic false positives. The good news is we do build other things
# instrumented (Boost libs, jemalloc, capnp/kj, gtest); but not libstdc++ (which official docs recommend);
# this can be done but requires more work (I would suggest switching to libc++ in that case from the
# start, as the clang people made it and themselves build it instrumented for their testing).
# So the bottom line: MSAN is a tough cookie and requires more work before enabling. In the meantime we
# have many layers of protection, including ASAN and UBSAN and lots of unit and integration tests.
# So this status quo is pretty good. TODO: Do the work/get MSAN functional/useful; un-exclude it then.
# - *SAN with gcc: We concentrate on clang sanitizers; they are newer/nicer; having to worry about differences
# between them is an excessive burden. So excluding gcc ASAN/UBSAN/TSAN.
# - TODO: Consider reducing to the newest clang. Generally they just keep improving; the chances of
# something being uncaught (incorrectly) with a later version are slim. Not impossible though. Look
# into it.
exclude:
- build-test-cfg: { id: relwithdebinfo-msan }
- compiler: { id: gcc-9 }
build-test-cfg: { id: relwithdebinfo-asan }
- compiler: { id: gcc-10 }
build-test-cfg: { id: relwithdebinfo-asan }
- compiler: { id: gcc-11 }
build-test-cfg: { id: relwithdebinfo-asan }
- compiler: { id: gcc-13 }
build-test-cfg: { id: relwithdebinfo-asan }
- compiler: { id: gcc-9 }
build-test-cfg: { id: relwithdebinfo-ubsan }
- compiler: { id: gcc-10 }
build-test-cfg: { id: relwithdebinfo-ubsan }
- compiler: { id: gcc-11 }
build-test-cfg: { id: relwithdebinfo-ubsan }
- compiler: { id: gcc-13 }
build-test-cfg: { id: relwithdebinfo-ubsan }
- compiler: { id: gcc-9 }
build-test-cfg: { id: relwithdebinfo-tsan }
- compiler: { id: gcc-10 }
build-test-cfg: { id: relwithdebinfo-tsan }
- compiler: { id: gcc-11 }
build-test-cfg: { id: relwithdebinfo-tsan }
- compiler: { id: gcc-13 }
build-test-cfg: { id: relwithdebinfo-tsan }
# Not using ubuntu-latest, so as to avoid surprises with OS upgrades and such.
runs-on: ubuntu-22.04
name: |
build-${{ matrix.compiler.id }}-${{ matrix.build-test-cfg.id }}
env:
build-dir: ${{ github.workspace }}/build/${{ matrix.build-test-cfg.conan-profile-build-type }}
install-root-dir: ${{ github.workspace }}/install/${{ matrix.build-test-cfg.conan-profile-build-type }}
# (Unfortunately cannot refer to earlier-assigned `env.` entries within subsequent ones.)
install-dir: ${{ github.workspace }}/install/${{ matrix.build-test-cfg.conan-profile-build-type }}/usr/local
# Target file, as read by sanitized executable being invoked.
san-suppress-cfg-file: ${{ github.workspace }}/install/${{ matrix.build-test-cfg.conan-profile-build-type }}/usr/local/bin/san_suppressions.cfg
# Relative path (including file name) to suppressions file in a given context (which is given as dir name off
# which this path works). Possible contexts as of this writing: ${{ github.workspace }}/<module>/src
# (suppressions endemic to lib<module>); ${{ github.workspace }}/.../<test's source code dir>
# (suppressions endemic to that test specifically). This suppressions file holds the compiler-version-independent
# entries. (Malformed and ignored if matrix.build-test-cfg is not `*san`, meaning not sanitizer-enabled.)
san-suppress-cfg-in-file1: sanitize/${{ matrix.build-test-cfg.sanitizer-name }}/suppressions_${{ matrix.compiler.name }}.cfg
# Same but contains compiler version-specific entries (on top of those in san-suppress-cfg-file).
san-suppress-cfg-in-file2: sanitize/${{ matrix.build-test-cfg.sanitizer-name }}/suppressions_${{ matrix.compiler.name }}_${{ matrix.compiler.version }}.cfg
# Run-time controls for various sanitizers. Invoke before running test but *after* assembling suppressions
# file ${{ env.san-suppress-cfg-file}} if any (clear it if needed, as it might be left-over from a previous
# test). The proper technique is: 1, which of the suppression contexts (see above) are relevant?
# (Safest is to specify all contexts; as you'll see just below, it's fine if there are no files in a given
# context. However it would make code tedious to specify that way everywhere; so it's fine to skip contexts where
# we know that these days there are no suppresisons.) Let the contexts' dirs be $DIR_A, $DIR_B, .... Then:
# 2, `{ cat $DIR_A/${{ env.san-suppress-cfg-in-file1 }} $DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
# $DIR_B/${{ env.san-suppress-cfg-in-file1 }} $DIR_B/${{ env.san-suppress-cfg-in-file2 }} \
# ... \
# > ${{ env.san-suppress-cfg-file }} 2> /dev/null; } || true`
# 3, {{ env.setup-tests-env }}.
#
# Notes about items in *SAN_OPTIONS:
#
# Unclear if we need disable_coredump=0; it might be a gcc thing due to historic
# issues with core size with ASAN; it is not documented in the clang *SAN docs for any
# sanitizers; but it is accepted for ASAN and TSAN, seemingly, and maybe is at worst
# harmless with out compiler versions. TODO: Maybe revisit.
#
# print_stacktrace=1 for UBSAN is recommended by documentation for nice stack traces
# (also that is why we apply sanitizers to RelWithDebInfo; DebInfo part for the nice
# stack traces/etc.; Rel part to reduce the considerable slowdown from some of the
# sanitizers).
#
# second_deadlock_stack=1 for TSAN: TODO: Explain. Don't see it in clang TSAN docs.
#
# Caution! Setting multiple *SAN_OPTIONS (while invoking only 1 sanitizer, as we do)
# seems like it would be harmless and avoid the `if/elif`s... but is actually a
# nightmare; there is some interaction in clang (at least 13-17) which causes the
# wrong sanitizer reading suppressions from another one => suppression parse error =>
# none of the demos/tests get anywhere at all. So set just the right one!
setup-tests-env: |
if [ "${{ matrix.build-test-cfg.sanitizer-name }}" = asan ]; then
export ASAN_OPTIONS='disable_coredump=0'
echo "ASAN_OPTIONS = [$ASAN_OPTIONS]."
elif [ "${{ matrix.build-test-cfg.sanitizer-name }}" = ubsan ]; then
export SAN_SUPP=1
export SAN_SUPP_CFG=${{ github.workspace }}/install/${{ matrix.build-test-cfg.conan-profile-build-type }}/usr/local/bin/san_suppressions.cfg
export UBSAN_OPTIONS="disable_coredump=0 print_stacktrace=1 suppressions=$SAN_SUPP_CFG"
echo "UBSAN_OPTIONS = [$UBSAN_OPTIONS]."
elif [ "${{ matrix.build-test-cfg.sanitizer-name }}" = tsan ]; then
export SAN_SUPP=1
export SAN_SUPP_CFG=${{ github.workspace }}/install/${{ matrix.build-test-cfg.conan-profile-build-type }}/usr/local/bin/san_suppressions.cfg
export TSAN_OPTIONS="disable_coredump=0 second_deadlock_stack=1 suppressions=$SAN_SUPP_CFG"
echo "TSAN_OPTIONS = [$TSAN_OPTIONS]."
fi
if [ "$SAN_SUPP" != '' ]; then
echo 'Sanitizer [${{ matrix.build-test-cfg.sanitizer-name }}] suppressions cfg contents:'
echo '[[[ file--'
cat $SAN_SUPP_CFG
echo '--file ]]]'
fi
# TODO: Ideally would go, like other things, under github.workspace somewhere (maybe build/),
# but that var isn't available in `strategy` up above, where we specify the compiler option
# pointing to this file. Surely we could cook something up; but meanwhile /tmp works fine.
msan-ignore-list-cfg-file: /tmp/msan_ignore_list.cfg
steps:
- name: Update available software list for apt-get
run: |
lsb_release -a
sudo apt-get update
- name: Checkout `ipc` repository and submodules (`flow`, `ipc_*`)
uses: actions/checkout@v4
with:
submodules: true
- name: Install clang compiler
if: |
matrix.compiler.install && (matrix.compiler.name == 'clang')
run: |
wget https://apt.llvm.org/llvm.sh
chmod u+x llvm.sh
sudo ./llvm.sh ${{ matrix.compiler.version }}
- name: Install gcc compiler
if: |
matrix.compiler.install && (matrix.compiler.name == 'gcc')
run: |
sudo apt-get install -y software-properties-common
sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
sudo apt-get update
sudo apt-get install -y gcc-${{ matrix.compiler.version }} g++-${{ matrix.compiler.version }}
- name: Install the latest version of Conan which is less than 2
run: |
pip install "conan<2"
- name: Add custom settings for Conan packages
if: |
matrix.build-test-cfg.conan-custom-settings-defs
run: |
conan config init
pip install PyYAML
CONAN_SETTINGS_PATH=$(conan config home)/settings.yml
python -c "
import yaml
with open('$CONAN_SETTINGS_PATH', 'r') as file:
data = yaml.safe_load(file)
${{ matrix.build-test-cfg.conan-custom-settings-defs }}
with open('$CONAN_SETTINGS_PATH', 'w') as file:
yaml.dump(data, file)
"
- name: Create Conan profile
run: |
cat <<'EOF' > conan_profile
[settings]
compiler = ${{ matrix.compiler.name }}
compiler.version = ${{ matrix.compiler.version }}
compiler.cppstd = 17
compiler.libcxx = libstdc++11
arch = x86_64
os = Linux
jemalloc:build_type = ${{ matrix.build-test-cfg.conan-profile-jemalloc-build-type }}
build_type = ${{ matrix.build-test-cfg.conan-profile-build-type }}
${{ matrix.build-test-cfg.conan-profile-custom-settings }}
[conf]
tools.env.virtualenv:auto_use = True
tools.build:compiler_executables = {"c": "${{ matrix.compiler.c-path }}", "cpp": "${{ matrix.compiler.cpp-path }}"}
${{ matrix.build-test-cfg.conan-profile-custom-conf }}
[buildenv]
CC = ${{ matrix.compiler.c-path }}
CXX = ${{ matrix.compiler.cpp-path }}
${{ matrix.build-test-cfg.conan-profile-custom-buildenv }}
[options]
flow:doc = False
flow:build = True
ipc:doc = False
ipc:build = True
EOF
if [ "${{ matrix.build-test-cfg.no-lto }}" != '' ]; then
echo 'ipc:build_no_lto = True' >> conan_profile
fi
# We need to prepare a sanitizer ignore-list in MSAN mode. Background for this is subtle and annoying:
# As it stands, whatever matrix compiler/build-type is chosen applies not just to our code (correct)
# and 3rd party libraries we link like lib{boost_*|capnp|kj|jemalloc} (semi-optional but good) but also
# unfortunately any items built from source during "Install Flow-IPC dependencies" step that we then
# use during during the build step for our own code subsequently. At a minimum this will slow down
# such programs. (For the time being we accept this as not-so-bad; to target this config at some things
# but not others is hard/a ticket.) In particular, though, the capnp compiler binary is built this way;
# and our "Build targets" step uses it to build key things (namely convert .capnp schemas into .c++
# and .h sources which are then themselves compiled/used in compilation). In the case of MSAN, this
# version of capnp compiler happens to trigger several MSAN failures (presumably they are not a true
# problem, and it's not our job really to sanitize capnp -- though we can file tickets for them and/or
# issue PRs; but I digress). So in MSAN mode our build step fails, when capnp compiler itself aborts
# with MSAN failures. One approach is to not apply MSAN to capnp compiler (no-go for now; it's a ticket
# as mentioned); another is to uses Conan to patch capnp package for them (not a bad idea; ticket filed);
# and lastly we can put the specific failures on the ignore-list for MSAN. For now we do that; while
# unpleasant it does get the job done. TODO: Revisit/resolve tickets/improve (see above).
- name: Prepare MSAN sanitizer compile-time config file(s)
if: |
(!cancelled()) && (matrix.build-test-cfg.sanitizer-name == 'msan')
run: |
cat <<'EOF' > $${ env.msan-ignore-list-cfg-file }}
[memory]
# Warning: In clang-18 there are breaking changes in how globs/regexes are interpreted. See docs.
# Currently assuming clang-17 or lower.
#
# capnp compiler MSAN failures suppressed:
src:*/kj/filesystem-disk-unix.*
src:*/bits/stl_tree.h
EOF
# Append to that the suppressed items from the source tree. These apply to the real code being
# built below, so just stylistically it is nicer to keep them there along with other
# sanitizers' suppressions. (Note, though, that MSAN does not have a run-time suppression
# system; only these ignore-lists. The others do also have ignore-lists though.
# The format is totally different between the 2 types of suppression.)
# Our MSAN support is budding compared to UBSAN/ASAN/TSAN; so just specify the one ingore-list file
# we have now. TODO: If/when MSAN support gets filled out like the others', then use a context system
# a-la env.setup-tests-env.
cat ${{ github.workspace }}/flow/src/sanitize/msan/ignore_list_${{ matrix.compiler.name }}.cfg \
>> $${ env.msan-ignore-list-cfg-file }}
echo "The combined MSAN ignore-list config file follows:"
cat $${ env.msan-ignore-list-cfg-file }}
- name: Install Flow-IPC dependencies with Conan using the profile
run: |
conan editable add flow flow/1.0
conan install \
. \
--profile:build conan_profile \
--profile:host conan_profile \
--build missing
- name: Build targets (libraries, demos/tests) with Conan
run: |
conan build .
- name: Install targets with Makefile
run: |
make install \
--directory ${{ env.build-dir }} DESTDIR=${{ env.install-root-dir }}
# Save runner space: blow away build dir after install.
rm -rf ${{ env.build-dir }}
# From now on use !cancelled() to try to run any test/demo that exists regardless
# of preceding failures if any. Same-ish (always()) with the log-upload at the end.
# Worst-case they'll all fail in super-basic ways and immediately; no extra harm done.
# From now on save logs in install-dir/bin/logs. They will be tarred up and uploaded
# as artifacts at the end. For those tests below that produce separate logs as files
# to begin-with, this is a no-brainer. For those (as of this writing the main one
# is unit_test; but also the link tests produce short logs too) who use stdout/stderr
# the reason to do this is 2-fold.
# - UBSAN in particular produces non-fatal error output about problems it detects.
# We need to analyze them after the fact and fail a step below in that case to
# alert developers who would then fix such new problems.
# - One can also build UBSAN-mode things with -fno-sanitize-recover which would
# cause abnormal program exit on *first* error. We do not do this for these
# reasons:
# - It is more convenient to let it continue and thus show all problems in one
# shot.
# - There is a side problem: At this time compile settings, including
# in UBSAN case `-fsanitize=undefined -fno-sanitize-recover`, are applied not
# just to our code or 3rd party libraries but also any other stuff built
# (due to setting C[XX]FLAGS in Conan profile). Targeting just the exact
# stuff we want with those is hard and a separate project/ticket.
# In the meantime -fno-sanitize-recover causes completely unrealted program
# halts during the very-early step, when building dependencies including
# capnp; some autotools configure.sh fails crazily, and nothing can work
# from that point on due to dependencies-install step failing. So at this
# time -fno-sanitize-recover is a no-go; it affects too much unrelated stuff.
# - It is more consistent/convenient to get all the bulky logs in one place as an
# artifact, rather than some in the pipeline output, others in the artifacts.
# (This preference is subjective, yes.)
# Here, as in all other tests below, we assemble a suppressions file in case this is a sanitized
# run; and we follow the procedure explained near setup-tests-env definition. To reiterate: to avoid
# tedium, but at the cost of mantainability of this file (meaning if a suppressions context is added then
# a few lines would need to be added here), we only list those contexts where *any* sanitizer has
# *any* suppression; otherwise we skip it for brevity. `find . -name 'suppressions*.cfg` is pretty useful
# to determine their presence in addition to whether the test itself has its specific suppressions of any kind.
- name: Run link test [`ipc_core` - Flow-IPC Core]
if: |
!cancelled()
run: |
cd ${{ env.install-dir }}/bin
mkdir -p logs/ipc_core_link_test
SUPP_DIR_A=${{ github.workspace }}/flow/src
{ cat $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } || true
${{ env.setup-tests-env }}
./ipc_core_link_test.exec > logs/ipc_core_link_test/console.log 2>&1
- name: Run link test [`ipc_transport_structured` - Flow-IPC Structured Transport]
if: |
!cancelled()
run: |
cd ${{ env.install-dir }}/bin
mkdir -p logs/ipc_transport_structured_link_test
SUPP_DIR_A=${{ github.workspace }}/flow/src
{ cat $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } || true
${{ env.setup-tests-env }}
./ipc_transport_structured_link_test.exec > logs/ipc_transport_structured_link_test/console.log 2>&1
- name: Run link test [`ipc_session` - Flow-IPC Sessions]
if: |
!cancelled()
run: |
cd ${{ env.install-dir }}/bin
mkdir -p logs/ipc_session_link_test
SUPP_DIR_A=${{ github.workspace }}/flow/src
{ cat $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } || true
${{ env.setup-tests-env }}
./ipc_session_link_test_srv.exec > logs/ipc_session_link_test/srv.console.log 2>&1 &
sleep 1
./ipc_session_link_test_cli.exec > logs/ipc_session_link_test/cli.console.log 2>&1
- name: Run link test [`ipc_shm` - Flow-IPC Shared Memory]
if: |
!cancelled()
run: |
cd ${{ env.install-dir }}/bin
mkdir -p logs/ipc_shm_link_test
SUPP_DIR_A=${{ github.workspace }}/flow/src
{ cat $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } || true
${{ env.setup-tests-env }}
./ipc_shm_link_test_srv.exec > logs/ipc_shm_link_test/srv.console.log 2>&1 &
sleep 1
./ipc_shm_link_test_cli.exec > logs/ipc_shm_link_test/cli.console.log 2>&1
- name: Run link test [`ipc_shm_arena_lend` - Flow-IPC SHM-jemalloc]
if: |
!cancelled()
run: |
cd ${{ env.install-dir }}/bin
mkdir -p logs/ipc_shm_arena_lend_link_test
SUPP_DIR_A=${{ github.workspace }}/flow/src
# As of this writing there are certain harmless UBSAN triggers within jemalloc, and ipc_shm_arena_lend
# is the thing that depends on jemalloc, and of the `link_test`s this is the only one that links
# ipc_shm_arena_lend (directly or otherwise). Hence compared to the other `link_test`s this adds the
# ipc_shm_arena_lend suppression context.
SUPP_DIR_B=${{ github.workspace }}/ipc_shm_arena_lend/src
{ cat $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
$SUPP_DIR_B/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_B/${{ env.san-suppress-cfg-in-file2 }} \
> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } || true
${{ env.setup-tests-env }}
./ipc_shm_arena_lend_link_test_srv.exec > logs/ipc_shm_arena_lend_link_test/srv.console.log 2>&1 &
sleep 1
./ipc_shm_arena_lend_link_test_cli.exec > logs/ipc_shm_arena_lend_link_test/cli.console.log 2>&1
- name: Run unit tests
if: |
!cancelled()
run: |
cd ${{ env.install-dir }}/bin
# Some newline issues with the possible additional args; so need to make a wrapper script
# and then redirect, as desired, its output.
cat <<'EOF' > ${{ env.install-dir }}/bin/run_unit_test.sh
./libipc_unit_test.exec ${{ matrix.build-test-cfg.unit-tests-params }}
EOF
mkdir -p logs/libipc_unit_test
SUPP_DIR_A=${{ github.workspace }}/flow/src
SUPP_DIR_B=${{ github.workspace }}/ipc_shm_arena_lend/src
# As of this writing there are TSAN suppressions for this test specifically. TODO: Revist them, and then this.
SUPP_DIR_OWN=${{ github.workspace }}/test/suite/unit_test
{ cat $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
$SUPP_DIR_B/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_B/${{ env.san-suppress-cfg-in-file2 }} \
$SUPP_DIR_OWN/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_OWN/${{ env.san-suppress-cfg-in-file2 }} \
> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } || true
${{ env.setup-tests-env }}
/usr/bin/bash -e ${{ env.install-dir }}/bin/run_unit_test.sh \
> logs/libipc_unit_test/console.log 2>&1
# For tests below where on failure it can be very helpful to look at higher-verbosity logs,
# any failing step is repeated with logging hiked up. Note that formally (and practically)
# higher-than-INFO verbosity is allowed to affect performance and thus is *not* right for
# the main integration test run. However on failure all bets are off, and we just want the
# info we can get. (Also higher-verbosity runs can take significantly longer; no one wants that.)
#
# As of this writing unit_test already runs with high verbosity, so we do not re-run on failure
# like these. The `link_test`s above are very basic tests, essentially there to ensure all built okay
# and run a sanity-check of a core feature of each of ours libs; so no re-run on failure there either.
# This follows the instructions in bin/transport_test/README.txt.
- name: Prepare run script for [transport_test - Scripted mode] variations below
if: |
(!cancelled()) && (!matrix.build-test-cfg.skip-transport-tests-sc)
run: |
cat <<'EOF' > ${{ env.install-dir }}/bin/run_transport_test_sc.sh
echo "Log level: [$1]."
cd ${{ env.install-dir }}/bin/transport_test
OUT_DIR_NAME=log_level_$1
OUT_DIR=../logs/transport_test/scripted/$OUT_DIR_NAME
mkdir -p $OUT_DIR
SUPP_DIR_A=${{ github.workspace }}/flow/src
{ cat $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } || true
${{ env.setup-tests-env }}
./transport_test.exec scripted $OUT_DIR/srv.log info $1 \
< srv-script.txt > $OUT_DIR/srv.console.log 2>&1 &
SRV_PID=$!
sleep 1
./transport_test.exec scripted $OUT_DIR/cli.log info $1 \
< cli-script.txt > $OUT_DIR/cli.console.log 2>&1 &
CLI_PID=$!
if wait $SRV_PID; then SRV_EC=0; else SRV_EC=$?; fi
echo "Server finished with code [$SRV_EC]."
if wait $CLI_PID; then CLI_EC=0; else CLI_EC=$?; fi
echo "Client finished with code [$CLI_EC]."
[ $SRV_EC -eq 0 ] && [ $CLI_EC -eq 0 ]
EOF
- name: Run integration test [transport_test - Scripted mode]
id: transport_test_scripted
if: |
(!cancelled()) && (!matrix.build-test-cfg.skip-transport-tests-sc)
run: |
/usr/bin/bash -e \
${{ env.install-dir }}/bin/run_transport_test_sc.sh \
info
- name: Re-run with increased logging, on failure only
if: |
(!cancelled()) && (steps.transport_test_scripted.outcome == 'failure')
run: |
/usr/bin/bash -e \
${{ env.install-dir }}/bin/run_transport_test_sc.sh \
data
# The following [Exercise mode] tests follow the instructions in bin/transport_test/README.txt.
# Note that the creation of ~/bin/ex_..._run and placement of executables there, plus
# /tmp/var/run for run-time files (PID files and similar), is a necessary consequence of
# the ipc::session safety model for estabshing IPC conversations (sessions).
- name: Prepare IPC-session safety-friendly run-time environment for [transport_test - Exercise mode]
if: |
(!cancelled()) && (!matrix.build-test-cfg.skip-transport-tests-ex)
run: |
mkdir -p ~/bin/ex_srv_run ~/bin/ex_cli_run
mkdir -p /tmp/var/run
cp -v ${{ env.install-dir }}/bin/transport_test/transport_test.exec \
~/bin/ex_srv.exec
cp -v ~/bin/ex_srv.exec ~/bin/ex_cli.exec
- name: Prepare run script for [transport_test - Exercise mode] variations below
if: |
(!cancelled()) && (!matrix.build-test-cfg.skip-transport-tests-ex)
run: |
cat <<'EOF' > ${{ env.install-dir }}/bin/run_transport_test_ex.sh
# Script created by pipeline during job.
echo "Log level: [$1]."
echo "Exercise sub-mode: [$2]."
echo "Sub-mode snippet (none or 'shm-?'): [$3]."
OUT_DIR=${{ env.install-dir }}/bin/logs/transport_test/exercise/$2
mkdir -p $OUT_DIR
SUPP_DIR_A=${{ github.workspace }}/flow/src
{ cat $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } || true
if [ "$3" == '-shm-j' ]; then # SHM-jemalloc mode => jemalloc is in fact exercised => suppress jemalloc stuff.
SUPP_DIR_B=${{ github.workspace }}/ipc_shm_arena_lend/src
{ cat $SUPP_DIR_B/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_B/${{ env.san-suppress-cfg-in-file2 }} \
>> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } || true
fi
${{ env.setup-tests-env }}
~/bin/ex_srv.exec exercise-srv$3 $OUT_DIR/srv.log info $1 \
> $OUT_DIR/srv.console.log 2>&1 &
SRV_PID=$!
sleep 5
~/bin/ex_cli.exec exercise-cli$3 $OUT_DIR/cli.log info $1 \
> $OUT_DIR/cli.console.log 2>&1 &
CLI_PID=$!
if wait $SRV_PID; then SRV_EC=0; else SRV_EC=$?; fi
echo "Server finished with code [$SRV_EC]."
if wait $CLI_PID; then CLI_EC=0; else CLI_EC=$?; fi
echo "Client finished with code [$CLI_EC]."
[ $SRV_EC -eq 0 ] && [ $CLI_EC -eq 0 ]
EOF
- name: Run integration test [transport_test - Exercise mode - Heap sub-mode]
id: transport_test_ex_heap
if: |
(!cancelled()) && (!matrix.build-test-cfg.skip-transport-tests-ex)
run: |
/usr/bin/bash -e \
${{ env.install-dir }}/bin/run_transport_test_ex.sh \
info heap
- name: Re-run with increased logging, on failure only
if: |
(!cancelled()) && (steps.transport_test_ex_heap.outcome == 'failure')
run: |
/usr/bin/bash -e \
${{ env.install-dir }}/bin/run_transport_test_ex.sh \
data heap_log_level_data
- name: Run integration test [transport_test - Exercise mode - SHM-classic sub-mode]
id: transport_test_ex_shm_c
if: |
(!cancelled()) && (!matrix.build-test-cfg.skip-transport-tests-ex)
run: |
/usr/bin/bash -e \
${{ env.install-dir }}/bin/run_transport_test_ex.sh \
info shm_classic -shm-c
- name: Re-run with increased logging, on failure only
if: |
(!cancelled()) && (steps.transport_test_ex_shm_c.outcome == 'failure')
run: |
/usr/bin/bash -e \
${{ env.install-dir }}/bin/run_transport_test_ex.sh \
data shm_classic_log_level_data -shm-c
- name: Run integration test [transport_test - Exercise mode - SHM-jemalloc sub-mode]
id: transport_test_ex_shm_j
if: |
(!cancelled()) && (!matrix.build-test-cfg.skip-transport-tests-ex)
run: |
/usr/bin/bash -e \
${{ env.install-dir }}/bin/run_transport_test_ex.sh \
info shm_jemalloc -shm-j
- name: Re-run with increased logging, on failure only
if: |
(!cancelled()) && (steps.transport_test_ex_shm_j.outcome == 'failure')
run: |
/usr/bin/bash -e \
${{ env.install-dir }}/bin/run_transport_test_ex.sh \
data shm_jemalloc_log_level_data -shm-j
# See earlier comment block about why we saved all the logs including for console-output-only tests/demos.
- name: Check test/demo logs for non-fatal sanitizer error(s)
if: |
(!cancelled()) && (matrix.build-test-cfg.sanitizer-name == 'ubsan')
run: |
cd ${{ env.install-dir }}/bin/logs
# grep returns 0 if 1+ found, 1 if none found, 2+ on error. So check results explicitly instead of -e.
# Namely, for us, the only OK result is an empty stdout and empty stderr.
# Otherwise either grep failed (unlikely) or found 1+ problems; either way redirection target will
# be not-empty, and we will force failure.
{ grep 'SUMMARY: UndefinedBehaviorSanitizer:' `find . -type f` > san_failure_summaries.txt 2>&1; } || true
if [ -s san_failure_summaries.txt ]; then
echo 'Error(s) found. Pipeline will fail. Failures summarized below.'
echo 'Please peruse uploaded log artifacts, resolve the issues, and run pipeline again.'
echo '[[[ file--'
cat san_failure_summaries.txt
echo '--file ]]]'
false
fi
echo 'No errors found in logs.'
- name: Package test/demo logs tarball
if: |
always()
run: |
cd ${{ env.install-dir }}/bin
tar cvzf logs.tgz logs
rm -rf logs # Save runner space.
- name: Upload test/demo logs (please inspect if failure(s) seen above)
if: |
always()
uses: actions/upload-artifact@v3
with:
name: ipc-logs-${{ matrix.compiler.id }}-${{ matrix.build-test-cfg.id }}
path: |
${{ env.install-dir }}/bin/logs.tgz
# TODO: Look into the topic of debuggability in case of a crash. Is a core generated? Is it saved?
# Do we need to manually save it as an artifact? For that matter we would then need the binary and
# ideally the source. For now we save any logs that are not printed to console, and where possible
# our tests/demos keep it the console exclusively (but in some cases, such as transport_test, it
# is not practical due to parallel execution and other aspects). In general it is always better that
# logs are sufficient; but look into situations where they are not.
#
# Don't forget situation where program exits due to a sanitizer reporting problem (ASAN, etc.); is
# there a core? Do we need one? Etc.
#
# Possibly this is all handled beautifully automatically; then this should be deleted.