Testing u-nit involves some different approaches, like unit testing and
fuzzy or qemu testing.

Unit test

For inittab, fstab and command line parser, as well as generic lexer.

To generate unit test executable, run `make tests`. It should generate
`inittab_test`, `lexer_test`, `fstab_test` and `cmdline_test` executables
- all must run with no issues.

Fuzzy testing

American Fuzzy Lop (AFL) is run on a single executable, using
`tests/data/parser/inittab` files as input for the fuzzy tool.

To generate AFL ready executable, run `make tests`. It should generate
`afl_inittab_test` executable - that can be run with a command like
`afl-fuzz -i tests/data -o fuzz-results/ ./afl_inittab_test @@`.
See AFL documentation for more details.

QEMU testing

Runs u-nit as PID 1 on a virtual environment using qemu. This virtual
environment needs a kernel image and rootfs. Currently, script that
perform this testing expects a directory `tests/init-qemu`, with files
for the kernel image, an optional initrd and root file system.
Those can be generated with help of `prepare-test-image.sh` script. It
uses `mkosi` [https://github.com/systemd/mkosi] to generate a debian
image, so a `mkosi` installation capable of generating debian images is
necessary. After image generation, script will extract kernel and initrd
files. Their names must be set on `qemu-tests.sh` file, on variables
KERNEL_FILE and INITRD_FILE respectively.

Besides those basic files, `prepare-test-image.sh` also generate
some other files necessary for tests:  `gcov.ext4` - where coverage
obtained by gcov is stored and two empty filesystems, `test-fs.ext4`
and `test-fs2.ext4`, used on fstab tests.

Note that one can create their own kernel directly, if desired. Just
change KERNEL_FILE variable on `qemu-tests.sh` so this new kernel is
used. If INITRD_FILE variable is left empty, no initrd file will
be used - useful for custom kernels that don't use initrd/initramfs.

Tests run on qemu focus on two aspects: inittab and fstab files.
Inittab tests for qemu environment are defined using two files:
XXX-inittab and XXX-inspect for each test XXX. These files live in
`tests/data/qemu/inittab` directory. First file defines an inittab file,
with all programs tha should be run. Second one defines inspections to
be done on u-nit log file. In this directory there's also a `default-fstab`
file to be used on all inittab tests.

The inittab file can use some helper executables to perform its tests.
One is the `sleep_test`, which receives as command line arguments an
arbitrary string and a number. The arbitrary string can be used to
inspect u-nit log file. The number is for how long, in seconds, the
program will sleep. After period finishes, program quits. This can help
test if program B starts only after program A runs.

Another helper is the `sleep_crash_test`, that takes the same two
arguments as `sleep_test`, but it ends with a crash after the timer.
Can be helpful to see if a safe program crashes abnormally, for
instance.

Finally, there are two more helpers: `is_running` and `is_not_running`.
They take as argument a regular expression to be run on `ps aux`
command. Helper `is_running` will log failure if a regular expression
is not found on `ps aux` output, `is_not_running` does the opposite.
They should help to see if a process is really running or not on a given
moment. Useful to test if a service is up, for instance. Note that
the regular expression must not match itself on `ps aux` output - or it
would have wrong results, as it appears on it. A good way to achieve
this is to start the regular expression with `[[]`.

The fstab tests live on `tests/data/qemu/fstab`. As with inittab, they
are divided into two categories: fstab files and inspect files, named
XXX-fstab and XXX-inspect. Both are sent to virtual environment, former
will be in `/etc/fstab` and later on `/usr/share/expected_mounts`.

To start shutdown on virtual environment, add a `kill -s USR2 1` on
inittab file, so u-nit will perform shutdown steps.

The inittab inspect file runs some regular expressions on u-nit log file.
Note that log file also contains any output of programs started by u-nit -
except those that define a controlling terminal. For instance, on start,
`sleep_test` prints `START: /usr/bin/sleep_test <string> <timeout>`.
Regular expressions are defined on three bash arrays variables: EXPECT,
EXPECT_IN_ORDER and NOT_EXPECT. The first searches for each expression
on the array on u-nit log file, and expect that they appear on log,
in the same order they defined on the array. The second simply expects
that each expression appears on u-nit log file, and the last doesn't
expect them at all.

The fstab inspect file basically lists expected mountpoints with
expected options. These options should be in the same format and order
returned by findmnt(8). To list options, use EXPECTED bash array
variable. Each entry must be in the format `MNTPOINT:EXPECTED_OPTIONS`,
like `/mnt/a:rw,relatime,data=ordered`. Another bash array variable
available is UNEXPECTED, that checks if no MNTPOINT entry listed is
actually mounted. For instance, entry `/mnt/b` checks if there's no
`/mnt/b` mountpoint. Useful to test if `noauto` mount option is respected
by u-nit.

Expected options are not automatically derived from fstab file because
this would be a rather complex task, as there'snot a 1:1 mapping from
the fstab options to the ones returned, since kernel version plays an
important role on mount options interactions. For instance, `relatime`
options is always defined unless `noatime` is.

In order to mount the rootfs to send files to virtual environment,
this test needs root permissions. To run it, simply run
`make run-qemu-tests`.
If some test fails, check `qemu-tests.log` file to see what happened.

Coverage information

Coverage information of QEMU tests can also be extracted. GCC gcov
is used to get this information. First step is to use `make coverage`
to compile u-nit with coverage instrumentation enabled.

Then, run tests normally (`make run-qemu-test` or `make run-qemu-fault`,
for instance).

Finally, extract coverage information using `make extract-coverage`
(this will need sudo to get info from inside target image). This step
also uses `lcov` (http://ltp.sourceforge.net/coverage/lcov.php) to
make a nice html report of the coverage. To see the report, call
`make show-coverage-report` to open the browser with it.

After using report, to delete coverage information gathered, to run
a new set of tests, use `make clean-coverage`.

Valgrind testing

Runs u-nit as PID 1 inside container environment under Valgrind.
Container is used instead of QEMU because it easily makes `/proc`
available for Valgrind. (On QEMU /proc wouldn't be mounted until
u-nit mount it.)

This tests perform all tests on `tests/data/qemu` directory with u-nit
under Valgrind, checking for memory issues. If any is found, this
check should fail and show Valgrind output on `qemu-tests.log`.

To run these tests, run `make run-valgrind-tests`.

Note that Valgrind doesn't implement `reboot` syscall, so it will
complain about "syscall 169 not implemented". This should not be an
issue. Indeed, as `reboot` call fails, u-nit shall exit with exit code
1 (`EXIT_FAILURE`), due to `reboot` call failure. On container
environment this is not an issue - container will simply exit. This
even allows some cleanup code to run - currently, on normal execution,
`reboot` is called before cleanup, so it never happens.

Naturally, this test expect that Valgrind is available on target image
 - rootfs.ext2.

Container being used for tests is systemd-nspawn (minimum
systemd-version: 233)
[https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html].
Note that to run the container, root permissions are necessary.
Also note that fstab tests are skipped on container environment - as
there's no simple way to setup on container the expected device nodes.

ASAN testing

Runs u-nit as PID 1 inside container environment with AdressSanitizer
enabled [https://github.com/google/sanitizers]. Container is used
instead of QEMU because it easily makes `/proc` available from ASAN.
(On QEMU /proc wouldn't be mounted until u-nit mount it.)

This tests perform all tests on `tests/data/qemu` directory with u-nit
with ASAN, checking for memory issues. If any is found, this
check should fail and show ASAN output on `qemu-tests.log`.

To run these tests, first ensure that u-nit is compiled with address
sanitizer support, running `make init-asan`, then, to executed the
tests, run `make run-asan-tests`.

Note that an abrupt end of u-nit - like calling `reboot` - doesn't give
a chance to ASAN output its data. To easily allow that and to keep
consistency with behaviour when running under Valgrind, u-nit doesn't
call `reboot` if compiled with address sanitizer. So it will end up
running all clean up code and exit with exit code 1 (`EXIT_FAILURE`),
just like when u-nit runs under Valgrind.

This test expect that AddressSanitizer libraires are available on
target image - rootfs.ext2.

Container being used for tests is systemd-nspawn (minimum
systemd-version: 233)
[https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html].
Note that to run the container, root permissions are necessary.
Also note that fstab tests are skipped on container environment - as
there's no simple way to setup on container the expected device nodes.

Fault injection

Runs u-nit, injecting faults on some library calls, like `calloc` or
`epoll_wait`. Uses libfiu (https://blitiri.com.ar/p/libfiu/) library
to inject the fails.

Failures are defined on `tests/data/qemu/fault-definitions` file. This
file has the set of functions that will fail, defined in the format
expected by libfiu. Note that several tests are defined there and each
test may define several functions to fail. A test is everything defined
between `""`. To have more than on function failing per test, new
functions need to be on next line, but still inside the same quotes. For
instance:

    "enable_random name=linux/io/dup2,probability=0.2
enable_random name=linux/io/timerfd_create,probability=0.2"

Defines a single test, in which both `dup2` and `timerfd_create`
may fail with a probality of 20%.

Each test is run against all tests defined on `tests/data/qemu`. As
tests are non deterministic - since they will fail at random, given
specified probability - each test is run five times. This is so that
it's possible to inject faults on a function that is used in different
parts of u-nit (so we don't always fault on the first `calloc`, for
instance).

To run fault injection tests, simply run `make run-qemu-fault`. Note
that these tests can take very long time. They are a nice way to
expand coverage of tests - since will help test error handling code -
so make sure to run them with coverage enabled and extract coverage.