Create a documentation directory and updating the serial subsystem docs

docs/serial.md

@@ -0,0 +1,180 @@
+# The sDDF Serial Subsystem
+
+## System architecture
+
+The serial subsystem adheres to the sDDF design principles of modularalised components split via
+separation of concerns which communicate via shared data structures and microkit channels.
+
+In order to transmit and receive via the uart device, clients of the system must interface with
+serial virtualisers, which in turn interface with the uart driver - the only protection domain with
+access to the uart device. Characters are passed between components via two shared memory regions -
+one a *data region* containing the characters themselves, and the other a *queue region* containing
+a queue data structure that allows components to deduce the last position in the data region written
+to or read by their neighbour. Queues are implemented as simple ring buffers containing a *head* and
+*tail* index, as well as additional *notification flags* which allow components to deduce whether
+the consumer or producer of the queue must be notified after a change of queue state has occurred.
+
+The implementation of these data structures, along with helper functions to use them, can be found
+in `include/sddf/serial/queue.h`. Further information on how these data structures work, as well as
+how components utilise the notification flags to avoid both deadlocks and unnecessary system calls
+can be found in the sDDF design document.
+
+## Accessing serial data structures
+
+Clients are advised not to access serial queue and data regions directly, and instead use *serial
+queue handles* which contain references to both the queue and data regions, as well as the data
+region size. While the queue and data regions are shared between protection domains, queue handles
+are not. This allows components to keep size as a local field to prevent tampering, providing
+protection against memory faults at invalid offsets due to incorrect changes to size. Utilising the
+queue handle also allows clients to make use of the serial queue library, which helps prevent
+invalid updates to queue state.
+
+### Transmission
+
+If a client wishes to print strings, it is recommended to do so via `sddf_printf` (which can be
+found in `include/sddf/util/printf.h`), rather than using the serial queue library directly.
+`sddf_printf` calls `sddf_putchar` on each character in its input string. If the definition of
+`sddf_putchar` is linked with `libsddf_util.a` (containing `util/putchar_serial.c`), rather than
+`libsddf_util_debug.a` (containing `util/putchar_debug.c`), `sddf_printf` outputs to the uart device
+using a simple multiplexing mechanism that vastly reduces jumbled client output.
+
+The multiplexing mechanism buffers characters written to the data region by the client, only
+notifying the transmit virtualiser when the region becomes full, or the `FLUSH_CHAR` is seen (which
+by default is a `\n`).
+
+If your protection domain requires each character to be output to serial individually,
+`util/putchar_serial.c` provides a `sddf_putchar_unbuffered` function, which notifies the transmit
+virtualiser for each character it receives. Examples of how both `sddf_printf` and
+`sddf_putchar_unbuffered` are used can be found in the `serial_server` protection domain in the
+serial example.
+
+Clients may also update serial transmit data structures themselves, using `util/putchar_serial.c` as
+a guide. If you wish to use `sddf_printf` linked to serial output, ensure to initialise both the
+queue handle and serial putchar library first, details of how to do this can be found below.
+
+### Reception
+
+In contrast to transmission which is client driven, receiving a character is device driven, and data
+is passed from the uart driver to the receive virtualiser, and finally to the client. The receive
+virtualiser passes non-special characters to whichever client is currently selected to receive
+input. By default this is client 0, but can be changed by the user by entering the switch character,
+followed by the client number and `SERIAL_TERMINATE_NUM`. Once a character has been written to a
+client's data region, the virtualiser will then notify the client depending on the value of the
+client's flag.
+
+If a protection domain wishes to receive characters, they must have a dedicated microkit channel to
+be notified on by the receive virtualiser, and they must ensure to set their notification flag if
+they require a notification upon a character being received. The standard protocol for this is
+demonstrated by the serial server protection domain in the serial example.
+
+## Configuration
+
+In `examples/serial/include/serial_config/serial_config.h` you will find a prototype for a serial
+config file. This file contains the system configuration information from your `.system` file which
+is needed for your serial components and clients to run correctly. It also contains configurable
+options for the serial subsystem which are described below:
+
+1. **SERIAL_NUM_CLIENTS** - this defines the number of clients the serial virtualisers will have.
+
+2. **DATA_REGION_SIZES**-  this defines the size of the data regions of each client. While the queue
+regions are of fixed size, clients can have variably sized data regions to buffer characters they
+wish to receive or transmit. Once data region sizes have been configured in the system and config
+file, the client and virtualiser queue initialisation functions should be updated to correctly
+initialise their queue handle data structures. Currently, these functions assume that the
+virtualiser's shared queue and data regions with their clients are mapped contiguously. This
+simplifies the initialisation process, as the virtualiser needs only to know the address of its
+first client's queue and data region and may deduce the addresses of the other client's regions from
+the size of the queue region and data regions. The subsystem does not require this contiguous layout
+to function correctly and any valid layout may be used, as long as the initialisation functions in
+the config file are updated accordingly.
+
+3. **SERIAL_TX_ONLY** - enable this if you only want to use the transmit functionality of the serial
+subsystem. This stops the uart driver from enabling the receive functionality of the device.
+
+4. **SERIAL_WITH_COLOUR** - enable this if you want client's outputs to be different colours. This
+mechanism works by appending a colour code before and after a client's string. Note that the
+transmit virtualiser supports up to 256 colours. Also, the transmit virtualiser does not check
+client output for colour sequences, so there is no guarantee that clients will only output in their
+assigned colour. Upon initialisation, the transmit virtualiser will print the name of each client in
+the colour assigned to it.
+
+5. **SERIAL_SWITCH_CHAR** and **SERIAL_TERMINATE_NUM** - these characters control the receive
+virtualiser's input switching mechanism. To switch the input stream to a different client, input
+**SERIAL_SWITCH_CHAR** followed by up to 4 numeric characters corresponding to the new client
+number, and terminate numeric input with **SERIAL_TERMINATE_NUM**. If the system is build in debug
+mode, the receive virtualiser will output a success message if the client number you entered is
+valid, and error message if invalid. Client 0 receives input by default.
+
+6. **UART_DEFAULT_BAUD** - this determines the baud rate that the uart driver will configure for
+the device. Baud rate is always set explicitly instead of detected automatically.
+
+7. **SERIAL_CONSOLE_BEGIN_STRING** - this string is printed by the transmit virtualiser upon
+initialisation completion.
+
+If the system file is changed, or the serial subsystem is included into another system, this config
+file will need to be edited or re-created to reflect the new system. Be sure to check that the
+`*_init\_sys` functions correctly initialise each protection domains data structures.
+
+## Interfacing with Other Systems
+To include the serial subsystem into an existing system, the following steps must be taken:
+### **`.system` File**
+You must update your system file to include serial data and queue regions for each client and the
+uart driver.
+
+You must also include the uart driver, transmit virtualiser, and optionally the receive virtualiser
+protection domains.
+
+Finally you must include channels between your clients and the virtualisers, as well as between the
+virtualisers and the uart driver. The channel numbers that clients use to notify the virtualisers
+may be arbitrary, but the virtualisers expect client channel numbers to be a consecutively
+increasing list starting from 1. The default channel numbers used between the virtualisers and the
+driver can be found either in the serial example system file, or as macros listed at the top of of
+each `.c` file. Channels may be changed, but they will need to be updated in the corresponding
+files.
+
+### **`serial_config` File**
+A new `serial_config` file must be created for your system, containing relevant details of the
+system file including client names and data region sizes, as well as updated initialisation
+functions for clients and virtualisers.
+
+### **Makefile**
+You must ensure to build the required serial component images (transmit virtualiser and receive
+virtualiser). This is most easily done by including the `serial/components/serial_components.mk`
+make snippet and ensuring the required components are listed within your images to be built.
+Similarly, you must ensure to build the correct uart driver for the device you are building for,
+which can most easily be accomplished by including the relevant make snipped (i.e.
+`drivers/serial/arm/uart_driver.mk`).
+
+Most serial components and clients will need to include the serial config file, so it is best to put
+its directory in your include paths.
+
+For each component you wish to print to serial when using `sddf_printf`, you must ensure they are
+linked with `libsddf_util.a` as opposed to `libsddf_util_debug.a` to resolve the definition of
+`sddf_putchar`. These libraries can be be built by including `util/util.mk`.
+
+### **Protection Domains**
+Each protection domain that prints to serial must include the serial queue library and config file.
+They must also have the following declarations - possibly repeated if input is required as well:
+
+```
+#define SERIAL_TX_CH 0
+
+char *serial_tx_data;
+serial_queue_t *serial_tx_queue;
+serial_queue_handle_t serial_tx_queue_handle;
+```
+
+The channel must match up with the system file, and is used with the queue handle to initialise the
+putchar library (by calling `serial_putchar_init`).
+
+Prior to passing the the queue handle to `serial_putchar_init`, it should first be initialised with
+the queue and data region addresses (which will be patched by microkit). A queue handle can be
+initialised directly using `serial_queue_init`, however is typically initialised indirectly using
+`serial_cli_queue_init_sys` defined in the serial config file. `serial_cli_queue_init_sys`
+initialises both the receive and transmit serial queue handles, and allows queue sizes to be
+parametrised via the microkit name variable of the client (useful if many clients share the same
+executable).
+
+If a client is receiving data, it is recommended to add notification handling code to its notified
+function which is invoked in response to a notification from the receive virtualiser. A simple
+example of this can be found in the serial example.

examples/serial/README.md

@@ -37,78 +37,28 @@ zig build -Dsdk=/path/to/sdk -Dboard=qemu_arm_virt qemu
 
 The final bootable image will be in `zig-out/bin/loader.img`.
 
-## Configuration
-
-In the serial example directory you will find the `include/serial_config/serial_config.h` file.
-This file contains system configuration information that is dependent on your `.system` file, as
-well as the following configuration options:
-
-1. **SERIAL_TX_ONLY** - enable this if you only want to use the transmit functionality of the
-serial subsystem. This stops the uart driver from enabling the receive functionality of the
-device.
-2. **SERIAL_WITH_COLOUR** - enable this if you want clients outputs to be different colours. This
-mechanism works by appending a colour code before and after a clients string. Note that the
-transmit virtualiser supports up to 256 colours. Also, the transmit virtualiser does not check
-client output for colour sequences, so there is no gaurantee that clients will only output in
-their own colour. Upon initialisation, the transmit virtualiser will print the name of each client
-in the colour assigned to it.
-3. **SERIAL_SWITCH_CHAR** and **SERIAL_TERMINATE_NUM** - these characters control the receive
-virtualisers input switching mechanism. To switch the input stream to a different client, input
-**SERIAL_SWITCH_CHAR** followed by up to 4 numeric characters corresponding to the new client
-number, and terminate numeric input with **SERIAL_TERMINATE_NUM**. Upon success there will be no
-output, while upon error the receive virtualiser will print a debug failure message. Client 0
-receives input upon initialisation.
-4. **UART_DEFAULT_BAUD** - this determines the baud rate that the uart driver will configure for
-the device. Baud rate is always set explicitly instead of detected automatically.
-5. **SERIAL_CONSOLE_BEGIN_STRING** - this string is printed by the transmit virtualiser upon
-initialisation completion. This is to support input beginning in the interfacing serial server.
-
-If the system file is changed, or the serial subsystem is included into another system, this config
-file will need to be edited or re-created to reflect the new system. Be sure to check that the 
-`*_init\_sys` functions correctly initialise each protection domains data structures.
-
-## Interfacing with Other Systems
-To include the serial subsystem into an existing system, the following steps must be taken:
-* **.system File**
-You must update your system file to include serial data and queue regions for each client and the
-uart driver. You must also include the uart driver, transmit virtualiser, and optionally the
-receive virtualiser protection domains. Finally you must include channels between your clients and
-the virtualisers, as well as between the virtualisers and the uart driver.
-* **`serial_config` File**
-A new `serial_config` file must be created for your system, containing relevent details of the
-system file including client names and queue sizes, as well as updated initialisation functions
-for clients and virtualisers.
-* **Makefile**
-You must include directories for **SERIAL_COMPONENTS**, the **UART_DRIVER** and your
-**SERIAL_CONFIG_INCLUDE**. You must also supply **SERIAL_NUM_CLIENTS**. You must add the uart
-driver, transmit virtualiser and optionally the receive virtualiser to your image list. You must
-add your serial include directory to your cflags, and finally you must include the uart driver
-and serial_components make files. For each component you wish to have access to the serial
-subsystem, you must link their printf object file with `libsddf_util.a` as opposed to
-`libsddf_util_debug.a`. This will ensure printf invokes the serial _sddf_putchar.
-* **Protection Domains**
-Each protection domain that outputs to serial must include the serial queue library as well as
-`serial_config.h`. They must also have the following declarations/definitions:
+## Description
+The serial server example system contains two clients which are both able to transmit and receive
+characters over serial. The clients are based on the same executable generated from
+`examples/serial/serial_server.c`.
 
-```
-#define SERIAL_TX_CH 0
+The serial server client demonstrates two methods of outputting to serial: the first uses
+`sddf_printf` (linked with `_sddf_putchar` defined in `util/putchar_serial.c`), and the second uses
+`sddf_putchar_unbuffered` defined in the same file.
 
-char *serial_tx_data;
-serial_queue_t *serial_tx_queue;
-serial_queue_handle_t serial_tx_queue_handle;
-```
+The first method demonstrates character buffering based on a flush character or queue capacity, the
+second demonstrates characters being transmitted in an unbuffered fashion, typically used by a REPL.
+
+The serial server client sits in an event loop, awaiting notification from the receive virtualiser.
+When a notification is received, the client outputs the character in an unbuffered fashion. Every 10
+characters the client prints a message in a buffered fashion.
+
+This notification handling process with the receive virtualiser demonstrates how the
+*producer_signalled* flag is to be used to prevent against missed notifications from the virtualiser
+which could result in missed characters or deadlock.
+
+In this example, **SERIAL_WITH_COLOUR** is enabled so each client prints with a
+different colour.
 
-If they require serial input then equivalent declarations must exist for the receive serial
-objects. Finally, during initialisation and prior to calling printf, they must initialise their 
-serial queue(s) by calling `serial_cli_queue_init_sys` as well as `serial_putchar_init` which
-allows them to also use `sddf_putchar_unbuffered`.
-
-## Example
-The serial server example system contains two clients which can both receive serial data as well
-as transmit. By default, the example has SERIAL_WITH_COLOUR enabled so each client prints with a
-different colour. Each client boots up and prints a hello world message when initialisation is
-completed, and waits for input. When a character is received, each client will re-transmit the
-character using `sddf_putchar_unbuffered` which flushes the character to the device immediately. Every
-tenth character each client will print a string containing their name using `sddf_printf` which
-calls the serial `_sddf_putchar`, flushing characters to the device only when a `\n` is
-encountered.
+## Documentation
+Further documentation for the serial subsystem can be found in `docs/serial.md`