Communication layer abstraction for expanding communication properties.

[Jakio815]

Abstract socket layers

In federated execution, currently, many functions are directly using sockets. We plan to separate this by adding an abstract socket layer to simplify programming and enable scaling communication.

Plans

Many functions are handling sockets directly. Instead of passing sockets directly, struct pointers will be passed.

The main functions to be changed are the read() and write() functions, which currently looks like this.

ssize_t write_to_socket_errexit(
		int socket,
		size_t num_bytes,
		unsigned char* buffer,
		char* format, ...)

ssize_t read_from_socket_errexit(
		int socket,
		size_t num_bytes,
		unsigned char* buffer,
		char* format, ...)

Instead of directly passing sockets, I will pass a new struct lf_network_port_t.

struct lf_network_port_t

Each functions will only put the lf_network_port_t into each functions, and it represents the destination of the message.

/// net_util.h
// The "abstract" base struct, contains any fields which are common to all: 
typedef struct {
  char * name;
  void * port;
} lf_network_port_t;

lf_network_port_t * lf_network_port_create();
void lf_network_port_free(lf_network_port_t * port);
int lf_network_port_send(lf_network_port_t *port, msg_t msg, dst_t dsg);
...

The low level child socket inherits lf_network_port_t.

///net_socket.h
// Low-level socket child struct. Struct inheriting lf_network_port_t
typedef struct {
  lf_network_port_t base;
  int socket;
} lf_network_port_socket_t;

///net_tls.h
// TLS struct inheriting lf_network_port_t
typedef struct {
  lf_network_port_t base;
  SSL_CTX *ctx;
  SSL *ssl;
  ...
} lf_network_port_tls_t;

Each functions are implemented inside the child.c files.

/// net_socket.c
lf_network_port_t *lf_network_port_create() {
  // Dynamically allocate the child struct
  lf_network_port_socket_t * sock = malloc(sizeof(lf_network_port_socket_t));
  // Create a socket
  sock.socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
  // Return an "abstract" pointer
  return sock.base;
}

So, the write() function would look like this.

ssize_t write_to_lf_network_port_errexit(
		lf_network_port_t *lf_network_port,
		size_t num_bytes,
		unsigned char* buffer,
		char* format, ...)

Inside the write() it will write in different ways, by which socket it uses.
If it is a normal socket, it would just send messages. If it's a TLS_socket, it will encrypt messages and send it, and if it's a DDS_socket, it will send it using DDS.

File structures

.
├── CMakeLists.txt
├── RTI
│   ├── CMakeLists.txt
│   ├── README.md
│   ├── enclave.c
│   ├── enclave.h
│   ├── enclave_impl.c
│   ├── message_record
│   │   ├── message_record.c
│   │   ├── message_record.h
│   │   └── rti_pqueue_support.h
│   ├── rti.Dockerfile
│   ├── rti.c
│   ├── rti_lib.c
│   └── rti_lib.h
├── clock-sync.c
├── federate.c
└── networks
    ├── CMakeLists.txt
    └── net_util.c

net_util.c will be moved under networks. The networks will future include network options such as net_tls.c. It is worked on #292.

Challenges

1. read and writes

There have to be a lot of changes in the current code to make a write() and read() function to work in multiple communication methods.

The main problem is the read() function called multiple times, during only one write(). The point is that the current code write() the total buffer at once, but on the other side that receives the message, it read() in sequences.

So example, a federate sends a MSG_TYPE_TIMESTAMP message, which is a 9-byte message with one byte message type, and 8 bytes of payload. Then, the RTI first calls the read() function reading one byte, and then calls the read() function again reading 8 bytes.
There are two problems with this.

1. Multiple system calls are inefficient.
2. Encryption is impossible. If the total message is encrypted, only the total message can be decrypted at once. So, reading bytes separately is impossible.

2. Will it work for pub-sub protocols?

It may not work for pub-sub based protocols such as MQTT and DDS.
There must be QoS settings to ensure the order of messages.
The full example code of a DDS publisher is in here.
MQTT example code is in here.

Challenges for security

1. RTI needs to be able to read the message, to get the federate_id.
   Does the RTI need to decrypt it? Then does the RTI have every federate’s keys?

2. Need to handle messages, which are longer than the maximum buffer size 256 bytes.
   E.g. Let’s say the message sent was 300 bytes. The message will get encrypted and longer. How do we handle this?

Current functions related to sockets (Yet being updated)

These are the current functions that are using sockets or ports. It is listed in the order of execution of the RTI.

RTI

• main()
  • void initialize_federate(federate_t* fed, uint16_t id) rti_lib.c
  • int32_t start_rti_server(uint16_t port) rti_lib.c
    • int create_server(int32_t specified_port, uint16_t port, socket_type_t socket_type) rti_lib.c
      • int create_real_time_tcp_socket_errexit() net_util.c
  • void wait_for_federates(int socket_descriptor) rti_lib.c
    • void connect_to_federates(int socket_descriptor) rti_lib.c

      • bool authenticate_federate(int socket) rti_lib.c

        • ssize_t read_from_socket_errexit() x2 net_util.c
        • ssize_t write_to_socket(int socket, size_t num_bytes, unsigned char* buffer) net_util.c
        • void send_reject(int socket_id, unsigned char error_code) rti_lib.c
          • ssize_t write_to_socket_errexit(int socket, size_t num_bytes, unsigned char* buffer, char* format, ...) net_util.c

      • int32_t receive_and_check_fed_id_message(int socket_id, struct sockaddr_in* client_fd) rti_lib.c

        • read_from_socket_errexit(), send_reject()

      • int receive_connection_information(int socket_id, uint16_t fed_id) rti_lib.c

        • read_from_socket_errexit(), send_reject()

      • int receive_udp_message_and_set_up_clock_sync(int socket_id, uint16_t fed_id)

        • read_from_socket_errexit(), send_reject()
        • void send_physical_clock(unsigned char message_type, federate_t* fed, socket_type_t socket_type)
          • write_to_socket_errexit()
          • void handle_physical_clock_sync_message(federate_t* my_fed, socket_type_t socket_type)
            • send_physical_clock()

      • void* federate_thread_TCP(void* fed) rti_lib.c

        • read_from_socket()
        • void handle_timestamp(federate_t *my_fed)
        • void handle_address_query(uint16_t fed_id)
        • void handle_address_ad(uint16_t federate_id)
        • void handle_timed_message(federate_t* sending_federate, unsigned char* buffer)
        • void handle_federate_resign(federate_t *my_fed)
        • void handle_next_event_tag(federate_t* fed)
        • void handle_logical_tag_complete(federate_t* fed)
        • void handle_stop_request_message(federate_t* fed)
        • void handle_stop_request_reply(federate_t* fed)
        • void handle_port_absent_message(federate_t* sending_federate, unsigned char* buffer)
    • shutdown() rti_lib.c

Federate

TODO List

• Abstract socket layer

Points to check

• RTI forwards the federate's message (e,g, A -> RTI -> B), but needs to read() it to get the federate_id (of B) .

[Jakio815]

Hi, @erlingrj I would like some insights of this.
Prof. Kim suggested to use void pointers inside lf_network_port instead of using #ifdef #endifs, and type cast them by using switch case. However, I'm concerned that at the compile time, the types will all have to be declared, which means all additional header files to use TLS, SST, or DDS will need to be included even though it is not used. What do you think is a better idea? Using ifdefs or type casting?

Marten and Prof. Kim recommended you, because you handled the C code adding different target platforms.

[erlingrj]

My initial thought is to make C-style inheritance. It would look like this:

// The "abstract" base struct, contains any fields which are common to all: 
typedef struct {
  char * name;
  int a;
} lf_network_port_t;
 
// Low-level socket child struct. Should be defined in its own header file
typedef struct {
  lf_network_port_t base;
  int socket;
} lf_network_port_socket_t;

// TLS socket child struct. This should be defined in its own header file
typedef struct {
  lf_network_port_t base;
  SSL_CTX *ctx;
  SSL *ssl;
} lf_network_port_tls_t;

Since the first field of lf_network_port_socket_t is a lf_network_port_t, then we can cast any lf_network_port_socket_t to a lf_network_port_t (also true for pointers).

I would then create a "virtual" API for the "abstract" lf_network_port_t which should be implemented by each child

lf_network_port_t * lf_network_port_create();
void lf_network_port_free(lf_network_port_t * port);
int lf_network_port_send(lf_network_port_t *port, msg_t msg, dst_t dsg);
...

This would be implemented in the source files for each of different types of ports. E.g. for the socket:

lf_network_port_t *lf_network_port_create() {
  // Dynamically allocate the child struct
  lf_network_port_socket_t * sock = malloc(sizeof(lf_network_port_socket_t));
  // Create a socket
  sock.socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
  // Return an "abstract" pointer
  return sock.base;
}

The challenge here becomes the following: "Can you define a common abstract API for all the network types so that they all have the same interface to the LF codebase?" The different network types have different ways of being configured and such, but can all of that be hidden behind a single, network-agnostic API?

So, in a sense, I agree with Prof Kim that we should use typecasts, but we should make it elaborate with inheritance and only do the typecasting to e.g. TLS socket inside of tls_socket.c and only include TLS specific header files there. Here is a nice resource for object-oriented programming in C: https://www.state-machine.com/oop

[Jakio815]

I totally agree that the largest challenge is as what you mentioned. "Is it possible to have a single API for all the network types". I also think it may not be possible actually, because not only the sending and receiving APIs, there are many different initializing settings to do for each network type.
My answer is kind of a "yes" because I think we could also wrap up the initialization and managing additional structs together. Not 100% sure but I'll have to go for a try.

I got your idea of separating each type into different files.

Such as

///  lf_network_port_tls.h
// TLS socket child struct. This should be defined in its own header file
typedef struct {
  lf_network_port_t base;
  SSL_CTX *ctx;
  SSL *ssl;
} lf_network_port_tls_t;

lf_network_port_t * lf_network_port_create();
void lf_network_port_free(lf_network_port_t * port);
int lf_network_port_send(lf_network_port_t *port, msg_t msg, dst_t dsg);
...

Checking how the platforms works will be great. Thanks!