Need API to signal when adding a new path is possible

[huitema]

When the multipath option is enabled, applications will want to create new paths based on their own logic: when they learn that a new network interface becomes available, when something goes wrong with and existing path, maybe when the load profile of the application changes. The new path should use the next available path_id N, but knowing that path_id=N is not sufficient. Creating a new path is only possible if both endpoints have agreed to enable that new path:

• both endpoints have advertised max_path_id >= N
• both endpoints have provided at least one connection ID wth path_id = N

The implementation in picoquic_demo checks these conditions in the "network loop" callback, in the handling of picoquic_packet_loop_after_receive, by checking:

int remote_cid_ready = (picoquic_obtain_stashed_cnxid(cb_ctx->cnx_client, 1) != NULL);

There are several issues with that:

• picoquic_obtain_stashed_cnxid is an internal API, which may well be need to be redesigned over time as we get more experience in handling multipath.
• the second argument 1 in the line above is a unique_path_id, but the application would need to track successive path creation attempts to track the correct value.
• the code is doing active polling after any packet is received, which can create quite a bit of overhead.
• the code is running in a low level packet loop callback, which should be focus on the handling of sockets and timers and is arguably the wrong place for application logic.

The code would be simpler and more efficient if we had better APIs:

• maybe a subscribe/event API to check whether and when it is possible to add a new path: a function to check whether path creation is possible, an a callback signalling when it is.
• maybe additional error codes in picoquic_probe_new_path_ex, so the application can just try to create a new path but get informed when some condition is not met yet -- instead of adding a bunch of logic to test these conditions in the application code.
• maybe have an asynchronous option for picoquic_probe_new_path_ex, such as queuing path creation until the necessary conditions are met.

[huitema]

There is another issue with picoquic_probe_new_path_ex: it takes as input both the client address and the server address. In most cases, the server address is constant: it should be the same as the address chosen by the client for the first connection. The code in picoquicdemo ensures that by looking for the destination address of path 0, but this is ambiguous: in some cases, the first path will be abandoned. Also, if preferred address is specified and verified, it would make sense to pick that instead of the user specified value. It would probably be better to leave the destination address unspecified, and have the code in picoquic_probe_new_path_ex pick the value.

The code also verifies the client address family, and check that it matches the "initial" value. But we could see the case of initiating a connection with IPv4, then switching to IPv6 if the server provides a preferred IPv6 address.

[sanjaybhat2004]

I agree, having better API's would make this easier for user of the library. I feel like option 1 and option 2 both seem like good choices, however if we choose option 3, it feels like we will be restricting the control the user of the API has, in case they later want to not create the new path after it has been queued.

In option 1 I can see how an implementation might be done, but in option 2 I have a question. Where is the client expected to call this path creation function, since the control flow would be inside the picoquic_packet_loop, and if the user needs to call the somewhere function inside the packet loop code then isn't the purpose creation of this API lost?

[huitema]

I have a PR in progress to add the error codes to picoquic_probe_new_path_ex, so the application can act upon them:

• PICOQUIC_ERROR_PATH_DUPLICATE: there is already an existing path with the same 4 tuple. This error only happens if the multipath extensions are not negotiated, because the multipath extensions allow creation of multiple paths with the same 4 tuple.
• PICOQUIC_ERROR_PATH_ID_BLOCKED: when using the multipath extension, the peers manage a max_path_id value. The code cannot create a new path if the path_id would exceed the limit negotiated with the peer. Applications encountering that error code should wait until the peer has increased the limit. They may want to signal the issue to the peer by queuing a PATHS_BLOCKED frame.
• PICOQUIC_ERROR_PATH_CID_BLOCKED: when using the multipath extension, the peers use NEW_PATH_CONNECTION_ID frame to provide CIDs associated with each valid path_id. The error occurs when the peer has not yet provided CIDs for the next path_id. Applications encountering the error should wait until the peer provides CID for the path. They may want to signal the issue to the peer by queuing a PATH_CIDS_BLOCKED frame.
• PICOQUIC_ERROR_PATH_ADDRESS_FAMILY: API error. The application is trying to use a four tuple with different address family for source and destination.
• PICOQUIC_ERROR_PATH_NOT_READY: API error. The application is trying to create paths before the connection handshake is complete. The application should wait until it is notified that the connection is ready.
• PICOQUIC_ERROR_PATH_LIMIT_EXCEEDED: The application is trying to create more simultaneous paths than allowed. It will need to close one of the existing paths before creating a new one.

The PR will also allow applications to leave the destination address parameter NULL, in which case the code will pick the default address for the server that matches the address family of the source address.

The "path limit exceeded" compares the number of current paths to the compilation parameter PICOQUIC_NB_PATH_TARGET. We may want to make that a configuration parameter.

Regarding the need for a signal that multipath is ready, we need to elements: checking whether ready, and being informed.
Checking could be done in the picoquic_probe_new_path_ex function itself, as anything else will duplicate code in that function. We will need an API do ask when path creation is possible, and then a callback when it is.

[huitema]

Regarding the application flow, I believe that this really falls within the application callback. The current picoquicdemo code places it in the packet loop callback because the early prototypes would dynamically create a socket, but now the socket creation and path creation functions are well separated. It is just a matter of moving the code around.

[huitema]

@sanjaybhat2004 you may want to check PR #1836 "Use path allowed API for testing multipath".

[sanjaybhat2004]

Looks good, hopefully it makes the work of adding new paths easier for people using the library!